Distributed graph neural network training method and system based on model migration
Patent Information
- Application Number
- CN202410142568.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-01
- Publication Date
- 2026-09-11
- Estimated Expiration
- 2044-02-01
AI Technical Summary
[0004]目前的技术通过重复缓存部分高频特征数据或者通过修改训练算法的方式来提高本地特征数据的命中率，然而，重复缓特征数据缓存需要占据额外的内存空间，修改训练算法的方式则会带来精度和收敛速度上的负面影响
[0031] The beneficial effects of this invention are: when feature vector data is located remotely, the method of this invention does not transmit the feature data from the remote location to the local location, but instead transmits the model to the destination server. By introducing a model transfer method, most of the feature vector transmission is eliminated, thereby significantly shortening the training time and improving training efficiency. Furthermore, since the method of this invention does not change the randomly assigned training data sequence for each distributed model, it does not affect the training accuracy.
Smart Images
Figure CN118821857B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer science and artificial intelligence systems. It provides a distributed graph neural network training method and system based on model transfer to address the communication bottleneck caused by the need to read a large amount of remote training data across nodes during distributed graph neural network training. Background Technology
[0002] Graph Neural Networks (GNNs) are a type of neural network trained on graph data. The trained GNN models are often used for inference tasks such as vertex classification, edge prediction, and graph classification within the graph. Graph data consists of two parts: topological data and feature data. Topological data records the connections between vertices and edges in the graph, while feature data is the embedding vector representation of each vertex in the graph, such as... Figure 1 As shown.
[0003] Real-world graph data is often extremely large, making it impossible to store in the memory of a single server. Distributed graph neural networks are a widely used solution. Specifically, they solve the problem of insufficient single-machine memory by dividing the graph data into multiple parts and distributing them across the memory of different GPU servers (nodes). Figure 2 As shown. Because the size of the graph's topology is smaller than the size of the vertex feature embeddings, many studies use a small amount of memory to redundantly store parts of the topology to reduce topology data transfer during sampling (e.g., vertices 1 and 3 on server 0). During training, each node has a graph neural network model. The training process involves many repeated iterations, each consisting of three phases: sampling, gathering, and computation. First, in the sampling phase, each model randomly selects a subset of training points and obtains a training subgraph through k-hop neighbor sampling. Then, in the gathering phase, the feature vectors of each point in the subgraph are collected. If a point's feature vector is not available locally, it is transferred from a remote location. In distributed graph neural network training, cross-node feature data acquisition is a bottleneck. Finally, in the computation phase, model parameters are updated and synchronized. Figure 3This demonstrates the process of performing an iterative training of a distributed GNN on two servers. At the beginning of each iteration, each worker node is randomly assigned a non-overlapping mini-batch of training vertices. For example, suppose server 0 receives a batch of training vertices {6,3}, while server 1 receives a batch of training vertices {5,0}. Then, the training process on each node independently performs the steps of 2-hop neighbor sampling (①), feature collection (②), and computation (③). Finally, the parameter gradients from the different worker nodes are synchronized, and the model parameters are updated (④).
[0004] Current technologies improve the hit rate of local feature data by repeatedly caching some high-frequency feature data or by modifying the training algorithm. However, repeatedly caching feature data requires additional memory space, while modifying the training algorithm will have a negative impact on accuracy and convergence speed. Summary of the Invention
[0005] To address the communication bottleneck caused by the large amount of cross-node feature data transmission during distributed graph neural network training, existing research typically reduces cross-node feature data transmission by compromising training accuracy, which negatively impacts training accuracy. To reduce the amount of remote feature data transmission without affecting training accuracy, this invention proposes a distributed graph neural network training method and system based on model transfer.
[0006] The specific technical solution adopted in this invention is as follows:
[0007] A distributed graph neural network training method based on model transfer includes:
[0008] A graph G(V, E) is constructed from the original data, and the data is distributed and stored in a distributed cluster by edge cutting; the distributed cluster contains several servers; each server has an initialized graph neural network model;
[0009] Based on a distributed cluster, labeled vertex data is used to train an initialized graph neural network model. The training consists of multiple epochs, each containing multiple iterations. Each iteration specifically includes:
[0010] Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex;
[0011] The training vertices of a batch of each graph neural network model are grouped according to the corresponding master server and assigned to the corresponding master server.
[0012] After the redistribution, each training vertex generates a micromap by skipping neighbor sampling;
[0013] Each graph neural network model is migrated to the corresponding server after the vertices of the corresponding batch of training vertices are reassigned. The model is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server. Vertex features of micrographs not stored on the corresponding server are obtained remotely.
[0014] When all graph neural network models have completed the training of the last micrograph, the accumulated gradients among all graph neural network models will be synchronized, and finally the model parameters will be updated to complete a batch of iterative training.
[0015] Repeat the training process for multiple batches until the graph neural network model reaches the required number of iterations and cycles.
[0016] Furthermore, the training using the vertex features of the micromap stored on the corresponding server includes forward propagation and backward propagation.
[0017] Furthermore, different neural network models to be migrated to the same server are interleaved during migration and training.
[0018] Furthermore, the graph neural network model includes GCN, GraphSAGE, and GAT.
[0019] Furthermore, each graph neural network model is migrated sequentially to the corresponding server after the vertices of the corresponding batch of training vertices are reassigned, and the model is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server, as detailed below:
[0020] Set N time slices. In the t-th time slice, the graph neural network model is transferred to the d-th time slice. On the server, , This indicates the modulo operation, and the gradient is accumulated using the vertex features of the micromap stored on the corresponding server. If there are vertex features in the current micromap that are not on the server, they are obtained from the remote server.
[0021] A distributed graph neural network training system based on model transfer, used to implement the aforementioned distributed graph neural network training method based on model transfer, the system comprising:
[0022] A distributed cluster, consisting of several servers, each with an initialized graph neural network model;
[0023] The data processing module is used to construct a graph G(V,E) from the raw data and distribute the data in a distributed cluster by cutting edges.
[0024] The training module is used to train an initialized graph neural network model based on a distributed cluster and using labeled vertex data. Training consists of multiple epochs, each containing multiple iterations. Each iteration specifically includes:
[0025] Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex;
[0026] The training vertices of a batch of each graph neural network model are grouped according to the corresponding master server and assigned to the corresponding master server.
[0027] After the redistribution, each training vertex generates a micromap by skipping neighbor sampling;
[0028] Each graph neural network model is migrated to the corresponding server after the vertices of the corresponding batch of training vertices are reassigned. The model is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server. Vertex features of micrographs not stored on the corresponding server are obtained remotely.
[0029] When all graph neural network models have completed the training of the last micrograph, the accumulated gradients among all graph neural network models will be synchronized, and the model parameters will be updated to complete a small batch of iterative training.
[0030] Repeat the training process for multiple batches until the graph neural network model reaches the required number of iterations and cycles.
[0031] The beneficial effects of this invention are: when feature vector data is located remotely, the method of this invention does not transmit the feature data from the remote location to the local location, but instead transmits the model to the destination server. By introducing a model transfer method, most of the feature vector transmission is eliminated, thereby significantly shortening the training time and improving training efficiency. Furthermore, since the method of this invention does not change the randomly assigned training data sequence for each distributed model, it does not affect the training accuracy. Attached Figure Description
[0032] Figure 1 This is a schematic diagram of the graph data;
[0033] Figure 2 This is a schematic diagram of the splitting of the training graph data (assuming it is split across two servers).
[0034] Figure 3 This is a typical GNN distributed training flowchart;
[0035] Figure 4 This is a conceptual diagram illustrating sub-graphs and micro-graphs; Figure 4(a) shows a subgraph sampled from vertices trained on a training batch {6,3} using a conventional method (different textures represent feature vectors of corresponding points distributed on two different servers); Figure 4 (b) corresponds to the present invention splitting the subgraph into micro-graphs. Figure 6 Hewei Figure 3 ;
[0036] Figure 5 This is a schematic diagram of the GNN training process based on model transfer in the present invention.
[0037] Figure 6 This is a flowchart of the training process for a GNN model used to predict the category of joint procurement of goods. Detailed Implementation
[0038] This invention discovers that graph neural network models differ from traditional visual models (CNN) or natural language processing (NLP) deep learning models in that, during distributed training, the amount of data acquired from remote machines is far greater than the amount of data for the model parameters themselves. Therefore, this invention provides a distributed graph neural network training method based on model transfer. This method, when feature vector data is located remotely, does not transfer the feature data from the remote machine to the local machine, but instead transfers the model to the destination server. In simpler terms, the general idea of this invention is to "prevent feature data from running alongside model computation, and instead allow model computation to run alongside feature data." However, it should be noted that this invention does not completely eliminate the transfer of feature vectors across machines; it only eliminates most of the feature vector transfer by introducing a model transfer method. Specifically, it includes the following steps:
[0039] (1) Construct a graph G(V,E) for the original data and distribute the data in a distributed cluster by edge cutting; the distributed cluster contains several servers; each server has an initialized graph neural network model;
[0040] (2) Based on a distributed cluster, the initialized graph neural network model is trained using labeled vertex data. The training consists of many epochs, each of which includes multiple iterations. Each iteration specifically includes:
[0041] (A) Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex;
[0042] (B) The training vertices of a batch of each graph neural network model are grouped according to the corresponding master server and assigned to the corresponding master server;
[0043] (C) After reallocation, each training vertex generates a micrograph by hop neighbor sampling; the micrograph of this invention refers to a computational graph created using k-hop sampling for each training vertex v in a batch; Figure 4 (a) shows a subgraph sampled from vertices trained on a training batch {6,3} using a conventional method (different textures represent feature vectors of corresponding points distributed on two different servers); Figure 4 (b) corresponds to the present invention splitting the subgraph into micro-graphs. Figure 6 Hewei Figure 3 (For ease of description, the ID of the training vertex v is used to represent its corresponding micromap.)
[0044] (D) Each graph neural network model is reassigned to the corresponding server according to the location of the training vertices of the corresponding batch, and is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server. Vertex features of micrographs not stored on the corresponding server are obtained remotely.
[0045] In one specific implementation, different neural network models to be migrated to the same server are interleaved during migration and training. Specifically, N time slices are set, where N is preferably equal to the number of servers in the distributed cluster. In the t-th time slice, the d-th graph neural network model is migrated to the d-th time slice. On the server, , This indicates the modulo operation, and the gradient is accumulated using the vertex features of the micromap stored on the corresponding server. If there are vertex features in the current micromap that are not on the server, they are obtained from the remote server.
[0046] (E) When all graph neural network models have completed the training of the last micrograph, the accumulated gradients between all graph neural network models will be synchronized, and the model parameters will be updated to complete a small batch of training.
[0047] (3) Repeat the training in multiple batches until the graph neural network model reaches the number of iterations and cycles.
[0048] by Figure 3 An example of batch training is used to demonstrate the training process in each iteration using the method of this invention (see...). Figure 5 This facilitates a comparison of how the method of the present invention reduces the amount of feature vector data transmitted between servers. Specifically:
[0049] Initially, server 0 has model 0, and server 1 has model 1. Figure 3To maintain consistency, in step (A), each graph neural network model is randomly assigned a batch of training vertices, where model 0 is randomly assigned training vertices {6,3} and model 1 is randomly assigned training vertices {5,0}.
[0050] Subsequently, in step (B), the training vertices of a batch for each graph neural network model are grouped according to the corresponding master server and assigned to the corresponding master server (①): server 0 receives vertices 6 and 5 (this is because server 0 caches vertices 6 and 5, which has the advantage of considering data locality, and the same applies below), while server 1 receives vertices 0 and 3.
[0051] Step (C) involves each server generating a micromap (②) through "k-hop neighbor sampling," at which point a micromap is obtained on server 0. Figure 6 Hewei Figure 5 Micro-image 0 and micro-image 0 are obtained on server 1. Figure 3 .
[0052] Step (D) involves dividing the training process into two time slices (③). In time slice 0, model 0 and model 1 respectively use micro... Figure 6 Training is performed on micromap 0, including forward and backward propagation. During this process, the vectors of two vertices are transferred between machines (vertices 1 and 4). After backpropagation is completed on model 0 and model 1, since model 0 has not yet finished processing this batch of data (there is still micromap 0), the training continues. Figure 3 (Not trained), Model 1 hasn't even calculated all the batch data yet (there's still micro) Figure 5 (Without training), the gradients within the model accumulate in the model without updating the model parameters.
[0053] Before time slice 1 begins, the parameters of model 0 are transferred to server 1 along with the accumulated gradients, and the parameters of model 1 are transferred to server 0 along with the accumulated gradients. During time slice 1, model 1 and model 0 respectively use micro... Figure 5 Hewei Figure 3 During training, the feature vectors of four vertices are transmitted between machines.
[0054] Finally, following step (E), the gradients accumulated in the two models are synchronized, and the model parameters are updated (④).
[0055] In the default method ( Figure 3 The original method transmitted a total of 8 vertex features (0, 1, 2, 3, 4, 5, 6, and 7), while the method of this invention transmits 6 (points 1 and 4 in time slice 0, and points 1, 0, 4, and 6 in time slice 1). Although this invention introduces additional transmission of model parameters and accumulated gradients, these data volumes are much smaller compared to vertex feature vectors. Therefore, the overall transmission volume is reduced.
[0056] The effects of the present invention will be further illustrated below with experimental data from a specific embodiment.
[0057] Figure 6 The graph data in the image represents a joint procurement graph of goods, where each vertex corresponds to one product, and each edge represents two products being purchased together. Each vertex has a corresponding embedding vector (the embedding vector can encode vertex features such as product information, price, and manufacturer). Assume the task of the GNN is to predict the category of each product (e.g., electronic devices, household goods; categories are numbered 0-9 in the graph, and Lx represents the category x of a product). The categories of some products in the graph are known; the GNN will use these known categories as training data to train the GNN model. After training, the GNN can use the trained model to predict the categories of products with unknown information.
[0058] The training method for a commodity joint procurement category prediction GNN model based on the method of this invention includes the following steps:
[0059] (1) Construct a joint procurement graph G(V, E) for the original data and distribute the data in a distributed cluster by edge cutting; the distributed cluster contains N servers; each server has an initialized graph neural network model; wherein the server ID number is s, and the assigned GNN model number is d, where s ∈ [0, N-1], and d=s;
[0060] (2) Based on a distributed cluster, the initialized graph neural network model is trained using labeled vertex data (product categories). The training consists of many epochs, each of which includes multiple iterations. Each iteration specifically includes:
[0061] (A) Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex;
[0062] (B) Group all root vertices in the batch based on the master server ID. Then, assign the vertices in each group to the corresponding master server.
[0063] (C) After reallocation, each server We need to use k-hop sampling to generate a micromap for each vertex assigned to this server, and these micromaps will form some subsets. ,in This indicates which GNN model the vertex needs to be trained on.
[0064] (D) Begin training for N time slices. In time slices... ,Model Migrate to server and using a subset of micromaps Training is then performed. If any vertex features in the current micrograph are not on the local server, they will be retrieved from a remote server. To ensure that model parameters are updated only after all micrographs contained in the entire subgraph have been trained, gradients obtained after training a micrograph are accumulated.
[0065] (E) Update model parameters. When training of the last micrograph in the subgraph is complete, the accumulated gradients will be synchronized across all GPU servers. Finally, update the model parameters to complete one iteration of training.
[0066] (3) Repeat the training in multiple batches until the graph neural network model reaches the number of iterations to complete the training of all training vertices in one cycle, and repeat the training in multiple cycles until the number of cycles is reached.
[0067] Based on the OGB-Products graph dataset, the average training time per epoch for training GNN models using different training methods and architectures (GCN, GraphSAGE, GAT) is shown in Table 1. The hardware platform used in the test consisted of four servers, each with an A100 GPU card, interconnected by a 10Gbps network during service. The software platform used Ubuntu v18.04, PyTorch v1.10.1+cu113, DGL v0.4.1, and Python v3.9.0. Parameter configuration: 10 epochs were tested and the average training time per epoch was taken (the table shows the average training time per epoch), with a batch size of 8000. Experimental results show that the method of this invention achieves speedups of up to 2.3x and 2.1x compared to traditional DGL and the state-of-the-art method P3, respectively. Furthermore, since the method of this invention does not change the randomly assigned training data sequence for each distribution model—for example, model 0 is still trained on subroots with 6 and 3, just like the default method—it does not affect training accuracy.
[0068] Table 1. Average training time per epoch for different models on the OGB-Products graph dataset (in seconds)
[0069] GCN(16) 796.34 656.75 506.29 GraphSAGE(16) 757.65 593.86 404.03 GAT(16) 1077.84 732 491.47 GCN(32) 922.13 816.2 394.55 GraphSAGE(32) 781.79 687.51 403.58 GAT(32) 1071.47 895.775 489.44
[0070] The models are GCN, GraphSAGE, and GAT, with the numbers in parentheses representing the parameters of the hidden layers in the models.
[0071] Corresponding to the aforementioned embodiment of a distributed graph neural network training method based on model transfer, the present invention also provides an embodiment of a distributed graph neural network training system based on model transfer.
[0072] This invention provides a distributed graph neural network training system based on model transfer, used to implement the aforementioned distributed graph neural network training method based on model transfer. The system includes:
[0073] A distributed cluster, consisting of several servers, each with an initialized graph neural network model;
[0074] The data processing module is used to construct a graph G(V,E) from the raw data and distribute the data in a distributed cluster by cutting edges.
[0075] The training module is used to train an initialized graph neural network model based on a distributed cluster and using labeled vertex data. Training consists of multiple epochs, each containing multiple iterations. Each iteration specifically includes:
[0076] Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex;
[0077] The training vertices of a batch for each graph neural network model are grouped according to the master server and assigned to the corresponding master server;
[0078] After the redistribution, each training vertex generates a micromap by skipping neighbor sampling;
[0079] Each graph neural network model is migrated to the corresponding server after the vertices of the corresponding batch of training vertices are reassigned. The model is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server. Vertex features of micrographs not stored on the corresponding server are obtained remotely.
[0080] When all graph neural network models have completed the training of the last micrograph, the accumulated gradients among all graph neural network models will be synchronized, and the model parameters will be updated to complete a small batch of training.
[0081] Repeat the training process for multiple batches until the graph neural network model reaches the required number of iterations.
[0082] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to in the description of the method embodiments. The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without creative effort.
Claims
1. A distributed graph neural network training method based on model transfer, characterized in that, include: Construct a graph G(V,E) from the original data and distribute the data in a distributed cluster by cutting edges; The distributed cluster comprises several servers; each server has an initialized graph neural network model. Based on a distributed cluster, labeled vertex data is used to train an initialized graph neural network model. The training consists of multiple cycles, each cycle including multiple iterations of training. Each iteration specifically includes: Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex; The training vertices of a batch of each graph neural network model are grouped according to the corresponding master server and assigned to the corresponding master server. After the redistribution, each training vertex generates a micromap by skipping neighbor sampling; Each graph neural network model is migrated to the corresponding server after the vertices of the corresponding batch of training vertices are reassigned. The model is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server. Vertex features of micrographs not stored on the corresponding server are obtained remotely. When all graph neural network models have completed the training of the last micrograph, the accumulated gradients among all graph neural network models will be synchronized, and finally the model parameters will be updated to complete a batch of iterative training. Repeat the training process for multiple batches until the graph neural network model reaches the required number of iterations and cycles.
2. The method according to claim 1, characterized in that, The training process utilizes the vertex features of the micromap stored on the corresponding server, including forward propagation and backward propagation.
3. The method according to claim 1, characterized in that, Different neural network models to be migrated to the same server are interleaved during migration and training.
4. The method according to claim 1, characterized in that, The graph neural network models include GCN, GraphSAGE, and GAT.
5. The method according to claim 1, characterized in that, Each graph neural network model is migrated sequentially to the corresponding server after the vertices of its corresponding batch of training vertices are reassigned. The model is then trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server, as detailed below: Set N time slices. In the t-th time slice, the graph neural network model is transferred to the s-th time slice. new On the server, s new = (d+t)%N, where % represents the remainder, and uses the vertex features of the micromap stored on the corresponding server for training and gradient accumulation. If there are vertex features in the current micromap that are not on the server, they are obtained from the remote server.
6. A distributed graph neural network training system based on model transfer, characterized in that, For implementing the distributed graph neural network training method based on model transfer as described in any one of claims 1-5, the system comprises: A distributed cluster, consisting of several servers, each with an initialized graph neural network model; The data processing module is used to construct a graph G(V,E) from the raw data and distribute the data in a distributed cluster by cutting edges. The training module is used to train an initialized graph neural network model based on a distributed cluster and using labeled vertex data. Training consists of multiple epochs, each containing multiple iterations. Each iteration specifically includes: Each graph neural network model is randomly assigned a batch of training vertices, where the server where the features of each vertex are stored is the master server of that vertex; The training vertices of a batch of each graph neural network model are grouped according to the corresponding master server and assigned to the corresponding master server. After the redistribution, each training vertex generates a micromap by skipping neighbor sampling; Each graph neural network model is migrated to the corresponding server after the vertices of the corresponding batch of training vertices are reassigned. The model is trained and gradients are accumulated using the vertex features of the micrographs stored on the corresponding server. Vertex features of micrographs not stored on the corresponding server are obtained remotely. When all graph neural network models have completed the training of the last micrograph, the accumulated gradients among all graph neural network models will be synchronized, and finally the model parameters will be updated to complete a batch of iterative training. Repeat the training process for multiple batches until the graph neural network model reaches the required number of iterations and cycles.