Distributed graph neural network training method based on heterogeneous equipment

By evaluating and dynamic programming of equipment training capabilities, combined with graph cutting and sampling strategies, the problem of low resource utilization of heterogeneous equipment in distributed graph neural network training is solved, and efficient edge device resource utilization and model robustness are achieved.

CN119089968AActive Publication Date: 2024-12-06ANHUI PERCEPTION FUTURE ELECTRONIC TECH CO LTD

Patent Information

Application Number
CN202411042515.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-12-06
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The existing distributed graph neural network training method cannot effectively utilize heterogeneous devices, resulting in low resource utilization of edge devices, and the random sampling algorithm loses model accuracy, increasing communication overhead and parallel training time between machines.

Method used

By evaluating the training ability of each device, dynamically planning task allocation, using graph cutting algorithms and different sampling strategies, combined with fault-tolerant detection mechanism, we ensure that each device completes the training task within the same time, and improves the utilization rate of edge devices and the robust model of models. sex.

Benefits of technology

It breaks through the limitations of heterogeneous device combination design, improves the utilization rate of edge devices, reduces the risk of memory overflow, enhances network stability and reliability, and reduces the loss of training time and structural information.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089968A_ABST
    Figure CN119089968A_ABST
Patent Text Reader

Abstract

The invention discloses a distributed graph neural network training method based on heterogeneous equipment, and the method comprises the steps: carrying out the data processing of large-scale graph data, and obtaining a plurality of sub-graphs with balanced nodes and edges through a graph cutting algorithm; an optimal equipment placement scheme is obtained by adopting a dynamic planning algorithm, and the training task amount of each piece of equipment is planned according to the training capability of heterogeneous equipment, so that all pieces of equipment finish training tasks as simultaneously as possible in the same iteration, and the utilization rate of edge equipment is improved. And meanwhile, a fault-tolerant mechanism is added, so that the network can still keep certain functional performance when the network node has an error or is lost.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of heterogeneous device training, and in particular to a distributed graph neural network training method based on heterogeneous devices. Background Art

[0002] Graphs are composed of nodes and the relationships (edges) between nodes, and have strong expressive power. With the maturity of machine learning (ML) technology, in order to process data with complex data structures such as graphs, researchers have combined convolutional neural networks (GCN) and graph embedding algorithms (GEA) to propose the concept of graph neural networks (GNN). Graph neural networks enable machine learning algorithms to expand from feature extraction of Euclidean space data to non-Euclidean space, and have achieved the best results in a variety of graph-related tasks, such as node classification, graph classification, and link prediction.

[0003] With the advent of the big data era, the scale of graphs has further expanded with the trend of the times. The memory and training resources of a single machine are not enough to support the training of a large graph, which is a new challenge for graph neural networks. Unlike in the past, this time it is not based on full-batch training, but a training based on sampled subgraphs (mini-batch) is adopted. In the training of the full graph, all nodes will participate in the training and gradient update. The advantage is that there is no loss of accuracy, but the disadvantage is that the memory overhead is large and it cannot train large-scale graphs. The training of sampled subgraphs solves this disadvantage very well. Usually, a group of central node groups are randomly selected, and a group of subgraphs are generated by sampling the nodes. Through the calculation and gradient update of the graph neural network of the subgraph, this training model is called a distributed graph neural network (DGNN). The advantages of distributed graph neural networks are low memory overhead and easy parallelization, but the random sampling algorithm has a lot of randomness, which will lose some model accuracy. This distributed structure model also brings many new problems. The communication overhead between machines and the time of parallel training are important components of distributed graph neural network training.

[0004] The current mainstream distributed training methods are based on DistDGL (referenced paper: DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs) and use GPUs with the same training capabilities for training. The resources of many edge devices are not fully utilized, and training resources are very precious. Improving the utilization rate of edge devices is a major challenge in the current field of graph neural networks.

[0005] In actual operation, a system usually involves multiple devices with different training capabilities (i.e., heterogeneous devices). Therefore, the present invention proposes a distributed graph neural network training method based on heterogeneous devices. This method is not only suitable for application scenarios of heterogeneous devices, but also can effectively improve the utilization rate of edge devices. Summary of the invention

[0006] In view of the defects of the prior art, the present invention proposes a distributed graph neural network training method based on heterogeneous devices. The training task amount of each device is planned according to the training capabilities of the heterogeneous devices, so that all devices can complete the training tasks as simultaneously as possible in the same iteration, thereby improving the utilization rate of edge devices.

[0007] The present invention specifically comprises the following steps: Step A: Evaluate the training capability of each heterogeneous device participating in the training, obtain the parameters of the training capability of each device, and rank the training capability of each device, specifically: The central node sends the same subgraph training task to each device using the same training model and training parameters, and records the time for each device to perform forward propagation and backward propagation, thereby obtaining the training capacity of each device and sorting them to prepare for the subsequent maximum utilization of each training device; During the evaluation process, in order to reduce the evaluation error caused by the performance evaluation or testing of the equipment, each device is tested repeatedly, and after obtaining multiple test data, the average is calculated.

[0008] Step B: Process the large graph data and use the graph cutting algorithm to obtain several subgraphs with balanced nodes and edges (the subgraphs must be able to be trained on the device with the weakest training capability); divide the large graph data that will overflow the memory if trained directly into subgraphs suitable for edge device training, so that edge devices can participate in sharing the training tasks and improve the utilization rate of edge devices. At the same time, it also effectively breaks through the limitation that large graphs cannot be trained on edge devices.

[0009] Step C: Use a dynamic programming algorithm to derive an optimal device placement plan. Devices with strong training capabilities will receive more training tasks, and devices with weak training capabilities will receive fewer training tasks. Each subgraph is placed on the corresponding device for training based on the obtained plan strategy to ensure that each device completes the training tasks assigned by the algorithm as much as possible within the same time. The edge device shares the training tasks of the neural network without increasing the overall training time, further improving the utilization rate of the edge device.

[0010] Step D: Sample the training nodes of the subgraph to obtain a set of mini-batches, and load the mini-batches into the neural network for training.

[0011] During the sampling process, different sampling strategies are adopted according to the corresponding needs, such as neighbor sampling, partial neighbor sampling, random walk and other different sampling algorithms.

[0012] Step E: Use the corresponding graph neural network model to train the subgraph, and synchronize the parameters to the central node for parameter aggregation and storage. The corresponding parameters obtained after aggregation are distributed to each training device through broadcasting for a new round of training. The graph neural network generates intermediate embeddings by aggregating the edge neighbor information of the target vertex, and then uses the neural network to convert the aggregated features into output features and send them to the next layer as input features. After superimposing multiple graph neural network layers, the final embedding is obtained.

[0013] Specifically, the representation of graph neural network training vertices is as follows: ; ; in Vertex Aggregated neighbor information, For the vertex The connected set of vertices, Represents a vertex The number of neighbors of Vertex In the Layer embedding, represents a nonlinear activation function, For the The learnable parameters of the layer.

[0014] When each round of training is completed, The parameters and the number of training rounds are sent to the central node. When the central node receives the parameters of each device, it aggregates them to obtain new parameters. , and broadcast it to all training devices. All devices perform the next round of training after receiving the new parameters.

[0015] Step F: Enable the fault-tolerant detection mechanism to detect whether each round of training equipment can train normally and pass parameters normally until the training is completed. During the training process, the training may not be completed normally due to problems such as memory overflow, bit misalignment, network communication failure, etc. Check whether the training equipment can send the training parameters to the central node normally, and whether it can normally receive the parameters passed by the central node. If an abnormality occurs, return to step C to obtain a new placement strategy, and broadcast the previous round of abnormal parameters to all training devices for parameter synchronization and retraining, reducing the losses caused by the impact of the failure, further improving resource utilization, and also improving the robustness of the distributed model. Stop training when the graph data is trained to a convergence state.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. It breaks through the limitation that mainstream training methods can only use devices with the same training capabilities. Different placement strategies can be designed through the combination of heterogeneous devices to train graph neural networks. Since direct training with large graphs will lead to memory overflow, the graph slicing algorithm is used to reduce the memory occupied by the graph, and devices with weaker edge capabilities such as Raspberry Pi and development boards can be used for training. At the same time, copying the neighbor nodes of some edge points in the graph can reduce the loss rate of structural information, thereby greatly improving the utilization rate of edge devices.

[0017] 2. A fault-tolerant mechanism has been added. When a network node fails or is lost, the network can still maintain a certain functional performance. When an edge device cannot work properly, the dynamic programming algorithm is called again to plan a new placement strategy. At the same time, the central node retains the training parameters. When a training node is lost, other edge devices can still be used normally and efficiently, which improves the stability and reliability of the network. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a flowchart of the distributed graph neural network training method based on heterogeneous devices; Figure 2 It is a structural diagram of heterogeneous devices, a central training node and other heterogeneous training devices; Figure 3 It is a sampling and aggregation method for mini-batch of graph neural networks. DETAILED DESCRIPTION

[0019] The following examples are used to further illustrate the implementation steps and purposes of the present invention. The following examples are used to illustrate the present invention, but are not all examples of the present invention.

[0020] like Figure 1 As shown, a distributed graph neural network training method based on heterogeneous devices specifically includes the following steps: Step A: Evaluate the training capability of all equipment involved in the training, such as Figure 2 As shown, the central node sends an identical subgraph training task to each device using the same training model and training parameters, and records the time for each device to perform forward propagation and backward propagation, thereby obtaining the training capability of each device, and sorting them to obtain the training capability ranking of m devices, in preparation for obtaining the placement plan next.

[0021] Step B: Use the METIS algorithm to process graph data. The METIS algorithm partitions the graph with the smallest edge cuts. The algorithm assigns densely connected vertices to the same partition to reduce the number of edge cuts between partitions. After a part of the fixed points are divided into a partition, the neighbor nodes of these vertices are copied from the points in other partitions. The copied nodes are called HALO nodes, which ensure that the nodes in the partition do not need to communicate with the devices of the training neighbor partitions when sampling to reduce the training time. At this time, s subgraphs can be obtained.

[0022] Step C: At this point, there are m devices and s subgraphs, where the number of subgraphs trained on the last device is k. Define To define the minimum training time and placement plan for training s subgraphs on m devices, represents the time required for the mth device to train k images, where , train the k subgraphs on the last device, and train the first sk subgraphs on the m-1 devices. Based on the dynamic programming algorithm, the state transfer equation can be obtained: According to the state transfer equation, the shortest training time for s subgraphs on m devices can be obtained, and the placement plan P with the shortest training time can be obtained.

[0023] Step D: Use neighbor sampling to obtain a set of mini-batches for the training nodes in the subgraph, such as Figure 3 As shown, the mini-batch is loaded into the neural network for training.

[0024] Step E: Load the mini-batch sampled in step D into the input of the graph neural network to obtain the structural information, and then load the node information into the graph neural network for training. The graph neural network is trained according to ; ;in Vertex Aggregated neighbor information, For the vertex The connected set of vertices, Represents a vertex The number of neighbors of Vertex In the Layer embedding, represents a nonlinear activation function, For the The learnable parameters of the layer.

[0025] Since the accuracy of training neural networks on graph data is much higher than that of traditional multi-layer perception, the graph neural network in this embodiment generates intermediate embeddings by aggregating the edge neighbor information of the target vertex, and then uses the neural network to convert the aggregated features into output features, which are sent to the next layer as input features. After superimposing multiple graph neural network layers, the final embedding is obtained.

[0026] Step F: Enable the fault tolerance mechanism. The specific process is as follows: The parameters are sent to the central node together with the number of training rounds. The central node first saves these parameters and then aggregates the parameters of all devices to obtain a new It is sent to all training devices. After receiving the parameters, the device performs the next round of training. At the same time, it detects whether the device is abnormal. If the device is abnormal, it returns to step C to redetermine the placement plan. The number of devices becomes m-1, and dynamic planning is performed. The central node sends the previous round number of the abnormal round and the parameters to the new corresponding device, and continues training. The training ends when the training data converges.

Claims

1. A distributed graph neural network training method based on heterogeneous devices, characterized by: The specific steps include: Step A: Evaluate the training capability of each heterogeneous device participating in the training, obtain the parameters of the training capability of each device, and rank the training capability of each device, specifically: The central node sends the same subgraph training task to each device using the same training model and training parameters, and records the time for each device to perform forward propagation and backward propagation, thereby obtaining the training capacity of each device and sorting them to prepare for the subsequent maximum utilization of each training device; Step B: Process the large graph data and use the graph cutting algorithm to obtain several subgraphs with balanced nodes and edges; Step C: Use a dynamic programming algorithm to obtain an optimal device placement plan. Devices with strong training capabilities will receive more training tasks, while devices with weak training capabilities will receive fewer training tasks. According to the obtained plan strategy, each subgraph is placed on the corresponding device for training, so that each device can complete the training task assigned by the algorithm at the same time. Step D: Sample the training nodes of the subgraph to obtain a set of mini-batches, and load the mini-batches into the neural network for training; Step E: Use the corresponding graph neural network model to train the subgraph, and synchronize the parameters to the central node for parameter aggregation and storage. The corresponding parameters obtained after aggregation are distributed to each training device through broadcasting for a new round of training; the graph neural network generates intermediate embeddings by aggregating the edge neighbor information of the target vertex, and then uses the neural network to convert the aggregated features into output features and send them to the next layer as input features; after stacking multiple graph neural network layers, the final embedding is obtained; Specifically, the representation of graph neural network training vertices is as follows: ; ; in Vertex Aggregated neighbor information, For the vertex The connected set of vertices, Represents a vertex The number of neighbors of Vertex In the Layer embedding, represents a nonlinear activation function, For the Learnable parameters of the layer; When each round of training is completed, The parameters and the number of training rounds are sent to the central node. When the central node receives the parameters of each device, it aggregates them to obtain new parameters. , and broadcast it to all training devices. All devices will perform the next round of training after receiving the new parameters; Step F: Enable the fault-tolerant detection mechanism to detect whether the equipment can train normally and transmit parameters normally in each round of training until the training is completed; During the training process, if an exception occurs, return to step C to obtain a new optimal placement strategy, and broadcast the parameters of the previous round where the exception occurred to all training devices for parameter synchronization, retrain, reduce the loss caused by the failure, and stop training when the graph data is trained to a convergence state.

2. The distributed graph neural network training method based on heterogeneous devices according to claim 1, characterized in that: In the evaluation process of step A, in order to reduce the evaluation error generated when the equipment is evaluated or tested for performance, each equipment is tested repeatedly, and after obtaining multiple test data, the average is calculated.

3. The distributed graph neural network training method based on heterogeneous devices according to claim 1, characterized in that: The subgraph obtained in step B must be trainable on the device with the weakest training capability.

4. The distributed graph neural network training method based on heterogeneous devices according to claim 1, characterized in that: In the sampling process of step D, different sampling strategies are adopted according to corresponding requirements, and the sampling strategy is selected from neighbor sampling, partial neighbor sampling or random walk.

Citation Information

Patent Citations

  • Distributed parallel graph neural network training method and system

    CN117808080A

  • Neural network training method and system based on distributed continuous time dynamic graph

    CN117875391A

  • Distributed graph neural network training optimization method based on partial historical embedding

    CN118211631A

  • Method, apparatus and system for training graph neural network

    WO2022057310A1

Cited By

  • Graph division method and system for graph neural network reasoning in heterogeneous edge scene

    CN120371541A