A distributed full-graph training computation method, system, device, and storage medium for graph neural networks.
By constructing computational dependency graphs and communication dependency graphs, and using the Metis algorithm to partition graph data on multi-GPU devices, distributed full-graph training of graph neural networks was achieved. This solves the problem of limited training on large-scale graph datasets and improves training efficiency and hardware utilization.
Patent Information
- Application Number
- CN202410856927.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-28
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-06-28
AI Technical Summary
Existing graph neural network models are limited by the memory capacity of a single GPU when training on large-scale graph datasets, making effective training difficult. Furthermore, existing distributed training techniques have insufficient acceleration effect and applicability on large-scale graph datasets and large GPU clusters.
A distributed full-graph training computation method for graph neural networks is adopted. By obtaining the node attributes of the graph dataset, computation dependency graph and communication dependency graph are constructed. The Metis algorithm is used to divide them onto various GPU devices to construct local graphs and cross-partition graphs, thereby achieving overlap and zero redundancy between computation and communication and improving training efficiency.
It effectively improves the computational efficiency of distributed full-graph training of graph neural networks, shortens the training time of large-scale graph datasets, and enhances hardware utilization and training efficiency.
Smart Images

Figure CN118504609B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of graph neural network technology, and in particular to a distributed full-graph training computation method, system, device and storage medium for graph neural networks. Background Technology
[0002] As the scale of graph datasets used for training graph neural network models continues to expand, and the complexity of graph neural network models themselves continues to increase, the demand for hardware resources and time costs for training graph neural network models have increased significantly. In the training process of graph neural network models, each iteration relies on the structural information of the entire graph and the node feature data. By continuously updating the model parameters, the output becomes more accurate, such as improving the prediction accuracy of nodes or graph attributes.
[0003] However, current graph datasets have reached the scale of billions of nodes and edges, causing the GPU memory required for training graph neural network models to far exceed the memory capacity limit of a single GPU, making training tasks difficult. To address this challenge, training graph neural network models on large-scale graph datasets requires a distributed strategy, that is, multiple GPU devices working together to train a single model. This distributed approach can not only significantly expand the total amount of GPU memory, but also achieve parallelization of the training process, thereby effectively shortening the training cycle while handling larger-scale graph datasets. Although there are some mature distributed training techniques for graph neural network models, the training acceleration effect and applicability of these techniques still need to be improved when dealing with large-scale graph datasets and large GPU clusters.
[0004] Therefore, providing a distributed full-graph training computation method for graph neural networks that can improve the training efficiency and applicability of graph neural network models is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] To address the aforementioned technical problems, the present invention aims to provide a distributed full-graph training computation method, system, device, and storage medium for graph neural networks. It achieves an efficient asynchronous pipeline based on partial aggregation with overlapping computation and communication, as well as zero-redundancy computation and communication during training. This can effectively improve the computation efficiency of distributed full-graph training of graph neural networks and shorten the computation time for training graph neural network models on large-scale graph datasets. The first objective of this invention is to provide a distributed full-graph training computation method for graph neural networks; The technical solution provided by this invention is as follows: A distributed full-graph training computation method for graph neural networks includes the following steps: Obtain the node attributes in the graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; Based on the test set nodes and their neighboring nodes, construct a computation dependency graph and a communication dependency graph; The computation dependency graph and the communication dependency graph are divided into various GPU devices according to the Metis algorithm. On each GPU device, a working process is started to initialize model parameters and obtain graph partition data. Construct a local graph and a cross-partition graph based on the graph partition data and the computation dependency graph; The graph neural network is trained in a distributed full graph manner using the local graph, the cross-partition graph, and the communication dependency graph.
[0006] Preferably, obtaining the neighbor nodes and the root node based on the test set specifically includes: Obtain the nodes of the test set, use the nodes of the test set as the root node to expand outwards to obtain the corresponding neighbor nodes and edges; The root node and the generated neighbor nodes are used as the root node for the next iteration; The process of obtaining the neighbor nodes and the root node is iterated up to a preset number of times to construct a computation dependency graph and a communication dependency graph.
[0007] Preferably, the computation dependency graph and the communication dependency graph are partitioned onto various GPU devices according to the Metis algorithm, and a worker process is started on each GPU device to initialize model parameters and obtain graph partition data, specifically including: The computation dependency graph and the communication dependency graph are divided into multiple partitions using the Metis partitioning function, and the data of each partition is saved on each GPU device after the partitioning is completed. The graph neural network model is initialized based on the hyperparameters input from the command line, and graph partition data is read from each GPU device.
[0008] Preferably, the step of constructing the local graph and the cross-partition graph based on the graph partition data and the computation dependency graph specifically includes: Local nodes and local edges are determined based on the graph partitioning data and the computation dependency graph to construct the local graph; The cross-partition graph is constructed by using remote neighbor nodes that need to communicate based on their characteristics and edges that do not belong to the local graph.
[0009] Preferably, the step of performing distributed full-graph training of the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph specifically includes: Remote neighbor node feature communication is performed between various GPU devices through the communication dependency graph, and partial aggregation calculations of the corresponding local graph are performed in parallel on each GPU device at the start of the communication. After the communication is completed, perform the corresponding cross-partition graph partial aggregation calculation and perform feature update calculation for the nodes within the partition.
[0010] The second objective of this invention is to provide a distributed full-graph training computation system for graph neural networks; The technical solution provided by this invention is as follows: A distributed full-graph training computation system for graph neural networks includes: an acquisition module, a production module, a partitioning module, a construction module, and a training module; The acquisition module is used to acquire node attributes in the graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; The production module is used to construct a computation dependency graph and a communication dependency graph based on the test set nodes and their neighboring nodes. The partitioning module is used to partition the computation dependency graph and the communication dependency graph to various GPU devices according to the Metis algorithm, and start a working process on each GPU device to initialize model parameters and obtain graph partition data. The construction module is used to construct local graphs and cross-partition graphs based on the graph partitioning data and the computation dependency graph; The training module is used to perform distributed full-graph training on the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph.
[0011] The third objective of this invention is to provide an electronic device; The technical solution provided by this invention is as follows: An electronic device, comprising: At least one processor; and A memory communicatively connected to the at least one processor, the memory storing a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method steps of the distributed full-graph training computation method for graph neural networks.
[0012] The fourth objective of this invention is to provide a computer-readable storage medium; The technical solution provided by this invention is as follows: A computer-readable storage medium for storing a computer program for causing a computer to execute the steps of a distributed full-graph training computation method for graph neural networks.
[0013] This invention provides a distributed full-graph training computation method for graph neural networks, comprising the following steps: obtaining node attributes in a graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; constructing a computation dependency graph and a communication dependency graph based on the test set nodes and their neighboring nodes; partitioning the computation dependency graph and the communication dependency graph onto various GPU devices according to the Metis algorithm, starting a worker process on each GPU device to initialize model parameters and obtain graph partition data; constructing a local graph and a cross-partition graph based on the graph partition data and the computation dependency graph; and performing distributed full-graph training of the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph. This method implements an efficient asynchronous pipeline based on partial aggregation with overlapping computation and communication, and zero-redundancy computation and communication during training, which can effectively improve the distributed full-graph training computation efficiency of graph neural networks and shorten the computation time for training graph neural network models on large-scale graph datasets.
[0014] The present invention also provides a distributed full-graph training computation system for graph neural networks. Since this system and the distributed full-graph training computation method for graph neural networks solve the same technical problem and belong to the same technical concept, they should have the same beneficial effects, and will not be described in detail here. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 A flowchart illustrating a distributed full-graph training computation method for graph neural networks provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure for pruning computational dependency graphs and communication dependency graphs provided in an embodiment of the present invention; Figure 3 A schematic diagram illustrating graph partitioning and computational dependency graph decomposition provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a distributed full-graph training computation system for graph neural networks provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0017] To enable those skilled in the art to better understand the technical solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0018] It should be noted that when a component is referred to as being "fixed to" or "set on" another component, it can be directly on or indirectly set on the other component; when a component is referred to as being "connected to" another component, it can be directly connected to or indirectly connected to the other component.
[0019] It should be understood that the terms "length", "width", "upper", "lower", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this application.
[0020] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "a plurality of" or "several" means two or more, unless otherwise explicitly specified.
[0021] It should be noted that the structures, proportions, sizes, etc., shown in the accompanying drawings of this specification are only for the purpose of assisting those skilled in the art in understanding and reading the content disclosed in the specification, and are not intended to limit the conditions under which this application can be implemented. Therefore, they have no substantial technical significance. Any modifications to the structure, changes in the proportions, or adjustments to the size should still fall within the scope of the technical content disclosed in this application, provided that they do not affect the effects and purposes that this application can produce.
[0022] like Figure 1 As shown, this embodiment of the invention provides a distributed full-graph training computation method for graph neural networks, comprising the following steps: S1. Obtain the node attributes in the graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; In step S1, the program reads a file containing graph adjacency matrix information from the disk and constructs a complete graph structure accordingly; it also reads a file containing node attributes from the disk and generates node attributes for the nodes in the graph, where the node attributes include: training set nodes, validation set nodes, and test set nodes; assuming the graph neural network model has a total of L layers.
[0023] S2. Construct a computation dependency graph and a communication dependency graph based on the test set nodes and their neighboring nodes; In step S2, neighboring nodes and the root node are selected from the nodes whose node attributes belong to the test set. These are used as the node computation dependency graph and communication dependency graph of the Lth layer of the graph neural network model assumed in step S1. Before training begins, the computation and communication dependency graphs of the original full graph training are pruned to remove unnecessary node neighbor dependencies and construct a zero-redundancy computation dependency graph and communication dependency graph. This avoids redundant node computation and neighbor feature communication, effectively improving the overall efficiency of distributed full graph training of graph neural networks on large-scale graphs.
[0024] S3. Based on the Metis algorithm, the computation dependency graph and the communication dependency graph are divided into various GPU devices, and a working process is started on each GPU device to initialize model parameters and obtain graph partition data; In step S3, since it runs on multiple servers and multiple GPU devices, the computation dependency graph and communication dependency graph need to be divided into various GPU devices using the Metis algorithm. Then, a worker process is started on each GPU device to initialize the model parameters and obtain the graph partition data.
[0025] S4. Construct a local graph and a cross-partition graph based on the graph partitioning data and the computation dependency graph; S5. Perform distributed full-graph training on the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph.
[0026] In steps S4 to S5, the node neighbor dependency relationship is first divided into local graph and cross-partition graph. Then, the graph neural network is trained in a distributed full graph using the local graph, cross-partition graph and communication dependency graph. This achieves the overlap of computation and communication during the training iteration process, greatly hides the node aggregation computation overhead during the training process, and effectively improves the hardware utilization and the distributed full graph training computation efficiency of the graph neural network.
[0027] Preferably, the step of constructing a computation dependency graph and a communication dependency graph based on the test set nodes and their neighboring nodes specifically includes: Obtain the nodes of the test set, use the nodes of the test set as the root node to expand outwards to obtain the corresponding neighbor nodes and edges; The root node and the generated neighbor nodes are used as the root node for the next iteration; The process of obtaining the neighbor nodes and the root node is iterated up to a preset number of times to form a computational dependency graph and a communication dependency graph.
[0028] In practical applications, nodes belonging to the test set are used as root nodes to expand outwards, selecting corresponding neighbor nodes and edges to form the computational dependency graph and communication dependency graph of the Lth layer of the graph neural network model. The root node of this iteration and the generated neighbor nodes are used as the root node of the next iteration, and the above process continues until it is repeated L times, generating L computational dependency graphs and communication dependency graphs. These are then used for computation and communication from the Lth layer to the 1st layer of the graph neural network model, respectively. Figure 2 As shown, Figure a represents the complete graph structure; Figure b represents the original dependency graph; and Figure c represents the pruned dependency graph, where nodes 1, 2, and 3 are training set nodes, and the rest are ordinary nodes. In the dependency graphs of Figures b and c, nodes represent node computation, and arrows represent neighbor feature communication. In Figure c, the lines marked with red dashed lines represent redundant feature communication removed during pruning, and the shaded green nodes represent redundant node computation removed during pruning. Before training begins, the computation and communication dependency graph of the original full graph is pruned to remove unnecessary node neighbor dependencies and construct a zero-redundancy computation and communication dependency graph, avoiding redundant node computation and neighbor feature communication, effectively improving the overall efficiency of distributed full graph training of graph neural networks on large-scale graphs.
[0029] Preferably, the computation dependency graph and the communication dependency graph are partitioned onto various GPU devices according to the Metis algorithm, and a worker process is started on each GPU device to initialize model parameters and obtain graph partition data, specifically including: The computation dependency graph and the communication dependency graph are divided into multiple partitions using the Metis partitioning function, and the data of each partition is saved on each GPU device after the partitioning is completed. The graph neural network model is initialized based on the hyperparameters input from the command line, and graph partition data is read from each GPU device.
[0030] In practical applications, the Metis partitioning function in the Metis algorithm provided by Python's DGL library is used to divide the computational dependency graph and the communication dependency graph into multiple partitions according to the processes of graph coarsening, initial partitioning, and fine-tuning. After partitioning, the data of each partition is saved on each GPU device. The graph neural network model is initialized according to the hyperparameters input in the command line, and the graph partition data is read from each GPU device. The Metis algorithm in this embodiment is a hierarchical partitioning algorithm. The core idea is to continuously reduce the size of the original graph given the original graph structure, and then partition the reduced graph structure to a certain extent. Finally, the partitioned small graphs are restored to the original graph structure to ensure the balance of each subgraph.
[0031] Preferably, the step of constructing the local graph and the cross-partition graph based on the graph partition data and the computation dependency graph specifically includes: Local nodes and local edges are determined based on the graph partitioning data and the computation dependency graph to construct the local graph; The cross-partition graph is constructed by using remote neighbor nodes that need to communicate based on their characteristics and edges that do not belong to the local graph.
[0032] In practical application, each working process utilizes graph partitioning data and the L complete computational dependency graphs and communication dependency graphs obtained from pruning in step S2. Based on the partition to which the nodes in the computational dependency graphs and communication dependency graphs belong, it generates a computational dependency graph belonging to that working process. Furthermore, it decomposes the original computational dependency graph into local graphs and cross-partition graphs based on whether the edges in the computational dependency graph belong to the current partition. For example... Figure 3 As shown, for a computation dependency graph of partition ①, nodes 0, 1, and 2 are local nodes, and the edges between them are local edges, which are used to construct the local graph; while nodes 3, 5, and 6 are remote neighbor nodes that need to perform feature communication, and the edges between them and the local nodes do not belong to the current partition, and are used to construct the cross-partition graph.
[0033] Preferably, the step of performing distributed full-graph training of the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph specifically includes: Remote neighbor node feature communication is performed between various GPU devices through the communication dependency graph, and partial aggregation calculations of the corresponding local graph are performed in parallel on each GPU device at the start of the communication. After the communication is completed, perform the corresponding cross-partition graph partial aggregation calculation and perform feature update calculation for the nodes within the partition.
[0034] In practical applications, in each iteration, the distributed full-graph training computation of the graph neural network performs L rounds of communication and computation, corresponding to the L layers of the graph neural network. The process of each round of communication and computation is as follows: First, based on the corresponding communication dependency graph of that layer, remote neighbor node feature communication is performed between GPU devices. At the same time as the communication begins, partial aggregation computation of the corresponding local graph is performed in parallel on each GPU device. The aggregation computation collects the feature information of each node's neighbor nodes and processes it through a certain aggregation function to generate a feature vector representing the local neighborhood information of the node. Common aggregation functions include sum, mean, and max. Partial aggregation computation means that each node in the local graph only performs aggregation computation on its local neighbor nodes, that is, neighbor nodes in the same partition. After the communication is completed, the corresponding cross-partition partial aggregation computation is performed, which means that each node in the cross-partition graph performs aggregation computation on its remote neighbor nodes, that is, neighbor nodes in different partitions. Finally, feature update calculations are performed on the nodes within the partition. This involves combining the aggregated information of each node with its own features and updating the node representation using an update function, which is typically a neural network layer. Through these steps, redundant computations and communication can be eliminated, allowing for overlap in computation and communication during training, improving hardware utilization, and maximizing training efficiency.
[0035] like Figure 4 As shown, this embodiment of the invention also provides a distributed full-graph training computation system for graph neural networks, including: an acquisition module, a first construction module, a partitioning module, a second construction module, and a training module; The acquisition module is used to acquire node attributes in the graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; The first construction module is used to construct a computation dependency graph and a communication dependency graph based on the test set nodes and their neighboring nodes; The partitioning module is used to partition the computation dependency graph and the communication dependency graph to various GPU devices according to the Metis algorithm, and start a working process on each GPU device to initialize model parameters and obtain graph partition data. The second construction module is used to construct a local graph and a cross-partition graph based on the graph partitioning data and the computation dependency graph; The training module is used to perform distributed full-graph training on the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph.
[0036] In practical application, an acquisition module, a first construction module, a partitioning module, a second construction module, and a training module are set up. The first module is connected to the acquisition module and the partitioning module. The partitioning module is connected to the second construction module. The second construction module is connected to the training module. The acquisition module sends the node attributes from the acquired graph dataset to the production module. The first construction module constructs computational dependency graphs and communication dependency graphs based on the test set nodes and their neighbor nodes, and then sends the computational dependency graphs and communication dependency graphs to the partitioning module. The partitioning module partitions the computational dependency graphs and communication dependency graphs onto various GPU devices according to the Metis algorithm, starts a working process on each GPU device to initialize model parameters, and acquires graph partitions. After obtaining the graph partition data, the graph partition data is sent to the construction module. The construction module then constructs the local graph and the cross-partition graph based on the graph partition data and the computation dependency graph, and sends the local graph and the cross-partition graph to the training module. The training module then performs distributed full-graph training on the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph. This system achieves an efficient asynchronous pipeline with overlapping computation and communication based on partial aggregation through the cooperation of the acquisition module, the first construction module, the partitioning module, the second construction module, and the training module. It also achieves zero-redundancy computation and communication during the training process, which can effectively improve the computational efficiency of distributed full-graph training of graph neural networks and shorten the computation time for training graph neural network models on large-scale graph datasets.
[0037] Furthermore, embodiments of this application also disclose an electronic device, Figure 5 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.
[0038] Figure 5 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the distributed full-graph training computation method for graph neural networks disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.
[0039] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.
[0040] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222 and data 223, etc., and the storage method can be temporary storage or permanent storage.
[0041] The operating system 221 manages and controls the various hardware devices and computer programs 222 on the electronic device 20 to enable the processor 21 to perform operations and processing on the data 223 in the memory 22. It can be Windows Server, Netware, Unix, Linux, etc. The computer program 222, in addition to including a computer program capable of performing the distributed full-graph training computation method for graph neural networks executed by the electronic device 20 as disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 223 may include data received by the distributed full-graph training computation device for graph neural networks from external devices, as well as data collected by its own input / output interface 25.
[0042] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0043] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned distributed full-graph training computation method for graph neural networks. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.
[0044] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.
[0045] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0046] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0047] If a flowchart is used in this application, it is used to illustrate the operations performed by the system according to embodiments of this application. It should be understood that the preceding or following operations are not necessarily performed in exact order. Instead, the steps can be processed in reverse order or simultaneously. Furthermore, other operations can be added to these processes, or one or more steps can be removed from them.
[0048] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A distributed full graph training computing method of a graph neural network, characterized in that, Includes the following steps: Obtain the node attributes in the graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; Based on the test set nodes and their neighboring nodes, construct the computation dependency graph and the communication dependency graph; The computation dependency graph and the communication dependency graph are divided into various GPU devices according to the Metis algorithm. On each GPU device, a working process is started to initialize model parameters and obtain graph partition data. Construct a local graph and a cross-partition graph based on the graph partition data and the computation dependency graph; Distributed full-graph training of the graph neural network is performed using the local graph, the cross-partition graph, and the communication dependency graph. The construction of computation dependency graphs and communication dependency graphs based on test set nodes and their neighboring nodes specifically includes: Obtain the nodes of the test set, use the nodes of the test set as the root node to expand outwards to obtain the corresponding neighbor nodes and edges; The root node and the generated neighbor nodes are used as the root node for the next iteration; The process of obtaining the neighbor nodes and the root node is iterated up to a preset number of times to construct a computation dependency graph and a communication dependency graph; The distributed full-graph training of the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph specifically includes: Remote neighbor node feature communication is performed between various GPU devices through the communication dependency graph, and partial aggregation calculations of the corresponding local graph are performed in parallel on each GPU device at the start of the communication. After the communication is completed, perform the corresponding cross-partition graph partial aggregation calculation and perform feature update calculation for the nodes within that partition.
2. The distributed full-graph training computing method of the graph neural network according to claim 1, characterized in that, The computation dependency graph and the communication dependency graph are partitioned onto various GPU devices according to the Metis algorithm. On each GPU device, a worker process is started to initialize model parameters and obtain graph partition data. Specifically, this includes: The graph is divided into multiple partitions using the Metis partitioning function, and the data of each partition is saved on each GPU device after the partitioning is completed. The graph neural network model is initialized based on the hyperparameters input from the command line, and graph partition data is read from each GPU device.
3. The distributed full-graph training computation method for graph neural networks according to claim 1, characterized in that, The construction of local and cross-partition graphs based on the graph partitioning data and the computation dependency graph specifically includes: Local nodes and local edges are determined based on the graph partitioning data and the computation dependency graph to construct the local graph; The cross-partition graph is constructed by using remote neighbor nodes that need to communicate based on their characteristics and edges that do not belong to the local graph.
4. A distributed full-graph training computation system for graph neural networks, characterized in that, include: The module includes an acquisition module, a first building module, a partitioning module, a second building module, and a training module. The acquisition module is used to acquire node attributes in the graph dataset, wherein the node attributes include: training set nodes, validation set nodes, and test set nodes; The first construction module is used to construct a computation dependency graph and a communication dependency graph based on the test set nodes and their neighboring nodes; The partitioning module is used to partition the computation dependency graph and the communication dependency graph to various GPU devices according to the Metis algorithm, and start a working process on each GPU device to initialize model parameters and obtain graph partition data. The second construction module is used to construct a local graph and a cross-partition graph based on the graph partitioning data and the computation dependency graph; The training module is used to perform distributed full-graph training on the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph. When the first construction module performs the step of constructing the computation dependency graph and communication dependency graph based on the test set nodes and their neighboring nodes, it is specifically used for: Obtain the nodes of the test set, use the nodes of the test set as the root node to expand outwards to obtain the corresponding neighbor nodes and edges; The root node and the generated neighbor nodes are used as the root node for the next iteration; The process of obtaining the neighbor nodes and the root node is iterated up to a preset number of times to construct a computation dependency graph and a communication dependency graph; When the training module performs distributed full-graph training of the graph neural network using the local graph, the cross-partition graph, and the communication dependency graph, it is specifically used for: Remote neighbor node feature communication is performed between various GPU devices through the communication dependency graph, and partial aggregation calculations of the corresponding local graph are performed in parallel on each GPU device at the start of the communication. After the communication is completed, perform the corresponding cross-partition graph partial aggregation calculation and perform feature update calculation for the nodes within that partition.
5. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor, the memory storing a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.
6. A computer-readable storage medium, characterized in that, The storage medium is used to store a computer program that causes a computer to perform the method according to any one of claims 1 to 3.
Citation Information
Patent Citations
Execution method and device of graph neural network task, electronic equipment and storage medium
CN117807246A
Distributed graph neural network training method based on incremental aggregation strategy
CN118095343A
