A method for optimizing the computational efficiency of distributed graph neural networks

Abstract

The present invention discloses an optimization method based on the computing efficiency of distributed graph neural networks, uses Deep Graph Library to provide basic support for graph data processing and the construction and training of graph neural networks, and uses GCCL to realize graph data communication optimization in a distributed environment to meet the experimental needs of graph neural network researchers.

Description

Technical Field

[0001] The present invention belongs to the field of graph data processing technology, and specifically relates to a method for optimizing the computing efficiency of a distributed graph neural network. Background Art

[0002] Currently, the main research direction in the field of graph neural networks is to develop graph neural network models with better information extraction capabilities and versatility. However, these performance requirements often require more complex model structures and an increasing amount of graph data to be processed. This significantly increases the GPU memory and training time required to train graph neural network models. This is undoubtedly a huge challenge for most graph neural network researchers. They either have to spend a lot of money to expand existing equipment or optimize the computing process at the expense of reduced model performance or increased model training time. Therefore, there is an urgent need for a method to alleviate the memory and time requirements for graph neural network model training in a distributed environment.

[0003] Deep Graph Library is a third-party library based on PyTorch and developed specifically for the implementation and training of graph neural networks. It encapsulates many commonly used graph neural networks and provides users with convenient interfaces for importing graph data, building graph neural network models, and training graph neural networks. Based on Deep Graph Library, it can meet the needs of some graph neural network researchers and facilitate users' learning and getting started. Therefore, it can adapt to the needs of this method for good support and expansion of graph data and graph neural network models, and is suitable as the main framework of this method.

[0004] Therefore, how to solve the problems of high memory requirements and long time required for graph neural network training through distributed communication optimization and graph neural network computing process optimization; and proposing a more convenient and fast graph neural network training solution for graph neural network researchers are issues that technical personnel in this field urgently need to solve. Summary of the Invention

[0005] In view of this, the present invention provides a method for optimizing the computing efficiency of a distributed graph neural network based on Deep Graph Library and GCCL, which can improve computing efficiency.

[0006] A method for optimizing the computational efficiency of distributed graph neural networks, including a data communication phase, a computational graph optimization phase, and a graph neural network training phase;

[0007] The data communication stage includes:

[0008] S11. The Deep Graph Library module submits the local graph data to the GCCL module;

[0009] S12, the GCCL module maps the graph data according to the graph partitions and sends the graph data located at other nodes to the GCCL modules of the corresponding nodes;

[0010] S13, the GCCL module receives graph data sent by other nodes;

[0011] S14. The GCCL module updates the local graph data according to the received graph data, and returns the updated graph data to the Deep Graph Library module;

[0012] The computation graph optimization phase includes:

[0013] S21, the Deep Graph Library module generates a computational graph based on the local graph data structure and submits the computational graph to the computational graph optimization module;

[0014] S22, the computation graph optimization module optimizes the computation graph according to the designed optimization process, and returns the optimized computation graph to the Deep Graph Library module;

[0015] The graph neural network training phase includes:

[0016] S31, the Deep Graph Library module generates corresponding GPU operators according to the optimized calculation graph and submits them to the GPU for calculation;

[0017] S32. After the GPU completes the calculation corresponding to the graph neural network training, it returns the calculation result to the Deep GraphLibrary module and updates the local graph data.

[0018] Preferably, when the GCCL module communicates with the GCCL module of another node, the GCCL module uses a shortest path spanning tree algorithm to obtain an optimal communication strategy based on the GPU topology in the distributed system, and the GCCL module communicates according to the communication strategy.

[0019] Preferably, in step 21, the computational graph generated by the Deep Graph Library according to the graph neural network model is optimized and then the corresponding operator is generated.

[0020] Preferably, the optimization of the computation graph includes operator fusion optimization, specifically including:

[0021] Operator fusion optimization classifies the computing operations according to the computing graph provided by the Deep Graph Library, and fuses several set computing nodes to generate a new computing operation, thereby optimizing the original computing graph.

[0022] Preferably, the optimization of the computation graph includes GPU kernel optimization, specifically including:

[0023] Based on the thread scheduling characteristics of the GPU and the computational characteristics of graph neural networks, GPU kernel optimization converts the optimized computational graph into operator instructions that the GPU can receive, and assigns a feature-adaptive thread group to each vertex in the graph for corresponding calculations, thereby achieving feature-level and vertex-level parallelism during training.

[0024] Preferably, graph data is stored in a compressed sparse row format.

[0025] Preferably, the compressed sparse rows extract the non-zero elements based on the adjacency matrix of the graph data, generate three arrays of vertex value V, row index and column index to establish an index for each vertex feature, thereby realizing the compression of the sparse adjacency matrix of the graph data.

[0026] Preferably, when generating compressed sparse rows, the vertices in the graph are arranged in descending order of in-degree, so that when reading vertices, vertices with larger degrees are started for calculation first and vertices with similar degrees are assigned to the same GPU block. After reversing the graph, the corresponding inverse compressed sparse rows are generated in descending order of vertex out-degree.

[0027] Preferably, the graph neural network model is GCN or GAT.

[0028] The present invention has the following beneficial effects:

[0029] The present invention discloses an optimization method based on the computing efficiency of distributed graph neural networks, uses DeepGraph Library to provide basic support for graph data processing and the construction and training of graph neural networks, and uses GCCL to optimize graph data communication in a distributed environment to meet the experimental needs of graph neural network researchers. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a schematic diagram of the overall process structure of the method provided by the present invention.

[0031] Figure 2 This is a flow chart of the data communication phase in the method provided by the present invention.

[0032] Figure 3 This is a flow chart of the computational graph optimization phase in the method provided by the present invention.

[0033] Figure 4 This is an example diagram of the computational graph optimization in the method provided by the present invention.

[0034] Figure 5This is a diagram of the finite state automaton used in the computational graph optimization stage of the method provided by the present invention. DETAILED DESCRIPTION

[0035] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0036] The present invention provides a method for optimizing the computing efficiency of a distributed graph neural network based on Deep Graph Library and GCCL, which includes a data communication stage, a computational graph optimization stage, and a graph neural network training stage.

[0037] The data communication stage includes:

[0038] S11. The Deep Graph Library module submits the local graph data to the GCCL module;

[0039] S12, the GCCL module maps the graph data according to the graph partitions and sends the graph data located at other nodes to the GCCL modules of the corresponding nodes;

[0040] S13, the GCCL module receives graph data sent by other nodes;

[0041] S14. The GCCL module updates the local graph data according to the received graph data, and returns the updated graph data to the Deep Graph Library module;

[0042] The computation graph optimization phase includes:

[0043] S21, the Deep Graph Library module generates a computational graph based on the local graph data structure and submits the computational graph to the computational graph optimization module;

[0044] S22, the computation graph optimization module optimizes the computation graph according to the designed optimization process, and returns the optimized computation graph to the Deep Graph Library module;

[0045] The graph neural network training phase includes:

[0046] S31, the Deep Graph Library module generates corresponding GPU operators according to the optimized calculation graph and submits them to the GPU for calculation;

[0047] S32. After the GPU completes the calculation corresponding to the graph neural network training, it returns the calculation result to the Deep GraphLibrary module and updates the local graph data.

[0048] To further optimize the above technical solution, the graph data communication in steps S12 and S13 does not adopt direct point-to-point communication between GPUs. Instead, the shortest path spanning tree algorithm is used based on the GPU topology in the distributed system to determine the optimal communication strategy before the graph neural network training begins.

[0049] To further optimize the above technical solution, the computational graph generated by the Deep Graph Library based on the graph neural network model is optimized before generating the corresponding operators, ensuring that the memory space used during the graph neural network training process is reduced and the computing efficiency is improved.

[0050] In order to further optimize the above technical solutions, the optimization of the computational graph is divided into operator fusion optimization and GPU kernel optimization;

[0051] Operator fusion optimization classifies the computational operations according to the computational graph provided by the Deep Graph Library, and fuses several appropriate computational nodes through a designed algorithm to generate a new computational operation, thereby optimizing the original computational graph.

[0052] Based on the thread scheduling characteristics of the GPU and the computational characteristics of graph neural networks, GPU kernel optimization converts the optimized computational graph into operator instructions that the GPU can receive, and assigns a feature-adaptive thread group to each vertex in the graph for corresponding calculations, thereby achieving feature-level and vertex-level parallelism during training.

[0053] To further optimize the above technical solution, graph data is stored in a compressed sparse row format;

[0054] Based on the adjacency matrix of the graph data, the compressed sparse row extracts the non-zero elements therein and generates three arrays of vertex value V, row index and column index to establish the index for each vertex feature, thereby compressing the sparse adjacency matrix of the graph data and greatly reducing the space required to store the graph matrix. At the same time, in order to balance the load within the subsequent GPU, when generating compressed sparse rows, the vertices in the graph are arranged in descending order of in-degree. This arrangement enables the vertices with larger degrees to start calculations first when reading vertices, and vertices with similar degrees are allocated to the same GPU block as much as possible, so that the time required to complete the calculation of vertices in the same block is similar, thereby improving GPU utilization. In addition, in order to meet the feature index requirements of the backpropagation process, after the graph is reversed, the corresponding inverse compressed sparse rows are generated in descending order of the vertex out-degree.

[0055] To further optimize the above technical solution, the supported graph neural network models include any graph neural network model supported by Deep Graph Library, including GCN and GAT.

[0056] This embodiment is a method for optimizing the computing efficiency of a distributed graph neural network based on Deep Graph Library and GCCL. The embodiment includes three entities: a Deep Graph Library client, a GCCL client, and a GPU.

[0057] 1. The Deep Graph Library client in this embodiment is no different from the ordinary Deep Graph Library client in terms of building and training the graph neural network model. However, there are some differences in initialization, graph data acquisition operations, and implementation of the computation graph:

[0058] (1) The Deep Graph Library client in the embodiment needs to invoke the GCCL client by calling the API provided by GCCL during initialization, and establish connections with other computing nodes in the distributed system through the GCCL client, where all computing nodes are required to be at least in the same local area network.

[0059] (2) The graph data used by the Deep Graph Library client in the embodiment is not directly obtained locally using the API provided by the DeepGraph Library. Instead, the graph data is obtained from the GCCL client after being partitioned and mapped by the GCCL client. Therefore, compared with the ordinary Deep Graph Library client, the DeepGraph Library client in the embodiment adopts a single-point training mode and does not need to deal with the partitioning problem of the graph data.

[0060] (3) The Deep Graph Library client in the example modifies the generation part of the computation graph and adds a computation graph optimization module based on the original extracted computation graph. By optimizing the computation graph structure and operator implementation, the computation efficiency is improved and the memory requirement during the computation process is reduced.

[0061] 2. The GCCL client in the embodiment is implemented by the GCCL library, and its main functions include:

[0062] (1) During the initialization phase of the GCCL client, it establishes communication with other GCCL clients on all computing nodes in the distributed system and plans the communication path between every two computing nodes using the shortest spanning tree algorithm.

[0063] (2) In the initialization phase of training, the original graph data is partitioned and mapped, and the graph data is distributed to all computing nodes in the distributed system, and the local computing nodes are provided with the graph partition data to which they belong.

[0064] (3) During the training phase, after each training cycle, the updated gradients of the graph neural network parameters calculated by the GPU and the new feature vectors of each vertex are synchronized and updated with other computing nodes to ensure the consistency of the graph neural network model and data in the distributed system.

[0065] 3. In principle, the GPU in the embodiment can be any GPU that supports CUDA version 10.1. The present invention uses Quadro RTX 6000.

[0066] The relationships between the above entities and data exchange are as follows Figure 1 As shown, the overall process details of the system will be explained later based on the data communication stage, computational graph optimization stage, and graph neural network training stage.

[0067] After the initialization of each entity is completed, the overall process of the data communication phase is as follows Figure 2 As shown, the specific implementation method will be introduced in detail below in combination with the flow chart and embodiments.

[0068] 1. Each time the graph neural network parameters and graph data are updated, the Deep Graph Library client provides the updated data to the GCCL client.

[0069] 2. After receiving new data, the GCCL client processes the vertex data that needs to be sent to other computing nodes along the outgoing edge according to the graph partition information, and packages several data sent to the same computing node to save time and bandwidth.

[0070] 3. After confirming that the corresponding GCCL client is ready to receive, send the corresponding data.

[0071] 4. The GCCL client checks whether all the remote map data required by the local map partition has been received. If not, it repeats the check until the remote map data is received.

[0072] 5. After the GCCL client completes receiving the data, it updates the data in this partition based on all the updated information and submits the updated data to the Deep Graph Library client for the next round of training.

[0073] 6. If the training of the graph neural network is not completed, repeat the process 1-5 until the training is completed.

[0074] The computation graph optimization phase is different from other phases. It is only run once after the initialization of this instance is completed. The overall process is as follows: Figure 3 As shown, the specific implementation method will be introduced in detail below in combination with the flow chart and embodiments.

[0075] 1. The Deep Graph Library module generates the corresponding computational graph based on the structure of the graph neural network model.

[0076] 2. The computational graph optimization module traverses the computational graph, classifies the computational operations therein, and fuses several appropriate computational nodes through a designed algorithm to generate a new computational operation, thereby optimizing the original computational graph.

[0077] 3. The Deep Graph Library module generates corresponding GPU executable kernels for all computational operations based on the optimized computational graph. In this process, kernel optimization is performed based on the designed feature thread adaptation group to improve GPU utilization.

[0078] After the computational graph optimization phase is completed, the training phase begins. The Deep Graph Library module submits the kernel to the GPU. The GPU reads the graph neural network parameters and graph data from the Deep Graph Library module according to the operations defined in the kernel and starts training. After each round of training, the GPU returns the results to the Deep Graph Library module. The Deep Graph Library module then enters the data communication phase by providing the data to the GCCL module. After the communication is completed, the Deep Graph Library module uses the new data for the next round of training until the graph neural network training is completed. At this point, the life cycle of a graph neural network training instance ends.

[0079] This paper constructs a distributed graph neural network computational efficiency optimization method based on Deep Graph Library and GCCL. This method addresses the high memory requirements and time consumption of graph neural network training through distributed communication optimization and graph neural network computational process optimization, providing a more convenient and rapid graph neural network training solution for graph neural network researchers. The Deep Graph Library provides basic support for graph data processing, the construction, and training of graph neural networks, and GCCL optimizes graph data communication in a distributed environment. This paper provides a more convenient and rapid graph neural network training solution for graph neural network researchers.

[0080] Specifically, in an application area such as social network analysis, social forums like Facebook need to recommend content that may be of interest to users based on their social connections in order to provide them with content that better suits their interests. To achieve this, social forums first need to construct a social network for their users, where vertices represent individual users and edges represent the following relationships between users. User profiles are then characterized by information such as each user's identity and browsing history.

[0081] After preparing the social network dataset, the social network can be used to train the graph neural network, so that the graph neural network can extract more accurate features based on the user's own characteristics and their social relationships. The features can then be used to recommend corresponding posts, advertisements or other users.

[0082] However, general social networks often have hundreds of millions of vertices, and the amount of data exceeds the training capabilities supported by a single GPU. Therefore, distributed graph neural network training methods are needed to provide more efficient training.

[0083] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for optimizing the computational efficiency of a distributed graph neural network, characterized in that: It includes the data communication stage, the computational graph optimization stage, and the graph neural network training stage; The data communication stage includes: S11. The Deep Graph Library module submits the local graph data to the GCCL module; S12, the GCCL module maps the graph data according to the graph partitions and sends the graph data located at other nodes to the GCCL modules of the corresponding nodes; S13, the GCCL module receives graph data sent by other nodes; S14. The GCCL module updates the local graph data according to the received graph data, and returns the updated graph data to the Deep Graph Library module; The computation graph optimization phase includes: S21, the Deep Graph Library module generates a computational graph based on the local graph data structure and submits the computational graph to the computational graph optimization module; S22, the computation graph optimization module optimizes the computation graph according to the designed optimization process, and returns the optimized computation graph to the Deep Graph Library module; The graph neural network training phase includes: S31, the Deep Graph Library module generates corresponding GPU operators according to the optimized calculation graph and submits them to the GPU for calculation; S32. After the GPU completes the calculation corresponding to the graph neural network training, it returns the calculation result to the Deep GraphLibrary module and updates the local graph data.

2. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 1, characterized in that: When the GCCL module communicates with the GCCL modules of other nodes, it uses the shortest path spanning tree algorithm to obtain the optimal communication strategy based on the GPU topology in the distributed system, and the GCCL modules communicate according to this communication strategy.

3. The method for optimizing the computing efficiency of a distributed graph neural network according to claim 1, wherein: In step 21, the computational graph generated by the Deep Graph Library according to the graph neural network model is optimized and then the corresponding operators are generated.

4. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 3, characterized in that: Computational graph optimization includes operator fusion optimization, specifically: Operator fusion optimization classifies the computing operations according to the computing graph provided by the Deep Graph Library, and fuses several set computing nodes to generate a new computing operation, thereby optimizing the original computing graph.

5. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 3, characterized in that: Optimization of the computational graph includes GPU kernel optimization, specifically: Based on the thread scheduling characteristics of the GPU and the computational characteristics of graph neural networks, GPU kernel optimization converts the optimized computational graph into operator instructions that the GPU can receive, and assigns a feature-adaptive thread group to each vertex in the graph for corresponding calculations, thereby achieving feature-level and vertex-level parallelism during training.

6. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 1, characterized in that: Graph data is stored in a compressed sparse row format.

7. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 6, characterized in that: The compressed sparse row extracts the non-zero elements from the adjacency matrix of the graph data, generates three arrays of vertex value V, row index and column index to establish an index for each vertex feature, and realizes the compression of the sparse adjacency matrix of the graph data.

8. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 7, characterized in that: When generating compressed sparse rows, the vertices in the graph are sorted in descending order of in-degree, so that when reading vertices, vertices with larger degrees are started for calculation first and vertices with similar degrees are assigned to the same GPU block. After reversing the graph, the corresponding inverse compressed sparse rows are generated in descending order of vertex out-degree.

9. A method for optimizing the computational efficiency of a distributed graph neural network according to claim 1, characterized in that: The graph neural network model is GCN or GAT.

Images