A heuristic graph partitioning method based on GPU computing power awareness

Abstract

The present invention discloses a heuristic graph partitioning method based on GPU computing power perception, which relates to graph data processing technology. This solution is proposed to solve the problem of unbalanced loads of various GPUs in the prior art. The method includes the following steps: edge partitioning the original graph to obtain the load subgraphs of each GPU, wherein the edge partitioning is performed by combining neighborhood expansion and simulated annealing; measuring the computational amount of each GPU subgraph to determine the allocation of subgraph training vertices; completing one-hop neighbors for cut vertices that are missing certain neighborhood information; and distributing each subgraph to each GPU. The advantages are that the structural features of the original graph are retained as much as possible on the subgraph, thereby ensuring that the training accuracy will not be lost too much; the partitioning time is accelerated and it is avoided to fall into the local optimum; the computational amount on the final load subgraph of each GPU matches its own computing power, thereby achieving an acceleration effect on the overall parallel training process.

Description

Technical Field

[0001] The present invention relates to graph data processing technology, and in particular to a heuristic graph partitioning method based on GPU computing power perception. Background Art

[0002] Traditional deep learning neural network models have achieved good results in processing regular data represented in Euclidean space. However, in fact, in many fields in the real world, the information is represented in the form of graph-structured data. When processing graph data, due to its irregularity, traditional neural network models cannot achieve good reasoning effects. Therefore, graph neural network GNN has emerged as an emerging neural network training model.

[0003] When processing graph-structured data information, GNN can achieve better information processing effects than traditional neural network models. In the GNN model, the embedding representation of each vertex is usually presented in matrix format, so the GNN training process is essentially a large number of matrix multiplication operations. Since GPUs are efficient in performing matrix multiplication, GPUs are currently commonly used to accelerate GNN training. With the advent of the big data era, information data in many fields needs to be represented by increasingly large-scale graphs. For example, platforms such as Facebook and Instagram, for example, the social relationship networks of their users are represented by graph-structured data, which even contain more than billions of vertices and trillions of edges. When training GNN models on such large-scale graph datasets, due to the limited memory space of GPUs, the complete original image data cannot be put into the GPU for GNN training at one time, which greatly increases the time cost of GNN model training. Therefore, this has become the main obstacle to the expansion of GNN to large graphs. In response to this challenge, the parallel training mode of single-machine multi-card is considered to be an effective solution, which can achieve the purpose of training GNN models on large-scale graphs, and at the same time, the overall training time cost will be significantly reduced. In the entire process of parallel training, each GPU acts as a computing node, and the communication between GPUs in each computing node is often the most time-consuming.

[0004] To address this problem, some scholars have proposed a solution for GNN parallel training at the subgraph-level. The overall process of subgraph-level GNN parallel training is as follows: First, the original graph is preprocessed, that is, the graph is partitioned. After partitioning, each subgraph will be distributed to each computing node participating in the training. In each epoch of training, each computing node will have a complete GNN model and use its own local subgraph copy to train the GNN model.

[0005] However, parallel training of GNNs currently faces the following difficulties: (1) Considering the uniqueness of the graph structure, each vertex and edge in the graph often has dependencies on its neighbors. Therefore, the integrity of the graph structure should be maintained as much as possible during the graph partitioning process. If a vertex participating in training in a subgraph is missing a large number of neighboring vertices in the subgraph, this will result in the inability of the vertex to aggregate complete neighborhood information during subsequent training, which is likely to cause a loss in training accuracy. (2) Currently, most parallel training platforms are often configured with different models of GPUs, and the configurations and working parameters of different models of GPUs may also differ. Existing graph partitioning methods usually do not pay attention to the differences in computing power between GPUs on heterogeneous training platforms, so that GPUs cannot perform at their best during parallel training.

[0006] A storage-optimized distributed graph processing method (CN 107122248 A) divides the graph equally according to the total number of nodes; a graph retrieval method based on subgraph partitioning (CN 115080776 A) uses a separate graph convolutional network to generate a graph node allocation matrix, and uses the node feature matrix and the first-order adjacency matrix of the graph to divide the nodes into corresponding subgraphs; a graph data partitioning method for load balancing (CN 115658975 A) makes the number of nodes between subgraphs as equal as possible by averaging the node degrees. None of the above studies consider that when there is a difference in GPU computing power, such methods may not be able to achieve true load balancing.

[0007] Therefore, it is currently necessary to design a graph partitioning algorithm that can perceive the differences in computing power of each GPU on a heterogeneous training platform, so that each GPU can perform at its best during parallel training, thereby accelerating the entire training process. Summary of the invention

[0008] The purpose of the present invention is to provide a heuristic graph partitioning method based on GPU computing power perception to solve the problems existing in the above-mentioned prior art.

[0009] The heuristic graph partitioning method based on GPU computing power perception described in the present invention comprises the following steps:

[0010] S1. performing edge partitioning on the original image to obtain a load sub-image of each GPU, wherein the edge partitioning is performed by combining neighborhood expansion and simulated annealing;

[0011] S2. Measure the computational effort of each GPU subgraph to determine the allocation of subgraph training vertices;

[0012] S3. Complete the one-hop neighbors for the cut vertices that are missing certain neighborhood information;

[0013] S4. Distribute each subgraph to each GPU.

[0014] The heuristic graph partitioning method based on GPU computing power perception described in the present invention has the following advantages: 1. In the stage of subgraph generation, the structural features of the original graph can be retained as much as possible on the subgraph by combining the currently more advanced edge partitioning ideas of neighborhood expansion, thereby ensuring that the training accuracy will not be lost too much; in addition, simulated annealing methods are also incorporated to accelerate the partitioning time to a certain extent and avoid falling into the local optimum. 2. When allocating the ownership of training vertices, the computing power differences of each GPU are fully considered, so that the computing amount of each GPU on the final load subgraph matches its own computing power, thereby achieving an acceleration effect on the overall parallel training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a flow chart of the heuristic graph partitioning method described in the present invention.

[0016] Figure 2 It is a schematic diagram of the GPU configuration on a single machine in an embodiment of the present invention. DETAILED DESCRIPTION

[0017] like Figure 1 and Figure 2 As shown, the heuristic graph partitioning method based on GPU computing power perception described in the present invention can maximize the best performance of each GPU on a heterogeneous GNN parallel training platform and accelerate the entire GNN parallel training process.

[0018] The specific steps include:

[0019] S1. Apply an edge partitioning scheme NESA that combines the ideas of neighborhood expansion and simulated annealing to the original graph to obtain the load subgraph of each GPU, corresponding to Figure 1 Step ① in the original Figure 1 Divide into two, and get two subgraphs with redundant vertices (that is, cut vertices) at the boundary. For example, vertex 0 and vertex 1 are the same vertex on different subgraphs. The specific process is as follows:

[0020] S101, the user inputs the original image data, the initial temperature T, and the target number of sub-image divisions N. Assume that there are two GPUs on a server, namely RTX3090 and A40. Figure 2 As shown, therefore N=2, and the initial temperature T=800 is selected.

[0021] S102. Randomly select a vertex from V as an initial seed vertex and put it into the seed set.

[0022] S103: Select all one-hop neighbors of the initial seed vertex and put them into the candidate seed set.

[0023] S104. Put all edges associated with the initial seed vertex into the edge set of subgraph p0.

[0024] S105 . Put the vertices of the union of the seed set and the candidate seed set into the vertex set of the subgraph p0.

[0025] S106, when the number of edges in the edge set of the subgraph p0 is less than the self-defined threshold, the threshold is selected as 1.8 / 3×|E|. If the difference between the vertex set of the subgraph p0 and the seed set is not an empty set, a new seed vertex is selected and put into the seed set by calculating the score of each vertex in the candidate seed set. The score calculation formula of each candidate seed vertex v_cand is as follows:

[0026] score(v_cand)=

[0027] T×|Ν(v_cand)|-|N(v_cand)-(N(v_cand)∩cand_seed_set)|(1).

[0028] Calculate the score of each vertex v_cand in the candidate set, select the vertex with the highest score as the new seed vertex and put it into the seed set, execute S103, S104, and S105 on the new seed vertex to update the candidate seed set, the edge set of subgraph p0, and the vertex set of subgraph p0, and gradually reduce the temperature T according to a certain ratio δ=0.8.

[0029] S107. Repeat S106 until the number of edges in the edge set of subgraph p0 exceeds 1.8 / 3×|E|, and the remaining edges naturally belong to subgraph p1.

[0030] S2. Based on the characteristic parameters of the graph, such as the graph Gini coefficient G and the normalized edge distribution entropy Hen, a heuristic scoring function is designed to measure the computational load of each GPU subgraph, and to guide the allocation of training vertices in the cut vertices at the boundaries of each subgraph, corresponding to Figure 1 In step ②, a vertex will be selected from vertex 0 and vertex 1 as the training vertex on its corresponding subgraph. For example, here we finally decide to use vertex 0 on subgraph 0 as the training vertex. The specific process is as follows:

[0031] S201. The user inputs the structural characteristic parameters of each subgraph required, including the graph Gini coefficient G, the normalized edge distribution entropy Hen, and the average degree D of the subgraph. avg , and count the number of training vertices remaining in each subgraph except the training vertices in the cut vertices, that is, the number of training vertices remaining except the cut training vertices v_cut_train i _local_train_v|.

[0032] S202, determine which GPU should be responsible for training the cut training vertex v_cut_train through the following heuristic scoring function:

[0033]

[0034]

[0035] Where α = 0.5, β = 1, γ = 0.2, GPU0 can perform 405.39 times of 27500 × 800 matrix multiplication per unit time, GPU1 can perform 323.89 times of 27500 × 800 matrix multiplication per unit time, so m ij =1.25,

[0036] p0_comp = 0.79, p1_comp = 0.59. Substituting the above parameters into formula (2) and (3), we get the score of vertex 0 in subgraph 0 is -1.88, while vertex 1 in subgraph Figure 1 The score is -2.22, so the training vertex will be assigned to subgraph 0, that is, vertex 0 should be used as a training vertex in subgraph 0.

[0037] S3. Apply the SBV scheme to all cut vertices of the two obtained subgraphs p0 and p1. If the number of neighbors of a cut vertex v_cut on a subgraph in the subgraph meets the following criteria, where λ = 0.6: Then all one-hop neighbors will be completed for the vertex in the subgraph, corresponding to Figure 1 Step ③, here in the sub Figure 1 Vertex 1 on the graph is missing 3 neighbors on this subgraph. At this time, it will be the child Figure 1 Vertex 1 on completes all its one-hop neighbors.

[0038] S4. Assign the obtained two subgraphs to GPU0 and GPU1 respectively, and then perform parallel training of the GCN or GAT model on the heterogeneous platform.

[0039] For those skilled in the art, various other corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all of these changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A heuristic graph partitioning method based on GPU computing power perception, characterized in that: The following steps are involved: S1. performing edge partitioning on the original image to obtain a load sub-image of each GPU, wherein the edge partitioning is performed by combining neighborhood expansion and simulated annealing; S2. Measure the computational effort of each GPU subgraph to determine the allocation of subgraph training vertices; S3. Complete the one-hop neighbors for the cut vertices that are missing certain neighborhood information; S4. Distribute each subgraph to each GPU; Step S1 includes the following sub-steps: S101. The user inputs the original graph data, including the vertex set V, the edge set E, the initial temperature T, and the subgraph partition target number N; S102. Randomly select a vertex from the vertex set V as an initial seed vertex and put it into the seed set; S103. Select all one-hop neighbors of the initial seed vertex and put them into the candidate seed set; S104. Put all edges associated with the initial seed vertex into the edge set E of the subgraph pi; S105. Put the vertices of the union of the seed set and the candidate seed set into the vertex set V of the subgraph pi; S106. When the number of edges in the edge set of subgraph pi is less than a user-defined threshold, and the difference between the vertex set V of subgraph pi and the seed set is not an empty set, a candidate seed with the highest score is selected as a new seed vertex and put into the seed set by calculating the score of each vertex in the candidate seed set; if the difference between the vertex set of subgraph pi and the seed set is an empty set, a vertex is randomly selected from the difference between the vertex set and the vertex set of subgraph pi as a new seed vertex and put into the seed set; substeps S103, S104, and S105 are re-executed for the new seed vertex to update the candidate seed set, the edge set E of subgraph pi, and the vertex set V of subgraph pi, and the temperature T is gradually reduced according to a certain ratio δ; The score calculation formula for each candidate seed vertex is as follows: score(v_cand)=T×|Ν(v_cand)|-|N(v_cand)-(N(v_cand)∩cand_seed_set)|; Where T represents the current temperature, Ν(v_cand) represents the one-hop neighbor set of the candidate seed vertex v_cand, |Ν(v_cand)| represents the number of its one-hop neighbors, and cand_seed_set represents the candidate seed set; S107. Repeat sub-step S106 until the number of edges in the edge set E of the sub-graph pi exceeds a user-defined threshold; Step S2 includes the following sub-steps: S201. The user inputs the required structural feature parameters in each subgraph pi, and counts the number of training vertices remaining in each subgraph except the cutting vertices, that is, the cutting training vertices v_cut_trian. i _local_train_v|; The structural feature parameters include the graph Gini coefficient G, the normalized edge distribution entropy Hen, and the average degree D of the subgraph pi avg ; S202. Determine which GPU should be responsible for training the cut training vertex through the following heuristic scoring function: in, represents the number of neighbors of the cut training vertex v_cut_train in the subgraph pi; N(v_cut_train) represents the number of neighbors of the cut training vertex v_cut_train in the original graph; p i _comp is the computational cost of cutting the training vertex v_cut_train in the subgraph pi; |p i _local_train_v| represents the number of training vertices currently assigned to the subgraph pi. The initial value is the number of training vertices in the subgraph pi minus the number of cut training vertices; m ij represents the computing power ratio of the i-th GPU to the j-th GPU; α represents the degree of concern for avoiding the loss of training accuracy during the allocation process, and β represents the degree of concern for matching the computational amount of the load subgraph pi on the GPU with its own computing power during the allocation process; γ represents the average degree D avg The degree of influence on the calculation amount of sub-graph pi.

2. According to claim 1, a heuristic graph partitioning method based on GPU computing power perception is characterized in that: Step S3 includes the following sub-steps: S301. Count the missing neighborhood information of the cut vertex v_cut in each subgraph pi; S302. Set a threshold λ; λ is a user-defined hyperparameter used to indicate the threshold of the amount of missing information that causes a significant loss in training accuracy; when When , complete the one-hop neighbors of the vertex on the boundary of subgraph pi; S303. Traverse all cut vertices in all subgraphs pi and repeat substeps S301 and S302.

3. According to claim 2, a heuristic graph partitioning method based on GPU computing power perception is characterized in that: Step S4 includes the following sub-steps: S401. Assign the divided sub-graphs pi to the corresponding GPUs; S402. Each GPU starts the parallel training task of the GNN model.

Images