Graph rearrangement algorithm for optimizing GNN training

Abstract

The invention discloses a graph rearrangement algorithm for optimizing GNN training, relates to artificial intelligence, and provides a scheme for solving the problem of memory access bottleneck in the prior art. Comprising the following steps: S1, adjacency list extraction and node classification; s2, performing initialization and random selection; s3, selecting a next node to be sorted by using a maximum public neighbor strategy and a random selection strategy; s4, iteration: repeatedly executing the step S3 until all non-isolated nodes are sorted; s5, a processing step of isolated nodes; and S6, updating node attributes. The method has the advantages that GNN training is optimized specially, millions of node maps can be sorted quickly, data locality is improved to a great extent, the hit rate of a memory access place is increased, and memory access delay is reduced. The method not only can be used for acceleration of GNN training, but also can be used for acceleration of SpMV; this is natural, because the locality of the data is increased, the hit rate of memory access can be improved when the data is accessed, and then the calculation of the SpMV is accelerated.

Description

Technical Field

[0001] The present invention relates to artificial intelligence, and particularly to a graph rearrangement algorithm for optimizing GNN training. Background Art

[0002] A graph is a classic non-Euclidean geometric data structure that describes the characteristics and structure of data through a series of nodes and their mutual relationships (edges). Graph Neural Networks (GNNs) have attracted extensive attention from researchers in the field of machine learning because of their ability to model and learn graph-structured data. It has not only changed the understanding of graph data but also provided powerful solutions to problems in multiple fields such as social network analysis, recommendation systems, and bioinformatics.

[0003] However, graph data has natural irregularity. Different from image data, any two data in graph data may have a connection relationship, so the locality of graph data is very low, which results in irregular memory access. The irregularity of graph data poses a huge challenge to the training of GNNs. Especially when dealing with large-scale graph data, the training and inference costs of GNN models are expensive, requiring a large amount of computing resources and time. How to optimize the memory access of graph data is the key to improving the training speed of GNNs.

[0004] In the technology of graph reordering, it is necessary to traverse all nodes and generate a sequential ID for the nodes to map the new permutation order. Most of the current graph reordering technologies are used in sparse matrix multiplication (SpMV) to accelerate the calculation speed of SpMV. In the prior art, "Rabbit Order: Just-in-time Parallel Reordering for Fast Graph Analysis; Junya Arai, Hiroaki Shiokawa, Takeshi Yamamuro, Makoto Onizuka and Sotetsu Iwamura" was mainly proposed for accelerating SpMV. It first calculates the modularity of each node with the top-level nodes in the existing community, and selects the most suitable target community for merging according to the size of the modularity, and finally forms several communities. This algorithm needs to traverse all nodes in the existing community multiple times and calculate the modularity, and the complexity of the algorithm is very high. And "When is Graph Reordering an Optimization? Studying the Effect of Lightweight Graph Reordering Across Applications and Input Graphs; Vignesh Balaji, Brandon Lucia" reorders the graph simply according to the degree of the nodes, which may not be able to reduce the overall physical distance between nodes. The two documents cannot solve the technical bottleneck of high latency caused by a large number of accesses to features during GNN training. Summary of the Invention

[0005] The object of the present invention is to provide a graph reordering algorithm for optimizing GNN training to solve the problems existing in the above-mentioned prior art.

[0006] The graph reordering algorithm for optimizing GNN training in the present invention includes the following steps:

[0007] S1. The step of extracting the adjacency list and classifying nodes;

[0008] S2. The steps of initialization and random selection;

[0009] S3. The step of using the maximum common neighbor strategy and the random selection strategy to select the next node to be sorted;

[0010] S4. The iterative step: repeat step S3 until all non-isolated nodes are sorted;

[0011] S5. The processing step of isolated nodes;

[0012] S6. Steps to update node attributes.

[0013] In the graph rearrangement algorithm for optimizing GNN training described in the present invention, the advantages are as follows: It is specifically optimized for GNN training, can quickly sort millions of node maps, greatly improve data locality, thereby increasing the hit rate of memory access and reducing the latency of memory access. It can not only be used to accelerate GNN training, but also to accelerate SpMV; this is natural because increasing data locality can improve the hit rate of memory access when accessing data, thereby accelerating the calculation of SpMV. Brief Description of the Drawings

[0014] Figure 1 It is a schematic flowchart of the algorithm described in the present invention.

[0015] Figure 2 It is a schematic diagram showing the effect of the graph data without being rearranged using the algorithm described in the present invention.

[0016] Figure 3 It is a schematic diagram showing the effect of the graph data rearranged using the algorithm described in the present invention.

[0017] Figure 4 It is a comparison chart of experimental results. Detailed Embodiments

[0018] The present invention is specifically optimized for GNN training to solve the access bottleneck of graph data in GNN. In GNN, not only nodes need to be accessed, but also the features of each node need to be accessed. These node features usually have very high dimensions. If the distribution of these nodes in memory is very scattered, accessing these node features will consume a large amount of time. In GNN training, nodes aggregate and transfer messages through the topological structure of the graph, that is, when a node is accessed, its neighbor nodes are also very likely to be accessed. Therefore, each node and its neighbors need to be placed as close as possible to increase data locality. After such an arrangement, the access to its features will be greatly accelerated. The present invention aims to increase node locality and reduce the physical distance between nodes and their neighbors by rearranging the graph. Graph rearrangement only changes the data layout but does not affect the graph structure or processing results. The purpose of graph rearrangement is to make nodes and their neighbors as close as possible in the arrangement, increase the hit rate of memory access, improve the speed of memory access, and thus improve the speed and accuracy of GNN training.

[0019] As Figure 1 shown, the graph rearrangement algorithm for optimizing GNN training described in the present invention includes the following steps.

[0020] S1. Steps of adjacency list extraction and node classification:

[0021] Batch extract the adjacency list from the graph, and divide the nodes into an isolated node set Vi and a non-isolated node set Vn; among them, the isolated node set Vi does not participate in the sorting process.

[0022] S2. Initialization and random selection steps:

[0023] S21. Initialize a set N ordered for storing the nodes that have been sorted, and randomly select a starting node from the non-isolated node set Vn as the starting point of sorting and add it to the set N ordered . The initial state of the set N ordered is an empty set.

[0024] S22. Initialize a set S for storing the unsorted neighbor nodes of the set N ordered ; maintain the frequency of each node appearing in the set S. For example, if the nodes stored in the set N ordered are [node 1, node 2, node 3], then the set S stores [[neighbors of node 1], [neighbors of node 2], [neighbors of node 3]].

[0025] S3. Steps to select the next node to be sorted using the maximum common neighbor strategy and the random selection strategy:

[0026] S31. If the set S is not empty, select a node with the highest frequency from it and move it to the set N ordered , and move the unsorted neighbor nodes of this node from the non-isolated node set Vn to the set S, and update the frequency of each node.

[0027] S32. If the set S is empty, randomly select a node from the unsorted non-isolated node set Vn and move it to the set N ordered , and move the unsorted neighbor nodes of this node from the non-isolated node set Vn to the set S, and update the frequency of each node.

[0028] S4. Iteration steps: Repeat step S3 until all non-isolated nodes are sorted.

[0029] S5. Processing steps for isolated nodes: Randomly shuffle the isolated node set Vi and move it to the end of the set N ordered . At this point, all nodes are sorted, and the nodes of the graph will have a new node order, that is, the elements in the set N ordered are sorted.

[0030] S6. Steps to update node attributes: Rearrange the various attributes of the graph according to the new node order to complete all processes of graph rearrangement.

[0031] Given a graph \(G(n, e)\), where \(n\) represents nodes and \(e\) represents edges, the locality of graph data can be measured by the sum of the distances between each node in the graph and its neighbors. Assume that the layout function \(\pi: v\rightarrow n\) represents assigning a position to any node, and the distance between any two nodes is \(\xi\). π (u, v)=\vert\pi(u)-\pi(v)\vert\), then the locality of the graph can be measured by \(\xi(G, \pi)=\sum\) (u,v)∈e \(\xi(u, v)\). The smaller this value is, the higher the locality of the data. After being processed by the algorithm of the present invention, the locality of the rearranged graph data is greatly enhanced, as Figure 3 shown. When the rearrangement is not performed using the algorithm described in the present invention, Figure 2 the sum of the distances between each node and all its neighbors in is 84, while after the rearrangement using the algorithm described in the present invention, the sum of the distances between each node and all its neighbors is 42, and the distance is reduced by 50%. Taking node 0 as an example, when the rearrangement is not performed using the algorithm described in the present invention, the sum of the distances between node 0 and its neighbors \([1, 3, 7, 8, 9]\) is 28, and after the rearrangement using the algorithm described in the present invention, the sum of the distances between node 0 and its neighbors is 12. It can be seen that the sum of the distances between node 0 and its neighbors is reduced by 57.1%. Compared with Figure 2 , the rearranged graph can make the positions of nodes and their neighbor nodes in memory as close as possible, thereby improving the hit rate of memory access and accelerating memory access. It can bring speed improvement to the training of both GNN and SpMV calculations. In addition, it is also beneficial to the layout and wiring in integrated circuits, as follows:

[0032] Optimize the training of GNN, reduce the training time, and improve the training accuracy. The present invention is mainly aimed at the scenario of GNN training. After the graph rearrangement, the locality of the graph data is enhanced, and then the locality of the features of the graph is also enhanced. When accessing the features of a node in the graph, the features of its neighbors will also be accessed. In the training of GNN, the access to high-dimensional features in memory is a bottleneck. After the rearrangement, the locality of the features is also enhanced. Therefore, when accessing node features, the cache hit rate can be increased, thereby accelerating the training of GNN. In the case of FOSGNN, which uses consecutive nodes to induce subgraphs for training, GNN can increase the connectivity of consecutive nodes in each batch, increase the number of edges in each batch, and thus improve the training accuracy.

[0033] Accelerate the calculation of SpMV. In a computer architecture with a multi-level cache structure, the cache hit rate can be increased by enhancing the spatial locality of non-zero data. Because when a piece of data is read, due to spatial locality, the data surrounding it in the sparse matrix will also be read into the cache. After graph rearrangement, the locality of data in memory is enhanced, and the elements around it will be read into the cache. A high proportion of the data in the cache are non-zero elements, which are the key to the SpMV calculation. Therefore, after graph rearrangement, the operation of SpMV can be accelerated. Using a 10,000x10,000 random matrix, repeated experiments were conducted under sparsity levels of {0.9, 0.8, 0.7, 0.6, 0.5}. In each experiment, the SpMV operation was performed 1,000 times and the average time was taken as its operation time. As Figure 4 shown, the experimental results indicate that graph rearrangement can achieve an acceleration factor of 1.10 - 1.35, which also proves that the method of the present invention can increase data locality and thus reduce the latency of memory access.

[0034] Reduce the wiring resources of VLSI (Very Large Scale Integration Circuit). The topological structure connecting each module of an integrated circuit is equivalent to a graph. Each node in the graph represents a module, and each edge represents the connection relationship between modules. After graph rearrangement, the locality of modules can be increased, that is, the distance between any two connected modules is reduced as a whole, thereby reducing the wiring resources between modules.

[0035] For those skilled in the art, various corresponding changes and deformations can be made according to the technical solutions and concepts described above, and all such changes and deformations should fall within the protection scope of the claims of the present invention.

Claims

1. A graph reordering algorithm for optimizing GNN training, characterized in that: The following steps are involved: S1. Steps of adjacency list extraction and node classification; S2. Initialization and random selection steps; S3. A step of selecting the next node to be sorted using the maximum common neighbor strategy and the random selection strategy; S4. Iteration step: Repeat step S3 until all non-isolated nodes are sorted; S5. Processing steps for isolated nodes; S6. Step of updating node attributes.

2. A graph reordering algorithm for optimizing GNN training according to claim 1, characterized in that: The step S1 specifically includes: extracting adjacency lists in batches from the graph, dividing the nodes into an isolated node set Vi and a non-isolated node set Vn; wherein the isolated node set Vi does not participate in the sorting process.

3. A graph reordering algorithm for optimizing GNN training according to claim 2, characterized in that: The step S2 comprises the following sub-steps: S21. Initialize a set N ordered It is used to store the nodes that have been sorted, and randomly selects a starting node from the non-isolated node set Vn as the starting point of the sort and adds it to the set N ordered The set N ordered The initial state of is an empty set; S22. Initialize a set S to store set N ordered Unsorted neighbor nodes of ; maintain the frequency of occurrence of each node in the set S.

4. A graph reordering algorithm for optimizing GNN training according to claim 3, characterized in that: The step S3 comprises the following sub-steps: S31. If the set S is not empty, select a node with the highest frequency and move it to the set N ordered And move the unsorted neighbor nodes of the node from the non-isolated node set Vn to the set S, and update the occurrence frequency of each node; S32. If the set S is empty, randomly select a node from the unsorted non-isolated node set Vn and move it to the set N ordered And move the unsorted neighbor nodes of the node from the non-isolated node set Vn to the set S, and update the occurrence frequency of each node.

5. A graph reordering algorithm for optimizing GNN training according to claim 4, characterized in that: The step S5 specifically includes: randomly shuffling the isolated node set Vi and moving it to the set N ordered The end of .

6. A graph reordering algorithm for optimizing GNN training according to claim 5, characterized in that: The step S6 specifically includes: rearranging the attributes of the graph according to the new node order.

Images