Graph neural network training acceleration method and system based on multi-GPU and multi-SSD

Abstract

The present invention discloses a graph neural network training acceleration method and system based on multiple GPUs and multiple SSDs, belonging to the field of graph neural network training. The method reads the physical architecture information of the server, models the communication link relationship between the CPU, GPU, and SSD as a communication topology graph, and converts the link process of the GPU pulling data into a single-source single-sink maximum flow problem, solves the optimal access traffic distribution scheme through the Ford-Fulkerson augmented path calculation, and then considers the access frequency of the graph vertices to allocate the graph vertices to the storage nodes with the best current performance indicators for storage; after finally completing the storage of all graph vertices in the graph data set, the server executes the batch-by-batch training task of the graph neural network. The present invention can greatly improve the training throughput of the system, meet the training of TB-level ultra-large-scale graphs, and realize efficient and low-latency multi-GPU multi-SSD data access.

Description

Technical Field

[0001] The present invention belongs to the field of graph neural network (GNN) training systems, and specifically relates to a graph neural network training acceleration method and system based on multiple GPUs and multiple SSDs. Background Art

[0002] Graph neural network is a powerful tool for processing graph structured data, which can effectively capture complex relationships and patterns in data. It has been widely used in many fields such as financial fraud, recommendation systems, social media analysis, etc. However, with the explosive growth of information, TB-level graph data has become more and more common. This large-scale graph data not only appears in the field of e-commerce, but is also widely used in other fields such as network security, financial technology, biomedicine, etc., promoting innovation and progress in all walks of life.

[0003] As the scale of graph data grows, how to efficiently train ultra-large-scale graph data has become a major challenge. In recent years, many graph training systems based on GPU sampling have been proposed. These systems can be divided into distributed graph training systems, single-machine memory-based graph training systems, and single-machine disk-based graph training systems. Although distributed computing solutions can effectively complete the training of large-scale graph neural networks, they also bring significant cost overhead and complex management difficulties. The graph training system based on single-machine memory uses multi-GPU training to significantly improve the processing speed and video memory capacity, but it is still limited by the size of CPU memory and cannot train TB-level ultra-large-scale graph data. Although the graph training system based on single-machine disk can process TB-level ultra-large-scale graph data, there is currently only a single GPU implementation solution in this scenario. Although it can be expanded to multi-GPU implementation, the actual physical communication topology between GPU and SSD is still not considered, resulting in low GPU utilization and load imbalance.

[0004] The advantages and disadvantages of some solutions in the existing technology are as follows:

[0005] 1. GNNLab is a graph neural network framework proposed at Eurosys 2022. It proposes a pre-sampling-based caching strategy that takes into account both the sampling algorithm and the GNN dataset, showing good cache performance. At the same time, each GPU is dedicated to graph sampling or model training tasks, and accelerates these two tasks by eliminating GPU memory contention. However, for GNNLab, the contents of its multiple GPU caches are the same, resulting in poor scalability of multi-GPU caches. At the same time, the management of graph topology is very coarse-grained. GNNLab completely stores the graph topology in the video memory of a single GPU, and the size of its trainable graph dataset is limited by the GPU video memory.

[0006] 2. Quiver is a graph training framework based on PyG. Quiver distributes the feature cache on multiple GPUs by hashing inside the NVLink clique, and uses NVLink to achieve cross-GPU access; Quiver replicates the feature cache between NVLink cliques. Quiver's design extends the cache space to the NVLink clique. However, some servers contain fewer GPUs in the NVLink clique, so Quiver also faces the problem of poor scalability of multi-GPU cache. In addition, Quiver only supports storing the graph topology completely in CPU memory or GPU memory. If the graph is stored in CPU memory, huge CPU-GPU data transmission overhead will be introduced when sampling the graph; if the graph is stored completely in GPU memory, the scale of the graph that can be supported for training will be limited by the size of the GPU video memory.

[0007] 3. Ginex uses a single-card GPU, CPU memory, and SSD to complete the training of graph data that exceeds the GPU memory size. It uses the CPU to perform super batch generation, fills the CPU memory by pre-sampling data and graph partitioning, and minimizes subsequent SSD access during the corresponding mini batch generation process. However, the CPU processing power of the Ginex system is severely limited, and the GPU utilization rate is as low as 0.6%. There are two reasons for this: 1) The CPU workload in super batch and mini batch preparation is too large, occupying more than 70% of the training time, resulting in reduced GPU utilization; 2) The GPU utilization rate is reduced.

[0008] 4. GIDS manages the disk IO process through the BAM driver, and in this way uses GPU direct and SSD for direct storage access to achieve large-scale graph neural network training for GPU. GIDS enables GPU threads to obtain feature vectors directly from SSD storage to solve memory capacity limitations. It introduces dynamic storage access accumulators, constant CPU buffers, and GPU software caches with window buffers to balance the resource utilization of the entire system, thereby improving end-to-end training throughput. However, due to the following three serious problems, the overall training throughput of GIDS is low: 1) Due to insufficient parallelism, the disk IO throughput cannot be maximized even if all GPU cores are used; 2) Due to severe GPU core contention problems, graph sampling, feature extraction, and model training are almost executed serially; 3) The GPU cache space is limited and cannot meet the needs of TB-level graphs, resulting in more than 73% of the data still being obtained from disk.

[0009] 5. Helios proposed a GPU-initiated asynchronous disk IO stack that allows the GPU to directly access graph data on the SSD and does not waste GPU cores between IO submission and IO completion, leaving most GPU cores for useful GNN cores. Secondly, it designed a GPU-managed heterogeneous cache that extends to CPU / GPU memory through a heterogeneous memory-aware cache layout and improves cache lookup throughput through GPU parallelism. However, due to its single-GPU implementation, the PCIe inlet bandwidth of a single GPU is only 20GB / s. This limitation significantly constrains the maximum throughput of the system.

[0010] In the context of a graph neural network training system based on a single machine disk, the current system only has a single GPU implementation, and the overall performance of the system is directly limited by the performance ceiling of a single GPU. For example, in the Helio system, the PCIe inlet bandwidth of a single GPU is only 20GB / s. This limitation significantly constrains the maximum throughput of the system. By implementing a multi-GPU architecture, the throughput limitation of a single GPU can be broken through and the overall system performance can be significantly improved. However, due to the diverse PCIe structures of different machines and the numerous communication link combinations under each structure, there is a lot of room for solving communication planning in the scenario of multiple GPUs and multiple SSDs. Therefore, there are the following key issues that need to be addressed in this field:

[0011] 1. The search space for hardware placement and PCIe traffic scheduling under a large number of hardware combinations is very large. The PCIe structures of different machines are diverse, and there are many combinations of communication links under each structure, so the solution space for communication planning is very large. The hardware structure of a single machine is generally connected by CPU, GPU, SSD through complex links such as PCIe switch, QPI, NVLink, etc., and there may be a large number of complex communication combinations. This complex connection method makes the solution space for communication planning very large.

[0012] 2. Considering the unbalanced nature of graph vertex degrees, it is very challenging to map the access traffic of CPU and SSD to the placement of vertex features. Summary of the invention

[0013] The purpose of the present invention is to solve the above-mentioned technical problems in the prior art and provide a graph neural network training acceleration method and system based on multiple GPUs and multiple SSDs.

[0014] The specific technical solutions adopted by the present invention are as follows:

[0015] In a first aspect, the present invention provides a graph neural network training acceleration method based on multiple GPUs and multiple SSDs, which is run on a server equipped with multiple GPUs and multiple SSDs, and includes:

[0016] S1. Read the physical architecture information of the server, model the communication link relationship between the CPU, GPU, and SSD as a communication topology diagram, and construct source points and sink points on the communication link diagram. Point the source point to each SSD and CPU, and point all GPUs to the sink point. Then test the bandwidth of each communication link between the CPU, GPU, and SSD to obtain the communication bandwidth of each communication link in the communication topology diagram.

[0017] S2. According to the modeled communication topology, the link process of GPU pulling data is converted into a single-source single-sink maximum flow problem. The Ford-Fulkerson augmenting path algorithm is used to solve the maximum flow problem with the goal of maximizing the flow, and the access flow allocation scheme for the two types of storage nodes, CPU memory and SSD, in the server is obtained.

[0018] S3. For the graph data set required for training the graph neural network, all graph vertices are sorted according to the access frequency of the graph vertices, and then storage nodes are allocated to the graph vertices one by one in order of access frequency from high to low. In the allocation process, it is necessary to dynamically calculate the current performance indicators of all storage nodes before each graph vertex is allocated, and allocate the graph vertex to the storage node with the best current performance indicator for storage, and record the mapping relationship between each graph vertex and the actual storage location;

[0019] S4. After completing the storage of all graph vertices in the graph data set, the server executes the batch training task of the graph neural network; and during the training process, after sampling a batch of graph vertices in each round, the feature information of the vertices must be read into the GPU memory from the storage node where each graph vertex is located according to the mapping relationship for training the current batch.

[0020] As a preferred embodiment of the first aspect above, in the server, the CPU, GPU and SSD are all communication-connected via a PCIe switch; and in the server, different CPUs are connected via QPI, and different GPUs are connected via NVLink or PCIe.

[0021] As a preferred embodiment of the above-mentioned first aspect, in S2, when the maximum flow problem is solved by the Ford-Fulkerson augmenting path algorithm, the solution objective is defined as maximizing the flow from the source point to the sink point, and the constraint condition is that the flow of each edge in the communication topology graph cannot exceed the communication bandwidth of this edge, and the incoming flow of each node except the source point and the sink point in the communication topology graph must be equal to the outgoing flow.

[0022] As a preferred embodiment of the first aspect, in S3, the specific allocation process of allocating graph vertices in the graph data set to storage nodes is:

[0023] S31, sorting the graph vertices in the graph data set according to the access frequency to form a graph vertex queue;

[0024] S32, taking out the graph vertex with the highest access frequency from the graph vertex queue as the vertex to be allocated, and creating a priority storage queue according to the performance sorting rules for the two types of storage nodes, CPU and SSD, on the server, and allocating the feature information of the vertex to be allocated in the graph data set to the storage node with available storage capacity and the highest priority in the priority storage queue for storage;

[0025] In the performance sorting rule, the priority of the CPU node is always higher than that of the SSD node, and the CPU and SSD storage nodes need to calculate the current performance index of each storage node respectively, and then sort them in the order of smaller performance index and higher priority; the performance index is the product of the first index value and the second index value; the first index value is the ratio of the number of graph vertices currently stored in the calculated storage node to the total number of graph vertices that can be stored in the calculated storage node, and the second index value is the ratio of the access frequency of the graph vertex to be allocated to the access traffic of the calculated storage node in the access traffic allocation scheme;

[0026] S32. Repeat S32 continuously to allocate the graph vertices in the graph vertex queue to the dynamically updated priority storage queue one by one until all graph vertices are allocated.

[0027] As a preferred embodiment of the above-mentioned first aspect, in the server, the total available storage capacity of all CPU nodes is set to a fixed percentage of the total size of the graph data set.

[0028] As a preferred embodiment of the above-mentioned first aspect, in S4, when the feature information of the vertices is read into the GPU memory, for the graph vertices stored on the CPU node, all GPUs in the server directly read the data from the CPU node in parallel through the physical communication link; for the graph vertices stored on the SSD node, all GPUs in the server read the data from the SSD node in parallel using the GPU Direct Storage technology.

[0029] As a preferred embodiment of the first aspect, when all GPUs in the server read data from the SSD node in parallel using the GPU Direct Storage technology, an IO stack solution under multi-GPU and multi-SSD initiated by the GPU is adopted, wherein:

[0030] After obtaining a batch of graph vertex lists, the GPU kernel generates NVMe commands for each graph vertex in the list in parallel based on the mapping relationship between the graph vertices and the actual storage locations, which is used to extract the corresponding vertex feature information from the SSD nodes storing the graph vertices in parallel; at the same time, each GPU thread can bind multiple I / O requests executed successively, and send NVMe commands through multiple SQ queues, thereby continuously extracting graph vertex feature information from the SSD; and each GPU in the server establishes independent SQ mapping relationships and CQ mapping relationships with different SSDs, and all GPUs are evenly assigned CQ queues and SQ queues, so that each GPU can directly initiate a request to access the SSD without the participation of the CPU.

[0031] In the second aspect, the present invention provides a graph neural network training acceleration system based on multiple GPUs and multiple SSDs, which includes a server equipped with multiple GPUs and multiple SSDs, and a training process control program is installed on the server. When the training process control program is executed by the processor, it can implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in any one of the first aspects above.

[0032] In a third aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in any one of the first aspects above can be implemented.

[0033] In a fourth aspect, the present invention provides a computer electronic device comprising a memory and a processor;

[0034] The memory is used to store computer programs;

[0035] The processor is used to implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in any one of the first aspects above when executing the computer program.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] The present invention models the complex communication links of a single-machine hardware structure, and establishes a mapping of access traffic to storage embedding after solving the optimal access traffic allocation plan for the storage node. In this way, the communication topology placement and traffic allocation plan that maximizes the multi-GPU PCIe throughput obtained through automated search is used to greatly improve the training throughput of the system and meet the training requirements for TB-level ultra-large-scale graphs.

[0038] The IO stack solution for multi-GPU and multi-SSD initiated by the GPU provided by the present invention can fully utilize the parallel processing capability of the GPU and the high throughput characteristics of the NVMe SSD to achieve efficient and low-latency multi-GPU and multi-SSD data access. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A schematic diagram of the steps of the graph neural network training acceleration method based on multiple GPUs and multiple SSDs;

[0040] Figure 2 A modeling diagram of a complex communication link in an embodiment of the present invention;

[0041] Figure 3 The communication link bandwidth under the Naive solution;

[0042] Figure 4 The communication link bandwidth under the solution of the present invention;

[0043] Figure 5 It is a comparison chart of the training time between the scheme of the present invention and GIDS;

[0044] Figure 6 This is a comparison chart of the training time between the four interpolation methods of the multi-card version of Helios and the optimal interpolation method of the solution of the present invention in the GPU2-GraphSage scenario;

[0045] Figure 7 This is a training time comparison chart between four interpolation methods of the multi-card version of Helios and the optimal interpolation method of the solution of the present invention in the GPU4-GAT scenario;

[0046] Figure 8 This is a comparison chart of the training time between the four interpolation methods of the multi-card version of Helios and the optimal interpolation method of the solution of the present invention in the GPU2-GAT scenario;

[0047] Fig. 9 This is a comparison chart of the training time between the four interpolation methods of the multi-card version of Helios in the GPU4-GraphSage scenario and the optimal interpolation method of the solution of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purpose, features and advantages of the present invention more obvious and easy to understand, the specific implementation mode of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in each embodiment of the present invention can be combined accordingly without conflicting with each other.

[0049] The present invention provides a method for accelerating graph neural network (GNN) training based on multiple GPUs and multiple SSDs. The acceleration method runs on a server equipped with multiple GPUs and multiple SSDs, wherein the server has its own CPU. Therefore, in a single-machine hardware, the CPU, GPU, and SSD are connected to each other through complex links such as PCIe switch, QPI, and NVLink, and there are a large number of complex communication combinations. This complex connection method makes the solution space for communication planning very large.

[0050] like Figure 1 As shown, in a preferred embodiment of the present invention, the specific implementation steps of the above-mentioned graph neural network training acceleration method based on multiple GPUs and multiple SSDs are shown in S1~S4. In this embodiment, the CPU, GPU and SSD in the above-mentioned server can all form a communication connection through a PCIe switch; and in the server, different CPUs are connected through QPI, and different GPUs are connected through NVLink or PCIe, which can be specifically determined according to the hardware conditions in the server. The specific implementation processes of S1~S4 are described in detail below.

[0051] S1. Read the physical architecture information of the server, model the communication link relationship between the CPU, GPU, and SSD as a communication topology diagram, and construct source points and sink points on the communication link diagram. Point the source point to each SSD and CPU, and point all GPUs to the sink point. Then test the bandwidth of each communication link between the CPU, GPU, and SSD to obtain the communication bandwidth of each communication link in the communication topology diagram.

[0052] In the traditional graph neural network training process, it is necessary to first sample nodes and then perform graph data training. In essence, the graph topology and feature information stored in the memory and SSD will be transmitted to the corresponding GPU first, and the GPU will perform model training after obtaining the sampling information of the graph. In this process, since the topology and feature information of the graph will be stored in the memory and each SSD in a dispersed manner in advance, data transmission blocking problems may occur at the same time under the complex links of a single machine, resulting in the inlet bandwidth of multiple GPUs not being fully utilized, which in turn leads to low system training throughput. Therefore, the above-mentioned S1 step of the present invention takes into account the diverse structures of PCIe of different machines and the numerous communication combination schemes under each structure, and the overall single-machine communication link can be modeled at the software level to facilitate the subsequent solution of the optimal access traffic distribution scheme.

[0053] In an example of the present invention, considering the data transmission bandwidth of different communication links and the read and write bandwidth of GPU to SSD, a dual-CPU server is taken as an example. Two A100 GPUs and 12 SSDs are installed on the server through PCIe switch. Different CPUs are connected through QPI, and different GPUs are connected through NVLink. The communication topology diagram of the single-machine server after modeling according to the above S1 is as follows: Figure 2 It should be noted that Figure 2 CPU1 and CPU2 are the two CPUs on the server, GPU0 and GPU1 are the two installed A100 GPUs, PLX node 1 and PLX node 2 represent the PLX bridge chips connected to the two A100 GPUs, the source node, sink node, two abstract temporary nodes and two intermediate nodes are all virtual nodes introduced for the convenience of modeling, and the communication bandwidth between any virtual node and other CPU and SSD actual nodes needs to be set to infinity (denoted as inf). Figure 2 The communication bandwidth of the communication link between two actual nodes needs to be determined through bandwidth testing.

[0054] S2. According to the modeled communication topology, the link process of GPU pulling data is converted into a single-source and single-sink maximum flow problem. The Ford-Fulkerson augmenting path algorithm is used to solve the maximum flow problem with the goal of maximizing flow, and the access flow allocation scheme for the two types of storage nodes, CPU memory and SSD in the server is obtained.

[0055] In the above step S2 of the present invention, according to the modeled communication topology, since the source and sink have been pre-set, the link process of the GPU pulling data can be converted into a single-source single-sink maximum flow problem, thereby maximizing the PCIe inlet bandwidth of the GPU. In an embodiment of the present invention, when the maximum flow problem is solved by the Ford-Fulkerson augmenting path algorithm, the solution goal is defined as maximizing the flow from the source to the sink, and the constraint condition is that the flow of each edge in the communication topology cannot exceed the communication bandwidth of this edge, and the inflow flow of each node in the communication topology except the source and sink must be equal to the outflow flow.

[0056] Specifically, in an embodiment of the present invention, the method for defining the maximum flow problem with a single source and a single sink is as follows:

[0057] Suppose the communication topology graph is N=(V,E), where V and E represent the node combination and edge set in the graph, respectively, and s and t are the source and sink points (s, t∈V) in the communication topology graph N. The capacity c of an edge (u,v) from node u to node v is denoted as c(u,v), which represents the maximum value of the flow that can pass through this edge, that is, the communication bandwidth measured by this edge. The flow of this edge (u,v) is denoted as f(u,v), and the above parameters in the communication topology graph N also need to comply with the following two constraints:

[0058] Constraint 1: For each edge (u,v)∈E, the capacity constraint must satisfy f(u,v)≤c(u,v). This constraint indicates that the flow of this edge cannot exceed its capacity.

[0059] Constraint 2: For each node v∈V other than the source and sink, ∑ u:(u,v)∈E f(u,v)=∑ u:(u,v)∈E f(v,u), where ∑ u:(u,v)∈E f(u,v) represents the inflow flow of node u, ∑ u:(u,v)∈E f(v,u) represents the outflow flow of node u. This constraint states that the sum of flows into a node v must be equal to the sum of flows out of this node.

[0060] The flow from the source to the sink is defined as: |f|=∑ v:(s,v)∈E f(s,u), where s is the source point of N. |f| represents the number of flows from the source point to the sink point. Therefore, the single-source single-sink maximum flow problem is to maximize |f|, that is, to plan the largest possible flow from the source point s to the sink point t, so as to maximize the PCIe inlet bandwidth of the GPU.

[0061] After the above maximum flow problem is constructed, the Ford-Fulkerson augmenting path algorithm can be used to solve the problem. The Ford-Fulkerson augmenting path algorithm belongs to the prior art, and its solution steps are as follows:

[0062] Step 1. Initialization

[0063] Let the initial flow of all edges f(u,v)=0, that is, there is no flow passing through any edge at the beginning.

[0064] Step 2. Find the augmenting path

[0065] In a residual network (i.e., a network after the edge capacity is adjusted to residual), starting from the source point s, find an augmenting path from s to t. If there is an augmenting path, find the minimum residual Δf of all edges on the path, which is the maximum flow that can be added along the path.

[0066] Step 3. Increase flow along the path

[0067] Increase the flow of all edges on the augmented path by Δf, that is, update the flow of each edge (u, v) on the path:

[0068] f(u,v)=f(u,v)+∆f

[0069] For the reverse edge (v,u), the reverse flow is also adjusted accordingly (this is done to handle more possible reverse flows):

[0070] f(v,u)=f(v,u)-∆f

[0071] Step 4. Update the residual

[0072] Update the residual of each edge in the residual network:

[0073] r(u,v)=c(u,v)-f(u,v)

[0074] If there is a reverse flow (i.e., f(u,v)<0), the corresponding reverse residual is also updated to the positive reverse flow value.

[0075] Step 5. Repeat steps 2 to 4, continuously searching for augmenting paths and increasing flow, until no augmenting path from s to t can be found.

[0076] Step 6. Ending conditions

[0077] When no augmenting path can be found, the maximum flow is the current flow value. The value of the maximum flow is equal to the sum of all flows from the source point s, that is:

[0078] |f|=∑ v:(s,v)∈E f(s,v)

[0079] After arranging the access pattern of the maximum flow, the size of the access flow of the CPU memory and each SSD storage node can be obtained, thereby obtaining the access flow distribution plan for the two types of storage nodes, CPU memory and SSD, in the server.

[0080] S3. For the graph data set required for training graph neural networks, all graph vertices are sorted according to the access frequency of the graph vertices, and then storage nodes are allocated to the graph vertices one by one in order of access frequency from high to low. During the allocation process, it is necessary to dynamically calculate the current performance indicators of all storage nodes before each graph vertex is allocated, and allocate the graph vertices to the storage nodes with the best current performance indicators for storage, and at the same time record the mapping relationship between each graph vertex and the actual storage location.

[0081] In the existing maximum flow model, only the inflow flow of each storage node can be obtained, but the actual amount of data stored in each storage node cannot be accurately known. In addition, the traditional storage allocation method lacks effective processing of data access heat, resulting in unbalanced load of storage nodes, especially when data with high access frequency is allocated to nodes with poor performance, which will reduce the overall performance of the system. In order to optimize this problem, the present invention proposes a "Skewness-aware" backpack loading algorithm in the above S3 step, which gives priority to the frequency of data access and allocates the hottest data samples (i.e., graph vertex data in the graph data set) to storage nodes with better performance, thereby achieving balanced distribution of data storage among multiple nodes. Specifically, the present invention provides a backpack loading algorithm (Skewness-awarePacking Algorithm) based on data access frequency, which allocates the data samples with the highest access frequency to CPU nodes first, and then allocates the remaining data samples to SSD nodes, and also comprehensively considers the sample access frequency and node performance when allocating CPU and SSD nodes. The algorithm can not only perform balanced allocation in terms of storage capacity, but also optimize the system load, thereby improving the storage and access efficiency of the overall system.

[0082] It should be noted that the data access frequency in the present invention may also be referred to as access heat, which refers to the total number of times a single data sample is accessed when all training sets in the graph dataset are multi-hop sampled.

[0083] In an embodiment of the present invention, in the skewness-aware packing algorithm based on data access heat in step S3, the specific allocation process of allocating graph vertices in the graph data set to storage nodes is:

[0084] S31. Sort the graph vertices in the graph data set according to access frequency to form a graph vertex queue.

[0085] In actual implementation, the graph vertices in the graph data set can be sorted in descending order according to the access frequency, so that the graph vertex queue can be taken out one by one to allocate storage locations, and the front vertex in the graph vertex queue is always the vertex with the highest access frequency.

[0086] S32. Take out the graph vertex with the highest access frequency from the above-mentioned graph vertex queue as the vertex to be allocated, and create a priority storage queue according to the performance sorting rules for the two types of storage nodes, CPU and SSD, on the server, and allocate the feature information of the vertex to be allocated in the graph data set to the storage node with available storage capacity and the highest priority in the priority storage queue for storage.

[0087] It should be noted that there are two types of storage nodes in the present invention, CPU and SSD. The CPU node has higher performance and can be used to prioritize the most frequently accessed data. The SSD node has lower performance than the CPU node and can be used to store relatively cold data samples. Therefore, when the present invention allocates storage locations to the graph vertices in the graph vertex queue one by one, it is necessary to sort the two types of storage nodes, and allocate the hottest graph vertices to the storage nodes with the best performance to maximize the overall transmission efficiency.

[0088] In an embodiment of the present invention, an optimized performance sorting rule is provided, and its specific practice is: setting the priority of the CPU node to always be higher than the SSD node, and after calculating the current performance index of each storage node separately within the two types of storage nodes, CPU and SSD, the nodes are sorted within the class in the order of smaller performance index and higher priority.

[0089] Therefore, the nodes at the front of the initialized priority storage queue are all CPU nodes, ensuring that the graph vertices with the highest access frequency in the graph vertex queue will be assigned to the CPU node. After the storage capacity of the CPU node is used up, the remaining graph vertices will be stored in the SSD node. Moreover, for the same type of CPU nodes or the same type of SSD nodes, since each has its own fixed available storage capacity and access traffic, the performance of each node in the same type is also different. Graph vertices with higher access frequency need to be assigned to storage nodes with better performance to maximize the overall transmission efficiency.

[0090] However, it should be noted that the feature information of each vertex to be allocated can only be allocated to the storage node with available storage capacity and the highest priority in the priority storage queue for storage. If the available storage capacity of a storage node is not enough to store the feature information of the vertex to be allocated, it needs to be excluded from the nodes that can be stored. Therefore, the storage node that can be allocated is the storage node with the highest priority among the storage nodes with available storage capacity in the priority storage queue, and is not necessarily the storage node with the highest priority in the priority storage queue.

[0091] In an embodiment of the present invention, the above performance index is set as the product of a first index value X1 and a second index value X2. The first index value X1 is the ratio of the number of graph vertices currently stored in the calculated storage node to the total number of graph vertices that the calculated storage node can store. The above second index value X2 is the ratio of the access frequency of the graph vertex to be allocated to the access traffic of the calculated storage node in the access traffic allocation scheme.

[0092] It should be noted that the number of graph vertices currently stored in the calculated storage node is constantly changing as the allocation proceeds. Therefore, after each successful allocation of a graph vertex, the number of allocated graph vertices of all nodes needs to be updated, and the order of storage nodes in the priority storage queue needs to be readjusted according to the current storage capacity load of the node to ensure that the storage node that best suits the current sample can be selected first during the next allocation.

[0093] S32. Repeat S32 continuously to allocate the graph vertices in the graph vertex queue to the dynamically updated priority storage queue one by one until all graph vertices are allocated.

[0094] It should also be noted that in the above server, in order to control the space occupied by the vertex feature information in the graph data set on the CPU memory, a fixed percentage can be set in advance, and the total available storage capacity of all CPU nodes can be set to this fixed percentage of the total size of the graph data set. The specific percentage value can be reasonably optimized according to the actual situation, and is preferably set to 1%. As a result, the hottest 1% of the graph vertices in the graph vertex queue will be assigned to the CPU node, and the rest of the graph vertices will be assigned to the CPU node.

[0095] S4. After completing the storage of all graph vertices in the graph data set, the server executes the batch training task of the graph neural network; and in the training process, after sampling a batch of graph vertices in each round, it is necessary to read the feature information of the vertices from the storage node where each graph vertex is located into the GPU memory according to the mapping relationship between the graph vertices and the actual storage location, for training the current batch.

[0096] In an embodiment of the present invention, in the above step S4, when the feature information of the vertices is read into the GPU memory, for the graph vertices stored on the CPU node, all GPUs in the server directly read the data from the CPU node in parallel through the physical communication link; for the graph vertices stored on the SSD node, all GPUs in the server use the GPU Direct Storage technology to read the data from the SSD node in parallel.

[0097] Traditionally, the GPU relies on the host CPU to complete data access to the SSD. Since the graph neural network training process requires fine-grained access to the topology and feature information of the nodes, if the traditional CPU-SSD data access mode is adopted, it will cause high CPU-GPU synchronization overhead, I / O traffic amplification and high CPU processing delay problems, resulting in low GPU PCIe bandwidth, which leads to low system throughput and slow training speed. In order to solve the above problems, in an embodiment of the present invention, when all GPUs in the above-mentioned server use GPU Direct Storage technology to read data from the SSD node in parallel, a multi-GPU and multi-SSD IO stack (IOStack) solution initiated by the GPU is adopted to make full use of GPU parallelism and maximize disk IO throughput. The IO stack solution is specifically divided into the following three parts:

[0098] 1. Thread-level parallel IO command submission

[0099] This part makes full use of the high parallel characteristics of the GPU, and the GPU kernel can submit multiple I / O requests at the same time. Specifically: After obtaining a batch of graph vertex lists, the GPU kernel generates NVMe commands for each graph vertex in the list in parallel based on the mapping relationship between the graph vertices and the actual storage locations, which is used to extract the corresponding vertex feature information from the SSD nodes storing the graph vertices in parallel. The specific generation method of NVMe commands is as follows: The command submission kernel first obtains the list of graph vertex IDs sampled in the current batch, obtains the access address field of the NVMe command corresponding to the ID according to the address information of each graph vertex ID assigned in the aforementioned mapping relationship between the graph vertices and the actual storage locations, and fills in the NVMe command for each vertex. The NVMe command can be used to extract the corresponding vertex features from the SSD. NVMe commands generally contain the logical block ID of the SSD and the GPU physical memory address used to store data.

[0100] 2. Multithreaded command binding and submission

[0101] Each GPU thread can bind multiple I / O requests to be executed in sequence, and send NVMe commands through multiple SQ (Submission Queue) queues, thereby continuously extracting graph vertex feature information from the SSD. It should be noted that each GPU thread can only execute one I / O request at a time, but it can bind multiple I / O requests to be executed in sequence. When one I / O request is executed, the next I / O request is executed immediately. This design can achieve thread-level parallel submission, greatly improving the system's I / O throughput. Binding multiple requests within a thread not only reduces the total number of threads required, but also ensures that enough NVMe commands are submitted to maximize disk I / O throughput.

[0102] 3. Data access optimization for multiple GPUs and multiple SSDs

[0103] Each GPU in the server establishes independent SQ (Submission Queue) mapping relationships and CQ (Completion Queue) mapping relationships with different SSDs. All GPUs are evenly assigned CQ queues and SQ queues by the system, so that each GPU can directly initiate access requests to the SSD without the involvement of the CPU. Since each GPU can directly access multiple SSDs, the GPU actively initiates access to the SSD without the involvement of the CPU, which can greatly reduce the bottleneck caused by traditional CPU data transfer.

[0104] Through this multi-GPU and multi-SSD IO stack solution design, the system can fully utilize the parallel processing capabilities of the GPU and the high throughput characteristics of the NVMe SSD to achieve efficient and low-latency multi-GPU and multi-SSD data access. It has the following three technical advantages:

[0105] 1. Maximize I / O throughput: Through thread-level parallel command submission and multi-request binding, the efficiency of disk I / O is improved and the system latency is reduced.

[0106] 2. Reduce CPU load: Transferring I / O tasks from the CPU to the GPU reduces the processing delay and load of the CPU, thereby increasing the operating speed of the overall system.

[0107] 3. Make full use of GPU and SSD resources: Direct data access between multiple GPUs and multiple SSDs reduces the PCIe bandwidth bottleneck and optimizes system resource utilization.

[0108] In order to better demonstrate the technical effect of the present invention, a server is taken as an example to compare the communication status of the training method shown in S1 to S4 of the present invention (referred to as the present invention solution) and the traditional Naive solution during large-scale graph neural network training. In this server, two A100 GPUs are installed in the same PLX, and 12 SSD cards form a communication topology through complex links such as PCIe, QPI, and NVLink. The effect achieved by the Naive solution is as follows: Figure 3 As shown, the effect achieved by the solution of the present invention is as follows Figure 4 As shown in the figure, the comparison shows that the solution of the present invention significantly improves the bandwidth of the PCIe entrances of the two GPUs and has a higher communication transmission efficiency.

[0109] Similarly, in order to better demonstrate the technical effect of the present invention, a performance test experiment was conducted on two servers (Server A has 4 A100 GPUs and 8 SSDs connected via PCIe and QPI, and Server B has 2 A100 GPUs and 8 SSDs connected via PCIe and QPI), considering GraphSage and GAT GNN models for 2-hop sampling. The four graph datasets used in the experiment (Papers100M, IGB, UK, ClueWeb) are shown in Table 1:

[0110] Table 1

[0111] Graph Dataset Metrics Papers100M IGB UK ClueWeb Number of vertices 111M 269M 0.79B 1B Number of edges 1.6B 4B 47.2B 42.5B Graph topology storage size 14GB 34GB 384GB 348GB Feature Dimension 128 1024 1024 1024 Feature storage size 56GB 1.1TB 3.2TB 4.1TB

[0112] Since different insertion methods of GPU and SSD will affect the experimental results, this experiment considers four more common insertion methods: GPU balance-SSD front board, GPU balance-SSD balance, GPU with PLX-SSD front board, GPU with PLX-SSD balance, and then search to determine the optimal insertion method.

[0113] First, the training time (Epoch Times) of the solution of the present invention (denoted as Ours) and the GIDS in the prior art are compared under the more common interpolation method of GPU balancing-SSD balancing. Figure 5 As shown, Scheme 1, Scheme 2, Scheme 3, Scheme 4, Scheme 5, Scheme 6, Scheme 7, and Scheme 8 represent eight different schemes, namely GIDS-GPU2-GAT, Ours-GPU2-GAT, GIDS-GPU2-GraphSage, Ours-GPU2-GraphSage, GIDS-GPU4-GAT, Ours-GPU4-GAT, GIDS-GPU4-GraphSage, and Ours-GPU4-GraphSage. GIDS-GPU2-GAT and Ours-GPU2-GAT respectively represent the methods of GIDS and the solution of the present invention for training the GAT model on server A with 2 GPUs, GIDS-GPU4-GAT and Ours-GPU4-GAT respectively represent the methods of GIDS and the solution of the present invention for training the GAT model on server B with 4 GPUs, GIDS-GPU2-GraphSage and Ours-GPU2-GraphSage respectively represent the methods of GIDS and the solution of the present invention for training the GAT model on server A with 2 GPUs, GIDS-GPU4-GraphSage and Ours-GPU4-GraphSage respectively represent the methods of GIDS and the solution of the present invention for training the GraphSage model on server B with 4 GPUs. Figure 5It can be seen from the results that GIDS will have OOM problems on the UK and ClueWeb datasets, resulting in the inability to perform training, while the present invention will not have this problem and has better performance.

[0114] Secondly, since the present invention can build a communication topology graph for a single server in advance, the present invention can search for all the plug-in methods of GPU and SSD on the server based on the communication topology graph, and determine their traffic distribution schemes respectively, and finally perform performance tests to determine the optimal plug-in method. In four scenarios, GPU2-GraphSage (training the GraphSage model on server A with 2 GPUs), GPU4-GAT (training the GAT model on server B with 4 GPUs), GPU2-GAT (training the GAT model on server A with 2 GPUs), and GPU4-GraphSage (training the GraphSage model on server B with 4 GPUs), the multi-card version of Helios is respectively compared with the optimal plug-in method and traffic distribution scheme obtained by searching in four different plug-in methods (GPU balanced-SSD front board, GPU balanced-SSD balanced, GPU with PLX-SSD front board, GPU with PLX-SSD balanced, respectively denoted as comparison scheme 1, comparison scheme 2, comparison scheme 3, and comparison scheme 4) and the solution of the present invention (referred to as the present solution in the figure), and the training time (Epoch Times) is compared. The comparison results in the four scenarios are as follows: Figure 6 , Figure 7 , Figure 8 and Fig. 9 As shown in the figure, it is obvious that the solution of the present invention (Ours) achieves better results than the multi-card version of Helios in all four scenarios, and the training time of a single Epoch is shorter.

[0115] In addition, based on the same inventive concept, in another embodiment of the present invention, a graph neural network training acceleration system based on multiple GPUs and multiple SSDs is provided, which includes a server equipped with multiple GPUs and multiple SSDs, and a training process control program is installed on the server. When the above training process control program is executed by the processor, it can implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in the aforementioned embodiment scheme.

[0116] It should be noted that the training process control program of the above-mentioned graph neural network (GNN) is the core part of the entire architecture, responsible for the efficient training of graph data distributed in multiple GPUs and SSDs. Unlike the traditional single-GPU or CPU-dominated training method, the multi-GPU multi-SSD architecture adopted by the training process control program greatly improves the system's processing ability for large-scale graph data. As mentioned above, during the data transmission process, the training process control program converts the access link of the GPU and SSD into a single-source single-sink maximum flow problem through maximum flow modeling, so that the bandwidth utilization of data transmission is maximized. At the same time, through the Ford-Fulkerson augmenting path algorithm, the system can find the optimal data transmission path after each update to ensure that the PCIe inlet bandwidth of the GPU is fully utilized, thereby improving the throughput of the overall system. In the actual allocation of storage data, the Skewness-aware backpack loading algorithm ensures that the hottest samples are stored in the CPU node or the SSD with better performance by sorting the access frequency of samples, while the cold data is allocated to the remaining storage nodes. This algorithm not only achieves load balancing in terms of storage capacity, but also combines the dynamic changes of data access in the system to achieve a complex balance between communication links and storage nodes.

[0117] It should be noted that the method steps shown in S1 to S4 above can essentially be implemented in the form of a computer program.

[0118] Therefore, based on the same inventive concept, the present invention also provides a computer electronic device corresponding to the graph neural network training acceleration method based on multiple GPUs and multiple SSDs provided in the above embodiment, which includes a memory and a processor;

[0119] The memory is used to store computer programs;

[0120] The processor is used to implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described above when executing the computer program;

[0121] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for a computer device to execute all or part of the steps of the method described in each embodiment of the present invention.

[0122] Therefore, based on the same inventive concept, the present invention provides a computer-readable storage medium corresponding to a graph neural network training acceleration method based on multiple GPUs and multiple SSDs, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it can implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described above.

[0123] It is understandable that the above-mentioned memory, storage medium and processor are corresponding hardware devices adapted in the server, and the specific models are not limited as long as they can realize the corresponding functions.

[0124] The above-described embodiments are only some preferred implementations of the present invention, but are not intended to limit the present invention. A person skilled in the relevant technical field may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent replacement or equivalent transformation falls within the protection scope of the present invention.

Claims

1. A graph neural network training acceleration method based on multi-GPU and multi-SSD, characterized in that: The acceleration method runs on a server equipped with multiple GPUs and multiple SSDs, including: S1. Read the physical architecture information of the server, model the communication link relationship between the CPU, GPU, and SSD as a communication topology diagram, and construct source points and sink points on the communication link diagram. Point the source point to each SSD and CPU, and point all GPUs to the sink point. Then test the bandwidth of each communication link between the CPU, GPU, and SSD to obtain the communication bandwidth of each communication link in the communication topology diagram. S2. According to the modeled communication topology, the link process of GPU pulling data is converted into a single-source single-sink maximum flow problem. The Ford-Fulkerson augmenting path algorithm is used to solve the maximum flow problem with the goal of maximizing the flow, and the access flow allocation scheme for the two types of storage nodes, CPU memory and SSD, in the server is obtained. S3. For the graph data set required for training the graph neural network, all graph vertices are sorted according to the access frequency of the graph vertices, and then storage nodes are allocated to the graph vertices one by one in order of access frequency from high to low. During the allocation process, it is necessary to dynamically calculate the current performance indicators of all storage nodes before each graph vertex is allocated, and allocate the graph vertex to the storage node with the best current performance indicator for storage, and record the mapping relationship between each graph vertex and the actual storage location; S4. After completing the storage of all graph vertices in the graph data set, the server executes the batch training task of the graph neural network; and during the training process, after sampling a batch of graph vertices in each round, the feature information of the vertices must be read into the GPU memory from the storage node where each graph vertex is located according to the mapping relationship for training the current batch.

2. The method for accelerating graph neural network training based on multiple GPUs and multiple SSDs as claimed in claim 1, characterized in that: In the server, the CPU, GPU and SSD are all connected to each other through a PCIe switch; and in the server, different CPUs are connected through QPI, and different GPUs are connected through NVLink or PCIe.

3. The method for accelerating graph neural network training based on multiple GPUs and multiple SSDs as claimed in claim 1, characterized in that: In S2, when the maximum flow problem is solved by the Ford-Fulkerson augmenting path algorithm, the solution objective is defined as maximizing the flow from the source point to the sink point, and the constraint condition is that the flow of each edge in the communication topology graph cannot exceed the communication bandwidth of this edge, and the incoming flow of each node in the communication topology graph except the source point and the sink point must be equal to the outgoing flow.

4. The method for accelerating graph neural network training based on multiple GPUs and multiple SSDs as claimed in claim 1, characterized in that: In S3, the specific allocation process of allocating graph vertices in the graph data set to storage nodes is: S31, sorting the graph vertices in the graph data set according to the access frequency to form a graph vertex queue; S32, taking out the graph vertex with the highest access frequency from the graph vertex queue as the vertex to be allocated, and creating a priority storage queue according to the performance sorting rules for the two types of storage nodes, CPU and SSD, on the server, and allocating the feature information of the vertex to be allocated in the graph data set to the storage node with available storage capacity and the highest priority in the priority storage queue for storage; In the performance sorting rule, the priority of the CPU node is always higher than that of the SSD node, and the CPU and SSD storage nodes need to calculate the current performance index of each storage node respectively, and then sort them in the order of smaller performance index and higher priority; The performance index is the product of the first index value and the second index value; The first index value is the ratio of the number of graph vertices currently stored in the calculated storage node to the total number of graph vertices that can be stored in the calculated storage node, and the second index value is the ratio of the access frequency of the graph vertices to be allocated to the access traffic of the calculated storage node in the access traffic allocation scheme; S32. Repeat S32 continuously to allocate the graph vertices in the graph vertex queue to the dynamically updated priority storage queue one by one until all graph vertices are allocated.

5. The method for accelerating graph neural network training based on multiple GPUs and multiple SSDs as claimed in claim 4, characterized in that: In the server, the total available storage capacity of all CPU nodes is set to a fixed percentage of the total size of the graph data set.

6. The method for accelerating graph neural network training based on multiple GPUs and multiple SSDs as claimed in claim 1, characterized in that: In S4, when the feature information of the vertices is read into the GPU memory, for the graph vertices stored on the CPU node, all GPUs in the server directly read the data from the CPU node in parallel through the physical communication link; for the graph vertices stored on the SSD node, all GPUs in the server read the data from the SSD node in parallel using the GPU Direct Storage technology.

7. The method for accelerating graph neural network training based on multiple GPUs and multiple SSDs as claimed in claim 6, characterized in that: When all GPUs in the server use the GPU Direct Storage technology to read data from the SSD node in parallel, an IO stack solution under multi-GPU and multi-SSD initiated by the GPU is adopted, wherein: After obtaining a batch of graph vertex lists, the GPU kernel generates NVMe commands for each graph vertex in the list in parallel based on the mapping relationship between the graph vertices and the actual storage locations, which is used to extract the corresponding vertex feature information from the SSD nodes storing the graph vertices in parallel; at the same time, each GPU thread can bind multiple I / O requests executed successively, and send NVMe commands through multiple SQ queues, thereby continuously extracting graph vertex feature information from the SSD; and each GPU in the server establishes independent SQ mapping relationships and CQ mapping relationships with different SSDs, and all GPUs are evenly assigned CQ queues and SQ queues, so that each GPU can directly initiate a request to access the SSD without the participation of the CPU.

8. A graph neural network training acceleration system based on multiple GPUs and multiple SSDs, characterized in that: It includes a server equipped with multiple GPUs and multiple SSDs, and a training process control program is installed on the server. When the training process control program is executed by a processor, it can implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: A computer program is stored on the storage medium. When the computer program is executed by the processor, it can implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in any one of claims 1 to 7.

10. A computer electronic device, characterized in that: including memory and processor; The memory is used to store computer programs; The processor is used to implement the graph neural network training acceleration method based on multiple GPUs and multiple SSDs as described in any one of claims 1 to 7 when executing the computer program.

Images