Historical embedded cache-based stand-alone graph neural network optimization method and system

Abstract

The invention discloses a single-machine graph neural network optimization method and system based on historical embedded caches, and the method comprises the steps: carrying out the pre-sampling analysis of a training data set, collecting sampling data, carrying out the calculation through combining with a heuristic method, obtaining the size of a historical embedded cache of each level, setting a cache according to the size of the historical embedded cache of each level, and executing the pre-sampling, cache replacement is simulated in each sampling process, sub-graph pruning is carried out, a node access sequence and a sub-graph topology connection relation are obtained, after a corresponding number of training sub-graphs are obtained through repeated sampling for multiple times, multiple steps of optimal feature cache replacement are calculated, and optimal feature cache replacement information is stored; and when the training sub-graph is loaded for actual training, historical embedded pushing / pulling is carried out in combination with the sampling sub-graph in each round of training, and after bottom-layer feature cache replacement is carried out in combination with the optimal feature cache replacement information, single-machine graph neural network optimization is realized.

Description

Technical Field

[0001] The present invention relates to a technology in the field of neural networks, specifically a single-machine graph neural network optimization method and system based on historical embedded cache. Background Art

[0002] When using a single machine to train large-scale graph data, existing neural network technology needs to set up feature caches in memory to alleviate the problem of high latency in accessing external memory and optimize training speed. Although the existing heuristic method combined with the optimal cache replacement algorithm can improve the feature cache hit rate to a certain extent, due to the random access characteristics of the graph data itself and the additional randomness introduced in the graph neural network training to meet the independent and identically distributed assumption of training samples, it still faces the defects of low cache hit rate, long data loading time, and low computational memory access ratio. Summary of the invention

[0003] In view of the problem that existing static graph-based technologies cannot be applied to dynamic sampling training scenarios and have low training efficiency in original training scenarios, the present invention proposes a single-machine graph neural network optimization method and system based on historical embedding cache. By setting a historical embedding cache and reusing historical embeddings in the graph neural network training process, the access to the underlying original feature data in the training process is effectively reduced, thereby improving the training efficiency without affecting the accuracy.

[0004] The present invention is achieved through the following technical solutions:

[0005] The present invention relates to a single-machine graph neural network optimization method based on historical embedding cache. The method collects sampling data by pre-sampling and analyzing the training data set, calculates the historical embedding cache size of each level in combination with a heuristic method, sets the cache according to the calculated historical embedding cache size of each level, and performs pre-sampling, that is, cyclically performs subgraph-based sampling: in each sampling process, cache replacement is simulated, subgraph pruning is performed to obtain node access sequences and subgraph topological connection relationships, and after repeated sampling for several times to obtain a corresponding number of training subgraphs, several steps of optimal feature cache replacement are calculated and the optimal feature cache replacement information is saved; when the training subgraph is loaded for actual training, historical embedding push / pull is performed in combination with the sampled subgraph in each round of training, and underlying feature cache replacement is performed in combination with the optimal feature cache replacement information, thereby realizing single-machine graph neural network optimization.

[0006] The single machine mentioned is a single server architecture consisting of a single server including a CPU, a GPU and a memory and an SSD external memory for storing the original graph data. During the training of the graph neural network, data is exchanged through the memory and the SSD external memory.

[0007] The pre-sampling analysis refers to: executing a round of sampling process according to the training nodes of the current batch, collecting the sampled sub-graph metadata obtained by sampling but not performing subsequent data loading and actual training process.

[0008] The sampled subgraph metadata includes the number of nodes NumNode in the i-th level of the subgraph i .

[0009] The execution pre-sampling simulates cache replacement through a multi-layer cache interactive replacement algorithm and obtains a node access sequence and prunes a subgraph through subgraph pruning, while calculating the optimal replacement information of the feature cache according to the node access sequence.

[0010] The subgraph pruning is performed by sampling and subgraph pruning in a cache directory obtained by pre-sampling simulation cache replacement.

[0011] The multi-layer cache interactive replacement algorithm is: information is transmitted between caches through sampling subgraphs, and neighbors of high-level cache nodes are avoided from appearing in low-level caches. The caches in each level use FIFO to perform cache replacement; when a high-level cache hits, it is removed from the sampling subgraph.

[0012] The cache replacement is further combined with the expiration threshold refresh, specifically: record and add batch timestamps when caching; make a judgment before each round of cache update, when the difference between the timestamp of the head element of the cache queue and the current timestamp exceeds the expiration threshold, first execute the expired cache eviction, and then perform the cache update, otherwise directly perform the cache update.

[0013] The simulated cache replacement means: simulated cache replacement is performed through the cache directory during the sampling process, and the number of nodes retained by each level of the cache directory is consistent with the historical embedded cache. The cache directory does not retain the actual data, but only retains the access node number. The cache directory is dynamically updated during the sampling process according to the multi-layer cache interactive replacement algorithm.

[0014] The loading of the training subgraph for actual training means: for a batch of training nodes, sampling recursively extracts the k-hop neighbors of the training nodes from the adjacency matrix of the graph; after sampling to obtain the node access sequence, data preparation is performed according to the access sequence (the original features of the nodes are loaded from the SSD external memory) and the prepared data is passed to the GPU, and the GPU performs the subsequent graph neural network calculation process.

[0015] The heuristic method is: combining the number of nodes NumNode at each level obtained by pre-sampling analysis i (i∈{1,2,…,k}), combined with the sampling fan-out parameter FanOut i(i∈{1,2,…,k}), historical embedding dimension HiddenFeatureDim, and GPU memory capacity limit B, calculate the size of the historical embedding cache at each level, including:

[0016] i) First configure the top-level history embedded cache (i=1) cache node number CacheNode i =NumNode i *CacheBatch, set the number of nodes cached by the lower layer history embedded cache CacheNode i =CacheNode i-1 *FanOut i *α(i∈{2,3,…,k}), finally calculate the estimated memory size used by the cache And compare it with the memory capacity limit B. When MemExpt≤B, the calculation process ends and the historical embedding cache is configured according to the calculation result; otherwise, execute step ii), where: α is the sampling neighborhood overlap;

[0017] ii) Calculate in sequence And find the first one that satisfies the requirement MemExpt k′ ≤B, configure history embedding according to the calculation result for levels less than or equal to k′; set the history embedding cache size to 0 for levels greater than k′; and return the configuration result.

[0018] The subgraph pruning means: when executing the sampling process, if the current sampling node hits in the corresponding cache directory, the hit status of the current sampling node is marked, and the current sampling node is removed from the subsequent recursive sampling; the above process is repeated for each level of sampling, and finally a pruned subgraph is obtained.

[0019] The historical embedding push / pull means: when executing the forward pass process of the neural network, if the embedding of the subgraph node is not calculated in real time from the original features (corresponding to the pre-sampling processing cache hit), it is necessary to pull the historical embedding of the corresponding node from the historical embedding cache and splice it into the corresponding tensor for forward pass at subsequent levels; for the miss embedding calculated in real time from the original features in the current round, it is necessary to push the calculated embedding value to the corresponding historical embedding cache to update the historical embedding cache. Technical Effects

[0020] The present invention collects and analyzes sampling data based on the pre-sampling analysis method, and calculates the multi-level historical embedded cache size according to the sampling data and training parameters; through multi-layer cache interactive replacement, the cache hit situation is obtained in real time in the pre-sampling process, and the hit node state is marked as a hit in time to avoid the influence of the neighbor nodes of the hit node on the next level of cache; in the pre-sampling process, the cache replacement is simulated by maintaining the cache directory, and the subgraph pruning is completed in combination with the multi-layer cache interactive replacement algorithm, and finally the number of bottom nodes in the subgraph is reduced, and the number of times the external storage is accessed is reduced. Compared with the prior art, the present invention can effectively reduce the size of the sampled subgraph, so as to reduce the amount of memory access in the training process, and significantly reduce the amount of calculation and memory access in the training process, thereby improving the training efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a flow chart of the present invention;

[0022] Figure 2 It is a schematic diagram of the pre-sampling analysis module;

[0023] Figure 3 This is a schematic diagram of the multi-layer cache interactive replacement process;

[0024] Figure 4 This is a graph showing the reduction in training time for this system. DETAILED DESCRIPTION

[0025] The present embodiment relates to a graph neural network optimization system based on a single-machine out-of-core architecture, including: a pre-sampling analysis module, a pre-sampling simulation cache replacement module and a trainer history embedding update module, wherein: the pre-sampling analysis module performs heuristic method calculation processing based on the sampled subgraph metadata information obtained by running the sampling, and obtains the historical embedding cache size result of each level; the pre-sampling simulation cache replacement module performs multi-layer cache interactive replacement algorithm processing based on the pre-sampling subgraph and cache directory status information, and obtains the pruned subgraph result to be trained; the trainer history embedding update module performs history embedding push / pull processing based on the pruned subgraph information to obtain the final training result.

[0026] like Figure 1 As shown, a graph neural network optimization method based on the above system is performed, in which the pre-sampling analysis module calculates the historical embedding cache size configuration of each level and then performs pre-sampling, that is, while cyclically performing sub-graph-based sampling, the pre-sampling process simulates cache replacement, and performs pre-sampling process sub-graph pruning in combination with cache information to obtain pruned training sub-graphs. After repeated several times to obtain the corresponding number of training sub-graphs, several steps of optimal feature cache replacement are calculated and cache replacement information is calculated; in the training stage, the training sub-graphs are loaded for actual training, and the historical embedding update module is used to push / pull the historical embedding and update the feature cache in each round of training.

[0027] The pre-sampling specifically includes:

[0028] Step 1: Maintain a cache directory for each layer of sampling, and initialize each cache directory to be empty at the beginning. In each training round, according to the training nodes of the current round and the cache directory status of the current layer, the training nodes are sampled based on the subgraph combined with the pre-sampling subgraph pruning.

[0029] The sampling algorithm described herein adopts but is not limited to a reservoir sampling algorithm.

[0030] Step 2: After sampling is completed, for nodes that are not hit in the cache directory of the corresponding level, the cache directory is updated by simulating cache replacement through pre-sampling processing, and then the missed nodes are updated to the corresponding cache directory through a multi-layer cache interactive replacement algorithm to simulate the cache replacement during the actual training process, so that the newly generated node embedding can be used in subsequent rounds.

[0031] Step 3: Repeat steps 1 and 2 k times in total, and finally sample k node sequences node_idx and k node adjacency matrices adjs. Run the optimal cache replacement algorithm based on the access information of node_idx in round k to sample the future access data obtained in advance, obtain the feature cache replacement information of each round and save it on the SSD for use in the training phase.

[0032] The optimal cache replacement algorithm means that the data blocks with the furthest future access time are replaced first during actual execution. Since the sampling obtains the subsequent access data in advance, it can pass through the optimal cache.

[0033] The feature cache replacement information includes: the newly replaced feature node number in_id, the replaced feature node number out_id and the position in_position where the replaced feature is stored.

[0034] The training phase specifically includes:

[0035] Step a, load the node sequence node_idx and node adjacency matrix adjs obtained in the pre-training process, and prepare the training data according to the current feature cache, specifically: for node n∈node_idx, if the feature of n is in the current feature cache, it is directly loaded from the cache, otherwise the feature of n is loaded from the SSD, and finally the original feature tensor batch_input is obtained, and the original feature tensor batch_input and the node adjacency matrix adjs are input into the neural network to be optimized for forward propagation-based training.

[0036] Step b: Push / pull historical embeddings during the training process. If the corresponding embedding value does not hit the historical embedding cache, the newly calculated embedding needs to be updated to the historical embedding cache.

[0037] The update described above, when the historical embedding cache is full, removes the embedding with the longest generation time in the cache according to the FIFO principle to reduce the lag of the historical embedding used, thereby ensuring the convergence and accuracy of training.

[0038] The historical embedding cache miss means that the node historical embedding in the corresponding position of the historical embedding cache is empty, indicating that the corresponding node has not been visited in the previous training round or has been evicted and replaced.

[0039] Step c: After the forward propagation is completed, the feature cache replacement information calculated in the pre-sampling phase is loaded, and the feature cache is updated according to the feature cache replacement information.

[0040] Step d: Repeat the above steps a, b and c for k times to complete one super batch training.

[0041] Step e: Delete the node sequence node_idx, node adjacency matrix adjs and feature cache replacement information saved on the SSD.

[0042] Through specific experiments, a Linux server equipped with a 40GB A100 GPU, where A100 contains 108 streaming multiprocessors (SMs), each with 64 FP32 cores. The PCIe type is PCI-E 4.0×16, with a maximum bandwidth of 31.5GB / s. The server is equipped with an Intel(R) Xeon(R) Silver 4310 CPU and 128GB host memory, and a 3.84TB Samsung Pm9a3 NVMe SSD hard drive. Pm9a3 can reach 200K input and output times per second (IOPS) and 6,900MB / s sequential read speed. Multiple real-world graph training datasets (including node labels, used for performance testing and accuracy measurement) are used as workloads, including ogbn-papers100M, ogbn-products and IGB-medium, as well as multiple real-world graph datasets (excluding node labels, as performance verification), including twitter and friendster.

[0043] like Figure 4Shown is a comparison chart of the system training performance of this embodiment. This embodiment and the baseline system that does not use historical embedding cache simultaneously perform graph neural network training on different graph data sets on a physical machine, and measure the training time for comparison. After measurement, the three data sets can all achieve a high training acceleration effect on this system. Compared with the baseline system that does not use historical embedding cache, the data set ogbn-papers100M in this embodiment can achieve a 18.3% reduction in training time and maintain the same training accuracy as the baseline; the data set ogbn-products can achieve a 23.4% reduction in training time, and the data set igb-medium can achieve a 20% reduction in training time.

[0044] Compared with the prior art, the pre-sampling simulated cache replacement module of this method updates the cache directory by simulating real cache replacement, and combines pre-sampling processing subgraph pruning to obtain pruned subgraphs, which reduces the number of original features that need to be accessed for training pruned subgraphs. It can achieve higher training efficiency and shorter end-to-end training delay without reducing training accuracy. Compared with related single-machine out-of-core graph neural network training systems, it can reduce training time by 15% and end-to-end training delay by 8%.

[0045] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A graph neural network optimization method based on a single-machine out-of-core architecture, characterized in that: By pre-sampling and analyzing the training data set to collect sampling data, and combining heuristic methods to calculate the historical embedding cache size of each level, the cache is set according to the calculated historical embedding cache size of each level, and pre-sampling is performed, that is, subgraph-based sampling is performed cyclically: cache replacement is simulated during each sampling process, subgraph pruning is performed to obtain node access sequences and subgraph topological connection relationships, and after repeated sampling for several times to obtain the corresponding number of training subgraphs, several steps of optimal feature cache replacement are calculated and the optimal feature cache replacement information is saved; when loading the training subgraph for actual training, historical embedding push / pull is performed in combination with the sampled subgraph in each round of training, and the underlying feature cache is replaced in combination with the optimal feature cache replacement information to achieve single-machine graph neural network optimization; The pre-sampling analysis refers to: executing a round of sampling process according to the training nodes of the current batch, collecting the sampled subgraph metadata obtained by sampling but not performing subsequent data loading and actual training process; The execution pre-sampling simulates cache replacement through a multi-layer cache interactive replacement algorithm and obtains a node access sequence and a pruned subgraph through subgraph pruning, while calculating the optimal replacement information of the feature cache according to the node access sequence; The subgraph pruning is performed by sampling and subgraph pruning in a cache directory obtained by pre-sampling simulation cache replacement.

2. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 is characterized in that: The multi-layer cache interactive replacement algorithm refers to: information is transmitted between caches through sampling subgraphs, neighbors of high-level cache nodes are avoided from appearing in low-level caches, and caches in each level are replaced using FIFO; when a high-level cache hits, it is removed from the sampling subgraph.

3. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 or 2 is characterized in that: The cache replacement is further combined with the expiration threshold refresh, specifically: record and add batch timestamps when caching; make a judgment before each round of cache update, when the difference between the timestamp of the head element of the cache queue and the current timestamp exceeds the expiration threshold, first execute the expired cache eviction, and then perform the cache update, otherwise directly perform the cache update.

4. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 is characterized in that: The simulated cache replacement refers to: performing simulated cache replacement through the cache directory when executing the sampling process. The number of nodes retained in each level of the cache directory is consistent with the historical embedded cache. The cache directory does not retain the actual data, but only retains the access node number. During the sampling process, the cache directory is dynamically updated according to the multi-layer cache interactive replacement algorithm.

5. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 is characterized in that: The loading of the training subgraph for actual training means: for a batch of training nodes, sampling recursively extracts the k-hop neighbors of the training nodes from the adjacency matrix of the graph; after sampling to obtain the node access sequence, data preparation is performed according to the access sequence (the original features of the nodes are loaded from the SSD external memory) and the prepared data is passed to the GPU, and the GPU performs the subsequent graph neural network calculation process.

6. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 is characterized in that: The heuristic method is: combining the number of nodes NumNode at each level obtained by pre-sampling analysis i (i∈{1,2,…,k}), combined with the sampling fan-out parameter FanOut i (i∈{1,2,…,k}), historical embedding dimension HiddenFeatureDim, and GPU memory capacity limit B, calculate the size of the historical embedding cache at each level, including: i) First, configure the top-level historical embedded cache (i=1) with the number of cache nodes CacheNodei=NumNodei*CacheBatch, and set the number of nodes cached by the lower-level historical embedded cache CacheNode i =CacheNode i-1 *FanOut i *α(i∈{2,3,…,k}), finally calculate the estimated memory size used by the cache And compare it with the memory capacity limit B. When MemExpt≤B, the calculation process ends and the historical embedding cache is configured according to the calculation result; otherwise, execute step ii), where: α is the sampling neighborhood overlap; ii) Calculate in sequence And find the first one that satisfies the requirement MemExpt k′ ≤B, configure history embedding according to the calculation result for levels less than or equal to k′; set the history embedding cache size to 0 for levels greater than k′; and return the configuration result.

7. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 is characterized in that: The subgraph pruning means: when executing the sampling process, if the current sampling node hits in the corresponding cache directory, the hit status of the current sampling node is marked, and the current sampling node is removed from the subsequent recursive sampling; the above process is repeated for each level of sampling, and finally a pruned subgraph is obtained.

8. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 is characterized in that: The historical embedding push / pull means: when executing the forward pass process of the neural network, if the embedding of the subgraph node is not calculated in real time from the original features (corresponding to the pre-sampling processing cache hit), it is necessary to pull the historical embedding of the corresponding node from the historical embedding cache and splice it into the corresponding tensor for forward pass at subsequent levels; for the miss embedding calculated in real time from the original features in the current round, it is necessary to push the calculated embedding value to the corresponding historical embedding cache to update the historical embedding cache.

9. The graph neural network optimization method based on a single-machine out-of-core architecture according to claim 1 or 5, characterized in that: The loading of the training subgraph for actual training specifically includes: Step a: load the node sequence node_idx and node adjacency matrix adjs obtained in the pre-training process, and prepare the training data according to the current feature cache. Specifically, for node n∈node_idx, if the feature of n is in the current feature cache, it is directly loaded from the cache, otherwise the feature of n is loaded from the SSD, and finally the original feature tensor batch_input is obtained. The original feature tensor batch_input and the node adjacency matrix adjs are input into the neural network to be optimized for training based on forward propagation. Step b: Push / pull historical embeddings during the training process. If the corresponding embedding value does not hit the historical embedding cache, the newly calculated embedding needs to be updated to the historical embedding cache. The update, when the historical embedding cache is full, removes the embedding with the longest generation time in the cache according to the FIFO principle to reduce the hysteresis of the historical embedding used, and ensures the training convergence and training accuracy; The historical embedding cache miss means that the node historical embedding is empty at the corresponding position of the historical embedding cache, indicating that the corresponding node has not been visited in the previous training round or has been evicted and replaced; Step c: after the forward propagation is completed, the feature cache replacement information calculated in the pre-sampling phase is loaded, and the feature cache is updated according to the feature cache replacement information; Step d, repeat the above steps a, b and c for k times to complete one super batch training; Step e: Delete the node sequence nose_idx, node adjacency matrix adjs and feature cache replacement information saved on the SSD.

10. A graph neural network optimization system based on a single-machine out-of-core architecture for implementing any of the methods described in claims 1-9, characterized in that: include: Pre-sampling analysis module, pre-sampling simulation cache replacement module and trainer history embedding update module, wherein: the pre-sampling analysis module performs heuristic calculation and processing based on the sampled subgraph metadata information obtained by running sampling to obtain the result of the history embedding cache size at each level; The pre-sampling simulation cache replacement module performs multi-layer cache interactive replacement algorithm processing according to the pre-sampling subgraph and cache directory status information to obtain the pruned subgraph result to be trained; the trainer history embedding update module performs history embedding push / pull processing according to the pruned subgraph information to obtain the final training result.

Images