A method and system for processing random walks of out-of-core graphs based on locality perception

Abstract

The present invention discloses a method and system for processing random walks of out-of-core graphs based on locality perception, which is applicable to first-order and second-order graph random walk processes. In the later stage of graph random walk, the divided subgraphs can be dynamically divided in a finer granularity based on spatiotemporal locality, and only the dynamically divided subgraphs are loaded into the memory each time to reduce the data transmission overhead between the disk and the memory. When the number of walkers is 10% remaining, the system uses temporal locality to combine the top 1% of nodes most frequently visited during the walks of the first 90% of walkers into a special subgraph and load it into the memory to complete a random walk. When the special subgraph is executed, the subgraph where the walkers are located is taken as the starting point, and a two-layer BFS strategy is used to perform a finer dynamic division of the subgraph where the walkers are located, and a smaller dynamic subgraph is loaded into the memory each time. The construction of the dynamic subgraph is completed by Smart SSD7, and the random walk process is carried out in the CPU, so that the subgraph division and the random walk process are guaranteed to be executed concurrently.

Description

Technical Field

[0001] The present invention relates to the technical field of computer systems, and in particular to a method and system for processing random walks of out-of-core graphs based on locality perception. Background Art

[0002] Random walk, also known as random walk, is a mathematical statistical model first proposed by Karl Pearson in 1905. It depicts a series of random trajectories, in which each step of movement is random and not subject to specific rules. This model can be used to represent irregular changes, similar to the random process record formed by an individual's drunken steps. Random walk can be regarded as a special case of Markov chain or Markov process, that is, each step of movement is only related to the current position, but not to the past position, showing the characteristics of "no memory". As a powerful mathematical tool, the random walk model has a wide range of applications in many fields such as physics, chemistry, biology, computer science, finance, and personalized recommendation technology. It not only helps us understand and simulate random processes in nature, but also provides us with new ideas and methods to solve complex problems.

[0003] Random walks are usually performed on a directed or undirected graph. Random walk models are divided into two categories: first order and second order. For a first order random walk, a walker starts from a node and randomly selects a neighbor node of the node as its next hop. If the specified length of the walk or other specific conditions are reached, the walk ends. For a second order random walk, the choice of a walker's next hop depends not only on itself, but also on the node of the walker's previous hop.

[0004] As the scale of graph data continues to expand, the memory of a single-machine system can no longer fully accommodate the entire graph. To solve this problem, there are usually two solutions: one is to use distributed computing to divide the graph into subgraphs and store them on each node of the distributed cluster; the other is to divide the graph into subgraphs and store them on a disk with a larger storage capacity, and call the subgraphs into or out of the memory as needed. Due to the high complexity and difficulty of distributed computing, current research mostly uses the second solution on a single-machine system to implement random walk operations on large graphs, namely out-of-core random walks. Since out-of-core random walks need to frequently call the divided subgraphs from the disk into and out of the memory, I / O performance becomes its main bottleneck. Currently, there are two main solutions for out-of-core random walks: synchronous random walks and asynchronous random walks. Each method can be further divided into one-stage and two-stage random walks. Therefore, in a stand-alone system, the implementation models of out-of-core random walks can be summarized into the following four types: a synchronous one-stage random walk model, a synchronous two-stage random walk model, an asynchronous one-stage random walk model, and an asynchronous two-stage random walk model.

[0005] The synchronous one-stage random walk model first divides the entire graph into several subgraphs and stores them on disk. Then, each subgraph is loaded into memory in turn to check whether there is an active node in the subgraph. If there is an active node, a random walk is performed from each active node, and then the subgraph is written back to disk. If there is no active node in the subgraph, it is directly replaced back to disk. This process continues until there are no active nodes in all subgraphs and the random walk ends. This synchronous random walk mode has obvious inefficiency for two main reasons: First, the walker on each active node can only move forward one step in each walk, so if the walker needs to jump out of the current subgraph, it needs to schedule the subgraph from disk to memory multiple times. Second, each time a subgraph is loaded into memory, it is done in the order of subgraph numbers, and even if there are no active nodes in the subgraph, it still needs to be rescheduled.

[0006] The only difference between the synchronous two-stage random walk model and the one-stage model is that the second stage needs to schedule multiple subgraphs from the disk. The synchronous walk model is very inefficient, which leads to the asynchronous random walk model.

[0007] Based on the asynchronous mode of one-stage random walk model, the entire graph is first divided into several subgraphs and stored in the disk. Then, each time the subgraph with the largest number of active nodes is selected from the disk as a candidate block and loaded into the memory. In the memory, the active nodes in the subgraph are walked asynchronously, that is, the walker on each active node can continuously perform multiple steps of walks until it jumps out of the subgraph or meets the end condition of the walk.

[0008] The two-stage random walk model based on asynchronous mode first divides the entire graph into several subgraphs and stores them on disk. In this model, the next walk of the walker on each active node depends not only on the current node, but also on the previous hop node of the walker. If the previous hop node is not in the memory, the walker on the active node cannot perform the next walk. Therefore, the goal of the two-stage random walk in asynchronous mode is to maximize the number of valid active nodes loaded into the memory, and walk in an asynchronous manner on this basis. Compared with the synchronous mode, the asynchronous mode significantly improves the I / O efficiency, but the I / O ratio still exceeds 60%, which is still the main bottleneck limiting the performance of out-of-core random walks.

[0009] In summary, in the existing out-of-core random walk scheme, I / O is still the main bottleneck of the system. Research has found that walkers on more than 90% of active nodes can complete the random walk process in a very short time, while 60% to 90% of the total I / O time is mainly concentrated on less than 10% of active nodes. This situation leads to a significant long-tail effect, so there is an urgent need for a method to effectively deal with the I / O delay problem caused by these 10% active nodes. Summary of the invention

[0010] The purpose of the present invention is to address the deficiencies of the prior art and provide a locality-aware out-of-core graph random walk processing method and system to solve the I / O bottleneck problem caused by the frequent scheduling of subgraphs by the last 10% walkers during the out-of-core random walk.

[0011] The present invention is achieved through the following technical solutions:

[0012] A locality-aware out-of-core graph random walk processing method includes the following steps:

[0013] (1) During the first and second phases of the random walk, the graph is dynamically partitioned by utilizing the spatiotemporal locality characteristics of the late random walk, and only the dynamically partitioned subgraph is loaded into memory each time;

[0014] (2) When the number of random walkers is reduced to 10%, the top 1% of nodes most frequently visited by the first 90% of walkers are identified and extracted using temporal locality, and a subgraph is constructed and loaded into memory to complete a random walk.

[0015] (3) After processing the subgraph consisting of the first 1% of active nodes, in the subsequent random walk process, the subgraph where the walkers are located is dynamically divided using spatial locality, that is, the subgraph is further decomposed and loaded into the memory for processing each time until all walkers complete their walk tasks.

[0016] Furthermore, the top 1% of nodes most frequently visited by the top 90% walkers during their walks are identified and extracted, specifically:

[0017] After the first 90% of walkers have completed their random walks, they need to record the number of visits to each node in the graph. After the walk is completed, all nodes in the graph are sorted from high to low according to the number of visits, and the top 1% of nodes with high visit frequency are selected to form a subgraph and stored in memory.

[0018] Furthermore, all nodes in the graph are sorted from high to low according to the number of visits, specifically:

[0019] Merge sort is used to sort the graph nodes from high to low according to the number of visits. Merge sort is a divide-and-conquer algorithm. First, the data is divided into two sub-arrays, which are sorted separately, and then the two sorted sub-arrays are merged into an ordered array.

[0020] Furthermore, the first 1% of the nodes with high access frequency are selected to form a subgraph, specifically: the extracted subgraph is stored in CSR format, and since there is a gap in the numbers of adjacent vertices in the subgraph, a hash array is used to establish a mapping relationship between the subgraph vertex ID and the index 0 to n-1;

[0021] First, extract the top 1% nodes with the highest frequency of access during the first 90% of the walks and arrange these nodes in order. Next, establish a one-to-one hash mapping relationship between each node ID and index 0 to n-1 to generate a hash array hash_array. Then, based on the number of edges of each node, construct an edge offset array offset_array. Finally, extract all the edges of each node and form an edge array edges_array. Therefore, the final representation of the subgraph is [hash_array, offset_array, edges_array].

[0022] Furthermore, the step (3) specifically includes the following sub-steps:

[0023] (3.1) In the first-stage random walk process, the selected subgraph is first loaded into the FPGA of Smart SSD7, and the nodes where the walkers are located in the subgraph are marked as active nodes. Then, two-layer breadth-first search (BFS) operations are performed from each active node to obtain the nodes required to construct a smaller dynamic subgraph. When performing two-layer BFS, the rules to be followed are as follows: For nodes whose edges are not completely within the current subgraph, called boundary nodes, the pruning strategy will be applied and the two-layer BFS operation will not be performed on them. In addition, when performing two-layer BFS on active nodes, if their child nodes do not belong to the current subgraph, these child nodes will not participate in the subsequent construction process of smaller subgraphs;

[0024] (3.2) After the active node completes the two-layer BFS, it will obtain all the nodes required to build the dynamic subgraph, and construct the subgraph by selecting the top 1% of the nodes with high access frequency to form a subgraph. At the same time, the random walk process on the host and the two-layer BFS process of the FPGA in the Smart SSD7 can be executed concurrently;

[0025] (3.3) Each time, four sub-graphs that have been divided need to be selected and loaded into the FPGA of Smart SSD7. When dynamically constructing the sub-graph, in addition to all the nodes obtained from the active node through two layers of BFS, the previous hop nodes of the walkers in these sub-graphs also need to be included. The other processes are exactly the same as the first-stage random walk process, which is also the difference between the second-stage random walk process and the first-stage random walk process.

[0026] Furthermore, the subgraph construction process can be performed in parallel with the random walk process, specifically including:

[0027] In the first phase of random walk, when the host is executing random walk, the subgraph containing the most walkers is selected from the remaining subgraphs that have been divided and loaded into the FPGA of Smart SSD7 to construct a dynamic subgraph. This operation ensures that the two phases can be executed in parallel;

[0028] In the second-stage random walk process, when the host is executing the random walk, four subgraphs containing the most walkers are selected from the remaining divided subgraphs and loaded into the FPGA of Smart SSD7 for dynamic subgraph construction. This operation ensures that the two stages can be executed in parallel.

[0029] Specifically, each time the divided sub-graph needs to be loaded into the FPGA of Smart SSD7 to construct a smaller dynamic sub-graph, including:

[0030] After determining the minimum data loading amount N that can break the disk startup time, in the first-stage random walk process, if the data amount of the current constructed dynamic subgraph is greater than N, the subgraph is directly loaded to the host side. If the data amount of the current constructed subgraph is less than N, the divided subgraph is continuously transferred from the disk to the FPGA for further dynamic subgraph construction until the total amount of data of all constructed dynamic subgraphs exceeds N.

[0031] For the two-stage random walk, the first four sub-graphs that have been divided are loaded into the FPGA. When the amount of data of the constructed dynamic sub-graph does not reach N, one more sub-graph that has been divided is loaded into the FPGA each time. The rest of the process is the same as the one-stage random walk process.

[0032] The present invention also discloses a locality-aware out-of-core graph random walk processing system, which is applied to the locality-aware out-of-core graph random walk processing method, and specifically includes the following contents:

[0033] Host CPU and Smart SSD side;

[0034] The host CPU processes the random walk process and updates the number of walkers in the subgraph;

[0035] The Smart SSD needs to store all the divided subgraphs and all walkers states required for the walking process on the one hand, and on the other hand, it needs to perform acceleration operations of sorting, two-layer BFS and dynamic subgraph construction.

[0036] Furthermore, the host CPU processes the random walk process and updates the number of walkers in the subgraph, including:

[0037] The walking process of the first 90% of walkers is completely executed in the host CPU, and the walking process of the last 10% of walkers and the division operation of the dynamic subgraph are executed in the Smart SSD. During the first-stage random walk, the host scheduling unit selects the subgraph containing the most walkers from the SSD and loads it to the host side; during the second-stage random walk, the host scheduling unit selects the four subgraphs containing the most useful walkers from the SSD and loads them to the host side. Subsequently, the random walk calculation unit performs a first-stage or a second-stage random walk on the loaded subgraph. During the walk, some walkers will terminate and some will jump to other subgraphs. The walker number update unit records these jumps and updates the number of walkers in the walker number buffer. At the same time, the accumulation unit records the number of visits to each node during the random walk until the first 90% of walkers have completed the walk.

[0038] Furthermore, in the walking process of the last 10% walkers, the dynamic subgraph partitioning operation is performed in Smart SSD, specifically as follows:

[0039] The Smart SSD consists of two components: SSD and FPGA accelerator. The SSD is used to store all the divided subgraphs and the status information of all walkers during the random walk. The FPGA accelerator is responsible for the division of dynamic subgraphs. The SSD and FPGA accelerator can communicate directly without coordination through the host CPU. The FPGA accelerator can dynamically construct the loaded subgraphs during the walk of the last 10% walkers, including:

[0040] After the first 90% of walkers complete their walks, the vertex sorting core first sorts the nodes of all subgraphs from high to low according to the access frequency of the nodes in the first 90% of walkers. Then, the top 1% of nodes are selected and dynamic subgraphs are constructed in the subgraph construction core. After the construction is completed, the dynamic subgraph is loaded into the SSD. Subsequently, the host CPU loads the constructed dynamic subgraph from the SSD to complete this round of random walks.

[0041] When this round of random walk is completed, for the first phase of random walk, the subgraph scheduling core will select the subgraph with the largest number of walkers from the SSD and send it to the BFS computing core for two-layer BFS. The active nodes obtained by the two-layer BFS will be sent to the subgraph construction core to complete the construction of the dynamic subgraph. The constructed dynamic subgraph will be stored in the SSD. The host CPU will then load the dynamic subgraph from the SSD to continue the random walk. At the same time, the subgraph scheduling core will continue to select the subgraph with the second largest number of walkers to build the dynamic subgraph, ensuring that the host CPU and Smart SSD can operate in parallel;

[0042] For the two-stage random walk, the subgraph scheduling core selects the four subgraphs with the largest number of walkers from the SSD and transmits them to the BFS computing core for two-layer BFS. The active nodes generated by the two-layer BFS and the previous hop nodes of the walkers in the subgraph will be sent to the subgraph construction core to complete the construction of the dynamic subgraph. The constructed subgraph will be stored in the SSD, and then the host CPU will load the subgraph from the SSD to continue the random walk. At the same time, the subgraph scheduling core continues to select the four subgraphs with the largest number of walkers from the remaining subgraphs for the next step of dynamic subgraph construction, thereby ensuring that the host CPU and Smart SSD can work in parallel.

[0043] The beneficial effects of the present invention are as follows:

[0044] The present invention is based on a locality-aware out-of-core graph random walk strategy, which dynamically partitions the subgraphs to be loaded into the memory during the last 10% of the walk. This method effectively reduces the amount of data transferred from the SSD to the memory, while ensuring that the loaded dynamic subgraphs contain most of the valuable nodes, thereby successfully solving the long-tail effect problem caused by the last 10% of walkers in the out-of-core graph random walk, and significantly alleviating the I / O bottleneck of the out-of-core random walk. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0046] Figure 1 It is a schematic diagram of a random walk processing method and system of an out-of-core graph based on locality perception preferred by the present invention;

[0047] Figure 2 This is a schematic diagram of the preferred Smart SSD7 hardware of the present invention;

[0048] Figure 3 This is a schematic diagram of a two-layer BFS five-stage pipeline module preferred in the present invention;

[0049] Figure 4 It is a schematic diagram of a preferred dynamic subgraph construction implementation module of the present invention;

[0050] Figure 5 It is a schematic diagram of a specific example of a preferred dynamic subgraph of the present invention;

[0051] Figure 6 This is a schematic diagram of an example of a preferred one-stage out-of-core random walk dynamic subgraph partitioning of the present invention;

[0052] Figure 7 It is a schematic diagram of an example of a preferred two-stage out-of-core random walk dynamic subgraph partitioning of the present invention.

[0053] Reference numerals:

[0054] 1-Host CPU; 2-Host scheduling unit; 3-Random walk calculation unit; 4-Walkers number update unit; 5-Walkers number buffer; 6-Accumulation unit; 7-Smart SSD; 8-SSD; 9-Graph storage area; 10-Walkers storage area; 11-FPGA accelerator; 12-Vertex sorting core; 13-Subgraph scheduling core; 14-BFS calculation core; 15-Subgraph construction core. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of this application. In the absence of conflict, the features in the following embodiments and implementations can be combined with each other.

[0056] As a powerful mathematical tool, the random walk model has been widely used in many fields such as physics, chemistry, biology, computer science, finance, and personalized recommendation technology. The random walk process is generally executed on a graph. As the size of the graph increases, it is difficult to accommodate the entire graph on a single system. The graph needs to be divided into blocks and stored on a larger disk or SSD8. Then, the sub-graphs are loaded from the disk or SSD8 into the memory as needed. Since the sub-graphs are frequently loaded in and out of the memory from the disk or SSD8, the I / O loading time in the current out-of-core random walk model accounts for most of the total execution time, and I / O has become the bottleneck of the out-of-core random walk. In fact, most of the I / O execution time is caused by the last 10% of walkers, and the I / O overhead brought by completing the first 90% of walkers is very small. Therefore, in view of the long-tail effect brought by the random walks of the last 10% of walkers, the present invention aims to provide an out-of-core graph random walk processing method and system based on locality perception, so as to solve the I / O overhead problem brought by the random walks of the last 10% of walkers, thereby solving the problem that I / O is the bottleneck of out-of-core random walks.

[0057] The present invention provides a method and system for processing random walks of out-of-core graphs based on locality perception. In the first-stage random walk and the second-stage random walk process, the subgraphs can be dynamically divided by using the spatiotemporal locality in the later stage of the random walk, and only the divided subgraphs are sent to the memory each time. The specific process is as follows:

[0058] Regardless of whether it is a one-stage random walk or a two-stage random walk, the first 90% of walkers still walk in the normal mode. When the number of walkers is only 10%, the time locality is first used to assemble the top 1% of nodes most frequently used by the first 90% of walkers into a dynamic subgraph and send it to the memory to complete a random walk.

[0059] After the subgraph consisting of the top 1% of nodes most frequently used in the first 90% of walkers is executed, the subgraph required for the remaining walkers is dynamically divided using the dynamic subgraph partitioning accelerator. The subgraph partitioning process and the random walk process are performed concurrently. In the dynamic subgraph partitioning accelerator, spatial locality is mainly used to dynamically partition the subgraph where the walkers are located. Each time it is divided into smaller subgraphs and sent to the memory, it can not only reduce the amount of data transmitted each time, but also make most of the nodes in the subgraph transmitted to the memory each time useful, so as to solve the long tail problem caused by the last 10% of walkers. The dynamic subgraph partitioning is continued until all walkers have finished walking.

[0060] After 90% of the current walkers have completed their walks, the system needs to be able to identify the top 1% of nodes that have been visited the most times. The specific execution process is as follows:

[0061] When the first 90% walkers perform random walks, they need to record the number of times each node in the graph is visited. After the current 90% walkers complete their walks, they sort the nodes in the graph from high to low according to the number of times they are visited, and then select the top 1% of nodes to construct a subgraph and store it in memory. The sorting process is completed in the dynamic graph partitioning accelerator, using merge sort to sort the graph nodes from high to low according to the number of visits. Merge sort is a divide-and-conquer algorithm that divides large problems into small problems to solve. It first divides the data into two sub-arrays, sorts them separately, and then merges the sorted two sub-arrays into an ordered array.

[0062] After sorting the graph nodes from high to low according to the number of visits by the top 90% walkers, the top 1% of nodes most frequently visited by the top 90% walkers can be assembled into a subgraph. The specific format of the subgraph is as follows:

[0063] The extracted subgraph is stored in CSR format. Since the ID numbers of adjacent vertices in the subgraph may be discontinuous and have large gaps, a hash map array needs to be constructed to map the vertex IDs in the subgraph to a continuous numbering interval from 0 to n-1.

[0064] First, extract the top 1% of nodes that are most frequently visited during the walks of the first 90% of walkers and arrange them in order. Next, create a one-to-one hash map hash_array for each node ID and a consecutive number between 0 and n-1. Then, based on the number of outgoing edges of each node, construct an array offset_array that records edge offsets. Finally, extract the outgoing edges of all nodes and form an edge array edges_array. Finally, the representation of the subgraph is [hash_array, offset_array, edges_array].

[0065] After completing a random walk of the subgraph consisting of the top 1% of nodes most frequently visited by the top 90% of walkers, the subgraph that has been divided can be further refined dynamically in the subsequent walks to generate smaller dynamic subgraphs, and these smaller dynamic subgraphs are loaded into memory. The specific execution process is as follows:

[0066] In the first phase of random walk, the selected subgraph is first loaded into the FPGA of Smart SSD7, and the nodes in the subgraph containing walkers are marked as active nodes. Then, two-layer BFS operations are performed from each active node, and the following rules must be followed: For nodes in the subgraph that have adjacent edges that do not completely belong to the subgraph, they are called boundary nodes. For these boundary nodes, a pruning strategy is adopted and two-layer BFS operations are not performed. When performing two-layer BFS on an active node, if its child node does not belong to the current subgraph, BFS stops at the child node.

[0067] After completing the two-layer BFS operation, obtain all the nodes required for the dynamic subgraph. Then, construct these active nodes and their associated edges into the subgraph required for the next round of random walks. This subgraph is constructed in the same way as the subgraph composed of the top 1% of nodes most frequently visited by the top 90% of walkers. At the same time, it should be ensured that the random walk process and the process of performing the two-layer BFS operation in the selected subgraph can be executed in parallel to improve efficiency.

[0068] The main difference between the second-stage random walk process and the first-stage random walk process is that each time, four subgraphs need to be loaded into the FPGA of Smart SSD7 at the same time. In the process of building dynamic subgraphs, in addition to performing two-layer BFS from the active node to obtain all related nodes, the nodes that the walkers in the previous walk of these subgraphs passed through must also be included. Other than that, the rest of the process is exactly the same as the first-stage random walk process.

[0069] At the same time, in order to make fuller use of computing resources, the dynamic subgraph construction process needs to be able to proceed concurrently with the random walk process. The specific process is as follows:

[0070] In the first phase of random walk, when the CPU is executing the random walk operation, a subgraph containing the most walkers is selected from the remaining subgraphs and loaded into the FPGA of Smart SSD7 to construct a dynamic subgraph. In this way, parallel processing of the two phases is achieved, thereby improving the overall performance.

[0071] For the two-stage random walk process, when the CPU is executing the random walk process, four subgraphs with the largest number of walkers are selected from the remaining divided subgraphs and loaded into the FPGA of Smart SSD7 for dynamic subgraph construction, so that the two stages can be carried out simultaneously.

[0072] For the random walk process of the last 10% of walkers, the system first builds a subgraph based on the top 1% of nodes most frequently accessed by the first 90% of walkers. Host CPU1 will first perform a round of random walks on this subgraph. For the remaining walkers, the system needs to dynamically transfer the already divided subgraphs to the accelerator for more refined subgraph construction based on the number of walkers in the subgraph in each iteration. As the random walk process progresses, the number of walkers gradually decreases, resulting in a gradual reduction in the size of the subgraph. When the size of the subgraph is too small, the time required to transfer the subgraph from SSD8 will be less than the disk startup time, which will cause a waste of disk bandwidth. To avoid this problem, when the size of the subgraph is too small, the system will select other blocks to continue subgraph construction until the time required to transfer the subgraph from the disk or SSD8 to the memory can exceed the disk startup time. The specific process is as follows:

[0073] The minimum data loading amount N that can break the disk startup time is measured. For a one-stage random walk, if the data size of the currently constructed dynamic subgraph is greater than N, the subgraph is directly sent to SSD8. If the data size of the currently constructed subgraph is less than N, it is necessary to continue to call the subgraphs that have been divided in SSD8 into FPGA to construct finer-grained dynamic subgraphs until the data size of the constructed subgraph exceeds N.

[0074] In the two-stage random walk, the main difference from the one-stage random walk process is that the first four subgraphs are loaded into the FPGA. When the amount of dynamically constructed subgraph data does not reach the threshold N, an additional subgraph is loaded into the FPGA each time. The rest of the operations are consistent with the one-stage random walk process.

[0075] The out-of-core graph random walk system based on locality awareness can solve the long-tail effect caused by the last 10% of walkers. An important means is to combine the Smart SSD7 hardware to dynamically divide the already divided sub-graphs in a finer granularity.

[0076] The overall architecture of a locality-aware out-of-core graph random walk processing system is as follows: Figure 1 As shown, the overall architecture includes a host CPU 1 and a hardware accelerator Smart SSD 7 .

[0077] Host CPU1 mainly processes the random walk process and updates the number of walkers in the divided sub-graph.

[0078] On the one hand, Smart SSD7 needs to store the entire graph and the status of all walkers during the walk, and on the other hand, it needs to perform acceleration operations such as sorting, two-layer BFS, and subgraph construction.

[0079] Host CPU1 mainly processes the random walk process, specifically:

[0080] During the first 90% of the walkers' walks, all calculations are completed on the host CPU1. For the first-stage random walk, the host scheduling unit 2 selects the subgraph containing the most walkers from SSD8 and loads it into the host memory. In the second-stage random walk, the host scheduling unit 2 selects the four subgraphs containing the most active walkers from SSD8 and loads them into the host. Subsequently, the random walk calculation unit 3 performs the first-stage or second-stage random walk calculation on the loaded subgraph. During the walk, some walkers will be terminated and some will jump to other subgraphs. The walkers number update unit 4 records the number of walkers that jump to other subgraphs and updates the relevant data in the walkers number buffer 5. At the same time, during the random walk, the accumulation unit 6 records the access frequency of each node until the walks of the first 90% of the walkers are completed.

[0081] After the current 90% of walkers have completed random walks, SmartSSD7 needs to assemble the dynamic subgraphs during the walks of the last 10% of walkers, and send the assembled dynamic subgraphs to the CPU for walks each time.

[0082] Smart SSD7 includes two components: SSD8 and FPGA accelerator 11. SSD8 mainly stores the entire graph structure and the status information of all walkers during the random walk. The graph structure is stored in the graph storage area 9, and the walkers status information is stored in the walkers storage area 10. The FPGA accelerator 11 mainly divides the blocks to obtain dynamic sub-graphs. SSD8 and FPGA accelerator 11 can communicate directly without coordination through the host CPU1.

[0083] FPGA accelerator 11 can construct the dynamic subgraph required for the walking process of the last 10% walkers. The main process is as follows:

[0084] After the current 90% of walkers have completed their walks, the vertex sorting core 12 first sorts the nodes of the entire graph from high to low according to the order in which the nodes are visited in the first 90% of walkers, and then takes out the first 1% of the nodes to construct a subgraph in the subgraph construction core 15, and then sends the constructed subgraph to SSD8. Finally, the host CPU1 loads the subgraph from SSD8 to complete this round of random walks.

[0085] When this round of random walk is completed, if it is a one-stage random walk, the accelerator subgraph scheduling core 13 takes out a subgraph with the largest number of walkers from SSD8, and then sends it to the BFS computing core 14 for two-layer BFS. The active nodes obtained from the two-layer BFS are sent to the subgraph construction core 15 to complete the construction of the subgraph. The constructed subgraph is sent to SSD8, and the host CPU1 takes the subgraph from SSD8 for random walk. At the same time, the accelerator subgraph scheduling core 13 continues to select the subgraph with the second largest number of walkers to construct a dynamic subgraph, so that the host CPU1 and SmartSSD7 can work in parallel at the same time.

[0086] If it is a two-stage random walk, the accelerator subgraph scheduling core 13 takes out the four subgraphs with the largest number of walkers from SSD8, and then sends them to the BFS computing core 14 for two-layer BFS. The active nodes obtained by the two-layer BFS and the previous hop nodes of the walkers in the subgraph are sent to the subgraph construction core 15 to complete the construction of the dynamic subgraph. The constructed subgraph is sent to SSD8, and the host CPU1 takes the subgraph from SSD8 for random walk. At the same time, the accelerator subgraph scheduling core 13 continues to select the four subgraphs with the largest number of walkers in the remaining divided subgraphs to construct the dynamic subgraph, so that the host CPU1 and Smart SSD7 can work in parallel at the same time.

[0087] The structure of Smart SSD7 is as follows Figure 2As shown, a locality-aware out-of-core graph random walk processing method and system can successfully solve the long-tail problem caused by the late I / O of the out-of-core random walk. One of the keys is to utilize the dynamic construction of the subgraph by Smart SSD7. Here we first introduce the specific structure of Smart SSD7.

[0088] Smart SSD7 consists of two parts: SSD8 memory domain and FPGA accelerator 11. SSD8 and FPGA accelerator 11 communicate directly in P2P mode without the help of host CPU1. Therefore, the divided sub-graphs can be directly transferred from SSD8 to the FPGA accelerator 11 for dynamic sub-graph construction. Finally, the constructed sub-graphs are sent to the host side for random walk, and the sub-graph construction process and the random walk on the host side are carried out concurrently.

[0089] The two-layer BFS computing core 14 implements Figure 3 As shown, the two-layer BFS of the present invention adopts a five-stage pipeline structure. The two-layer BFS is defined as follows: if only active nodes are extracted, it is called a one-layer BFS; if active nodes and their child nodes are extracted, it is called a two-layer BFS.

[0090] The active state of the nodes in the subgraph is represented by a bitmap, and the activity of each node is identified by two bits. For a one-phase random walk, the bit of the node where the walkers are located is set to 1, and the other nodes are set to 0. For a two-phase random walk, first set the bit of the node where the walkers are located to 1, and the other nodes are set to 0, and then traverse the previous hop node of the walkers again. If the ID of the previous hop node is 0 in the bitmap, mark the position as 2; if the ID of the previous hop node is 1 in the bitmap, do not modify it. The information of the node in the subgraph is represented in the form of a five-tuple, including the node ID, the starting offset and ending offset of the node's neighbors, and all neighbors of the node; if it is a two-phase random walk, the previous hop node ID of the node must also be included; if it is a one-phase random walk, the previous hop node ID of the node is the node itself.

[0091] The five-stage pipeline implementation of the two-layer BFS is as follows: in the first stage, first scan the entire bitmap information and take out the nodes marked as 1 in the bitmap; in the second stage, obtain the starting offset and ending offset corresponding to the neighbors of the active node; in the third stage, obtain all neighbors of the active node according to the starting offset and ending offset corresponding to the active node. In order to improve the execution efficiency and make full use of the internal bandwidth of the FPGA, the neighbors of the node are read in a burst form; in the fourth stage, filter the obtained neighbor nodes. If the value of the neighbor in the bitmap is 1 or 2, ignore it. If the mark of the neighbor node in the bitmap is 0, mark the neighbor node as 1 in the corresponding position of the bitmap; in the fifth stage, if it is a one-stage random walk, scan the entire bitmap again, take out the nodes marked as 1 and send them to the next stage. If it is a two-stage random walk, take out the nodes marked as 1 or 2 in the bitmap and send them to the next stage.

[0092] The input of the two-layer BFS operation is the active nodes where the walkers are located in the subgraph, and the output is all non-duplicate nodes after the two-layer BFS. The five stages of the two-layer BFS operation are executed concurrently in the form of a pipeline.

[0093] The subgraph construction is implemented as follows Figure 4 As shown, the subgraph construction of the present invention is implemented using a three-stage pipeline.

[0094] The input of the subgraph is the nodes that need to construct the subgraph after two layers of BFS operations. In the first stage of subgraph construction, the following two operations are performed concurrently: first, the offset corresponding to the active node is obtained, and at the same time, a hash table about the active node is constructed. The purpose of the hash table is to obtain the corresponding subscript of the node according to the node ID. The hash table is constructed using id m →m; In the second stage, the following two operations are performed according to the obtained offset: The offset of the node is expressed in the form of (start_offset, end_offset). First, the cumulative offset of each node's neighbors is calculated. The cumulative offset of the i-th position is calculated as offset i =offset i-1 + (end_offset-start_offset), when i=0, offset 0 = 0, and at the same time, obtain the neighbors corresponding to these nodes; in the third stage, the subgraph is constructed according to the cumulative offsets calculated in the second stage, the node neighbors and the constructed hash table. The subgraph is represented in the form of a triple: (hash_array, offset_array, edges_array), and hash_array uses id m →The mapping representation of m, id mIndicates the node ID, offset_array uses (offset 0 , offset 1 , …, offset n-1 ) is expressed in the form of offset m Indicates where the neighbors of node m start in edges_array, offset m+1 Indicates where the neighbors of node m end in edges_array. edges_array uses (ngh_id 0 , ngh_id 1 , …, ngh_id m ), ngh_id m It is the neighbor node corresponding to the node. The neighbor nodes of all nodes are placed in an array. The corresponding neighbor node of any node can be obtained through the offset_array structure.

[0095] The input of the subgraph construction operation is the node ID after two layers of BFS, and the output is a subgraph constructed using three primitive groups (hash_array, offset_array, edges_array). The above three stages are executed concurrently in a pipeline manner, and in each stage, the corresponding small stages of each stage can be executed concurrently.

[0096] Figure 5 is a specific instance of a subgraph representation, Figure 5 (a) in the figure is a corresponding graph structure, which is divided into 4 blocks. Figure 5 (b) in the figure is the specific representation of the subgraph (blk1). Assuming that all nodes in blk1 are active nodes, there are 3 nodes in the corresponding subgraph. The specific representation of the subgraph is represented by three data structures. The first data structure is hash_array. In blk1, node 3 corresponds to subscript 0, node 4 corresponds to subscript 1, and node 5 corresponds to subscript 2. The subscripts here correspond to the positions of each element in the offset_array structure. The second data structure corresponds to offset_array. Each element in the structure is an offset. m , offset m Indicates the starting position of the neighbor corresponding to the node m, offset m+1Indicates the end position of the neighbors of node m. In blk1, node 3 has 2 edges, the first element in offset_array is 0, and the second element in offset_array is 0+2=2. Node 4 has 1 edge in blk1, the third element in offset_array is 2+1=3. Node 5 has two edges, and the fourth element in offset_array is 3+2=5. The third data structure is edges_array, each element of which corresponds to the neighbors of a node. The neighbors of node 3 in blk1 are 4 and 5, and the first two elements in edges_array are 4 and 5. The neighbor of node 4 is 0, and the third element in edges_array is 0. The neighbors of node 5 are 4 and 8, and the last two elements in edges_array are 4 and 8.

[0097] The above three structures can be used to perform a random walk process. For example, the current walker starts from node 3. First, it finds from the hash_array structure that node 3 corresponds to position 0 in the offset_array array, indicating that the corresponding position of the neighbor of node 3 in the edges_array array is [0, 2). The left is closed and the right is open. Then the next hop of the random walk will be selected from nodes 4 and 5. If the next hop is node 5, it is found from the hash_array structure that node 5 corresponds to position 2 in the offset_array array, indicating that the corresponding position of the neighbor of node 3 in the edges_array array is [3, 5). The left is closed and the right is open. Then the next hop of the random walk will be selected from nodes 4 and 8. If the node selected for the next hop is 8, it is found in the hash_array structure that node 8 does not exist in the node ID, indicating that the walker has jumped out of this block and the random walk should be interrupted.

[0098] Example 1 is as follows:

[0099] This embodiment is a specific example of a method and system for processing random walks on an out-of-core graph based on locality perception. Figure 6This is a schematic illustration of a one-stage out-of-core random walk dynamic subgraph partitioning example. The one-stage out-of-core random walk is divided into three stages. The first stage is to complete the walk of the first 90% of walkers normally. The second stage is to extract the top 1% of nodes that are most frequently visited by the first 90% of walkers during the walk to form a subgraph. The remaining walkers walk on this subgraph once. The third stage is to use the node where the walkers are located in the subgraph as the starting point to perform two-layer BFS operations to construct a dynamic subgraph. If all the edges of a node and the node are in a subgraph, the node is called an internal node. Otherwise, the node is called a boundary node. If only the node where the walkers are located and its corresponding edges are loaded, it is called only one layer of BFS operation. If the active node and its child nodes and their corresponding edges are loaded, it is called two layers of BFS. When performing two layers of BFS, a pruning strategy is adopted, that is, if the node where the walker is located is a boundary node, only the node and its corresponding edges are loaded, and one layer of BFS operation is performed. The first stage of the one-stage locality-aware out-of-core graph random walk processing method is a normal walk, which does not require much introduction. The second stage extracts hot nodes to construct a subgraph. The process is relatively simple and will not be introduced in detail here. The following example only introduces the third stage in detail.

[0100] In this example, if Figure 6 The graph structure described in (a) is divided into 3 blocks. In blk0, there is a walker located at node 0 and has 2 steps to go. In blk1, there are two walkers, located at nodes 3 and 5 respectively. The walker at node 3 has 5 steps to go, and the walker at node 5 has 2 steps to go. In blk2, there is a walker located at node 9 and has 3 steps to go.

[0101] First, we select the subgraph blk1 with the largest number of walkers and load it from SSD8 into the embedded FPGA. Then, we perform two-layer BFS on node 3. Since node 3 is an internal node, the nodes after BFS on node 3 are 3 and 5. Since there is also a walker on node 5, which is also an active node, we need to perform two-layer BFS extension on node 5. Node 5 is also an internal node, so the nodes after two-layer BFS on node 5 are 5 and 7. Therefore, the nodes after two-layer BFS on the active nodes in blk1 are nodes 3, 5, and 7. We construct the subgraph of these three nodes according to the previous subgraph construction method. The subgraph result is as follows: Figure 6The "load blk1" shown in (b) in the figure loads the obtained subgraph into the memory instead of the entire blk1. Then the walkers on nodes 3 and 5 walk on the subgraph. Assuming that the walking trajectory of node 3 is 3→5→7→9, the walker on node 3 walks from blk1 to blk2, a total of 3 steps. The walking trajectory of node 5 is 5→7→8. Since the walker on node 5 has 2 steps left to walk, this walk just completes 2 steps and the walk process of the walker ends. At this time, the number of walkers in blk1 is 0, the number of walkers in blk2 is 2, and the number of walkers in blk0 is 1. The block with the largest number of walkers is selected for BFS extension. Therefore, the block selected for the next walk is blk2, and blk2 is loaded from SSD8 into the embedded FPGA. The node where the walkers are located is node 9. Two layers of BFS expansion are performed from node 9. Since node 9 is a boundary node, node 9 is pruned. The BFS expansion result in blk2 is node 9. A subgraph is constructed for node 9. The result is as follows: Figure 6 "Load blk2" as shown in (b) in the figure, and then send the subgraph into memory. There are two walkers starting from node 9. The walker that jumps in from blk1 has a walk trajectory of 9→2, and the walker on node 9 in blk2 has a walk trajectory of 9→2. The two walkers starting from node 9 have only walked forward 1 step. At this time, the number of walkers in blk1 and blk2 are both 0, and the number of walkers in blk0 is 3. There are 2 walkers on node 2 and 1 walker on node 0. First, two layers of BFS are performed from node 2. Node 2 is an internal node and the expansion results are 2 and 0. Then two layers of BFS are performed from node 0. Node 0 is an internal node and the expansion results are 0 and 1. The final expansion results in blk0 are nodes 0, 1, and 2. The subgraph construction result is as follows: Figure 6 The "load blk0" shown in (b) in the figure sends the subgraph into the memory for random walk. There are 2 walkers on node 2. The trajectory of the walker walking from blk1 is 2→0. At this time, the walker from blk1 has reached the number of steps it needs to walk, and the walking process is terminated. The walker from blk2 walks from node 2 with a trajectory of 2→0→1. Including one round of walking, a total of 3 steps are taken, reaching the specified number of walking steps, and the walking process is terminated. The walker on node 0 walks with a trajectory of 0→1→4, reaching the specified number of walking steps, and the walking process ends. At this time, there are no walkers in all blocks, and the random walk ends.

[0102] If we follow the normal random walk process, we need to load the three subgraphs blk0, blk1 and blk2 in sequence, and the number of edges that need to be loaded is 6+7+4=17 edges. After using the subgraph partitioning method, the total number of edges that need to be loaded is 4+2+4=10 edges. For real large graphs, the dynamic block partitioning method can reduce the subgraph size to less than 10% of the original block. Moreover, as the walk process progresses, the size of the subgraph will become smaller and smaller, which can greatly improve the utilization rate of useful edges loaded into memory.

[0103] Example 2 is as follows:

[0104] This embodiment is a specific example of a two-stage out-of-core random walk dynamic block partitioning method based on locality.

[0105] Figure 7 This is a schematic illustration of an example of dynamic division of a two-stage out-of-core random walk. Both the two-stage out-of-core random walk and the one-stage out-of-core random walk are completed in three stages. Here, the third stage of the two-stage random walk is still introduced in detail, and the first two stages are not repeated.

[0106] The first-stage random walk only loads one subgraph each time. The second-stage random walk needs to use the walker's previous hop node, so multiple subgraphs need to be loaded into memory for execution. There is a walker in blk0 located at node 0, with 2 steps to go, and the previous hop node is node 2. There are two walkers in blk1, located at nodes 3 and 5 respectively. The walker at node 3 has 5 steps to go, and the previous hop node is node 1. The walker at node 5 has 2 steps to go, and the previous hop node is node 3. There is a walker in blk2 located at node 9, with 3 steps to go, and the previous hop node is node 8. Figure 7(a) in the figure divides the entire graph structure into three blocks, and stipulates that two blocks are loaded into the memory each time. First, the two blocks with the largest number of valid walkers are selected each time for dynamic subgraph division. The combination of blk0 and blk1 or blk1 and blk2 has three walkers, but if blk1 and blk2 are loaded, there are only two valid walkers, namely the walker on node 5 and the walker on node 9. The walker of node 3 needs to rely on node 1 in blk0 to walk, so blk0 and blk1 should be sent from SSD8 to the embedded FPGA for dynamic subgraph construction. In blk0, two layers of BFS are performed starting from node 0. Node 0 is an internal node. After two layers of BFS, the nodes obtained are 0 and 1. In addition, the previous hop node 2 of the walker on node 0 needs to be loaded. Therefore, the nodes used to construct the subgraph in blk0 are nodes 0, 1, and 2. In blk1, two layers of BFS are performed from nodes 3 and 5 respectively, and the previous hop node of walkers is added. The nodes used to construct the subgraph in blk1 are nodes 3, 5, and 7. The subgraph is constructed by constructing the nodes corresponding to blk0 and blk1, and the subgraph obtained is as follows: Figure 7 "Load blk0 and blk1" as shown in (b) in the figure. The walker on node 0 has a walking trajectory of 0→1→3, and the walking step count is reached to terminate the walking process. The walker trajectory of node 3 is 3→5→7→9, and the walker trajectory of node 5 is 5→7→9. Node 5 has reached the walking step count and terminates the walking process. At this time, there are two walkers in blk2, and there are no walkers in blk0 and blk1. Since the previous hops of the walkers on node 9 are nodes 7 and 8 respectively, blk1 and blk2 should be loaded from SSD8 into FPGA for subgraph construction. First, two layers of BFS are performed starting from node 9. Node 9 is a boundary node and should be pruned. The expansion result is only node 9. At the same time, the previous hop nodes of the two walkers, nodes 7 and 8, need to be loaded. The construction result of the subgraph is shown as follows: Figure 7 As shown in (b) of the figure, "load blk1 and blk2" is used to send the constructed subgraph to the host for random walk. The walk trajectories of the two walkers are both 9→2. At this time, there are two walkers in blk0, and their previous hops are both node 9. At this time, blk0 and blk2 need to be loaded into the FPGA for subgraph construction. The nodes obtained from the two-layer BFS of node 2 are 0 and 2. At the same time, the previous hop node of the walkers, node 9, needs to be loaded. The constructed subgraph is as follows: Figure 7As shown in (b) of the figure, "Loading blk0 and blk2", the walker trajectory from blk1 is 2→0, reaching the number of walk steps, and the walk process ends. The walker trajectory from blk2 is 2→0→1, reaching the total number of walk steps, and the walk process ends. At this time, there are no walkers in all blocks, and the second-stage random walk ends.

[0107] The dynamic block construction method is used to perform a two-stage out-of-core random walk process. In this example, the total number of edges that need to be loaded is 8+5+4=17 edges, while the number of edges that need to be loaded using the traditional method is (6+7)+(7+4)+(4+6)=34 edges. Especially for real graphs in reality, the use of dynamic subgraph partitioning methods in the later stage of random walk can greatly improve the utilization rate of edges loaded into the host.

[0108] The construction process of the dynamic subgraph is carried out inside the FPGA, which makes full use of the direct communication feature of the FPGA and SSD8 in the Smart SSD7. In addition, the construction of the subgraph and the execution of the random walk on the host side can be executed concurrently. These are the keys to the dynamic subgraph being able to eliminate the I / O long tail problem in the later stage of the random walk.

[0109] The above embodiments are only used to illustrate the design ideas and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, any equivalent changes or modifications made based on the principles and design ideas disclosed by the present invention are within the protection scope of the present invention.

Claims

1. A locality-aware out-of-core graph random walk processing method, characterized in that: The following steps are involved: (1) During the first and second phases of the random walk, the graph is dynamically partitioned by utilizing the spatiotemporal locality characteristics of the late random walk, and only the dynamically partitioned subgraph is loaded into memory each time; (2) When the number of random walkers is reduced to 10%, the top 1% of nodes most frequently visited by the first 90% of walkers are identified and extracted using temporal locality, and a subgraph is constructed and loaded into memory to complete a random walk. (3) After processing the subgraph consisting of the first 1% of active nodes, in the subsequent random walk process, the subgraph where the walkers are located is dynamically divided using spatial locality, that is, the subgraph is further decomposed, and each time the decomposed subgraph is loaded into memory for processing until all walkers complete their walk tasks; It includes the following sub-steps: (3.1) In the first-stage random walk process, the selected subgraph is first loaded into the FPGA of Smart SSD7, and the nodes where the walkers are located in the subgraph are marked as active nodes. Then, two-layer breadth-first search (BFS) operations are performed from each active node to obtain the nodes required to construct a smaller dynamic subgraph. When performing two-layer BFS, the rules to be followed are as follows: For nodes whose edges are not completely within the current subgraph, called boundary nodes, the pruning strategy will be applied and the two-layer BFS operation will not be performed on them. In addition, when performing two-layer BFS on active nodes, if their child nodes do not belong to the current subgraph, these child nodes will not participate in the subsequent construction process of smaller subgraphs; (3.2) After the active node completes the two-layer BFS, it will obtain all the nodes required to build the dynamic subgraph, and construct the subgraph by selecting the top 1% of the nodes with high access frequency to form a subgraph. At the same time, the random walk process on the host and the two-layer BFS process of the FPGA in the Smart SSD7 can be executed concurrently; (3.3) Each time, four sub-graphs that have been divided need to be selected and loaded into the FPGA of Smart SSD7. When dynamically constructing the sub-graph, in addition to all the nodes obtained from the active node through two layers of BFS, the previous hop nodes of the walkers in these sub-graphs need to be included. The rest of the process is exactly the same as the one-stage random walk process.

2. The method according to claim 1, characterized in that The first 1% of nodes most frequently visited by the first 90% walkers during their walks are identified and extracted, specifically: After the first 90% of walkers have completed their random walks, they need to record the number of visits to each node in the graph. After the walk is completed, all nodes in the graph are sorted from high to low according to the number of visits, and the top 1% of nodes with high visit frequency are selected to form a subgraph and stored in memory.

3. The method according to claim 2, characterized in that The method of sorting all nodes in the graph from high to low according to the number of visits is as follows: Merge sort is used to sort the graph nodes from high to low according to the number of visits. Merge sort is a divide-and-conquer algorithm. First, the data is divided into two sub-arrays, which are sorted separately, and then the two sorted sub-arrays are merged into an ordered array.

4. The method according to claim 2, characterized in that: The first 1% of the nodes with high access frequency are selected to form a subgraph, specifically: the extracted subgraph is stored in CSR format, and since there is a gap in the numbering of adjacent vertices in the subgraph, a hash array is required to establish a mapping relationship between the subgraph vertex ID and the index 0 to n-1; First, extract the top 1% nodes with the highest frequency of access during the first 90% of the walks and arrange these nodes in order. Next, establish a one-to-one hash mapping relationship between each node ID and index 0 to n-1 to generate a hash array hash_array. Then, based on the number of edges of each node, construct an edge offset array offset_array. Finally, extract all the edges of each node and form an edge array edges_array. Therefore, the final representation of the subgraph is [hash_array, offset_array, edges_array].

5. The method according to claim 1, characterized in that The subgraph construction process can be performed in parallel with the random walk process, and specifically includes: In the first phase of random walk, when the host is executing random walk, the subgraph containing the most walkers is selected from the remaining subgraphs that have been divided and loaded into the FPGA of Smart SSD7 to construct a dynamic subgraph. This operation ensures that the two phases can be executed in parallel; In the second-stage random walk process, when the host is executing the random walk, four subgraphs containing the most walkers are selected from the remaining divided subgraphs and loaded into the FPGA of Smart SSD7 for dynamic subgraph construction. This operation ensures that the two stages can be executed in parallel.

6. The method according to claim 1, characterized in that Each time, the divided sub-graph needs to be loaded into the FPGA of Smart SSD7 to construct a smaller dynamic sub-graph, including: After determining the minimum data loading amount N that can break the disk startup time, in the first-stage random walk process, if the data amount of the current constructed dynamic subgraph is greater than N, the subgraph is directly loaded to the host side. If the data amount of the current constructed subgraph is less than N, the divided subgraph is continuously transferred from the disk to the FPGA for further dynamic subgraph construction until the total amount of data of all constructed dynamic subgraphs exceeds N. For the two-stage random walk, the first four sub-graphs that have been divided are loaded into the FPGA. When the amount of data of the constructed dynamic sub-graph does not reach N, one more sub-graph that has been divided is loaded into the FPGA each time. The rest of the process is the same as the one-stage random walk process.

7. A locality-aware out-of-core graph random walk processing system, characterized in that: The system is applied to a locality-aware out-of-core graph random walk processing method as described in any one of claims 1 to 6, and specifically includes the following contents: Host CPU and Smart SSD side; The host CPU processes the random walk process and updates the number of walkers in the subgraph; The Smart SSD needs to store all the divided subgraphs and all walkers states required for the walking process on the one hand, and on the other hand, it needs to perform acceleration operations of sorting, two-layer BFS and dynamic subgraph construction.

8. The system according to claim 7, characterized in that The host CPU processes the random walk process and updates the number of walkers in the subgraph, including: The walking process of the first 90% of walkers is completely executed in the host CPU, and the walking process of the last 10% of walkers and the division operation of the dynamic subgraph are executed in the Smart SSD. During the first-stage random walk, the host scheduling unit selects the subgraph containing the most walkers from the SSD and loads it to the host side; during the second-stage random walk, the host scheduling unit selects the four subgraphs containing the most useful walkers from the SSD and loads them to the host side. Subsequently, the random walk calculation unit performs a first-stage or a second-stage random walk on the loaded subgraph. During the walk, some walkers will terminate and some will jump to other subgraphs. The walker number update unit records these jumps and updates the number of walkers in the walker number buffer. At the same time, the accumulation unit records the number of visits to each node during the random walk until the first 90% of walkers have completed the walk.

9. The system according to claim 8, characterized in that In the walking process of the last 10% walkers, the dynamic subgraph division operation is performed in Smart SSD, specifically: The Smart SSD consists of two components: SSD and FPGA accelerator. The SSD is used to store all the divided subgraphs and the status information of all walkers during the random walk. The FPGA accelerator is responsible for the division of dynamic subgraphs. The SSD and FPGA accelerator can communicate directly without coordination through the host CPU. The FPGA accelerator can dynamically construct the loaded subgraphs during the walk of the last 10% walkers, including: After the first 90% of walkers complete their walks, the vertex sorting core first sorts the nodes of all subgraphs from high to low according to the access frequency of the nodes in the first 90% of walkers. Then, the top 1% of nodes are selected and dynamic subgraphs are constructed in the subgraph construction core. After the construction is completed, the dynamic subgraph is loaded into the SSD. Subsequently, the host CPU loads the constructed dynamic subgraph from the SSD to complete this round of random walks. When this round of random walk is completed, for the first phase of random walk, the subgraph scheduling core will select the subgraph with the largest number of walkers from the SSD and send it to the BFS computing core for two-layer BFS. The active nodes obtained by the two-layer BFS will be sent to the subgraph construction core to complete the construction of the dynamic subgraph. The constructed dynamic subgraph will be stored in the SSD. The host CPU will then load the dynamic subgraph from the SSD to continue the random walk. At the same time, the subgraph scheduling core will continue to select the subgraph with the second largest number of walkers to build the dynamic subgraph, ensuring that the host CPU and Smart SSD can operate in parallel; For the two-stage random walk, the subgraph scheduling core selects the four subgraphs with the largest number of walkers from the SSD and transmits them to the BFS computing core for two-layer BFS. The active nodes generated by the two-layer BFS and the previous hop nodes of the walkers in the subgraph will be sent to the subgraph construction core to complete the construction of the dynamic subgraph. The constructed subgraph will be stored in the SSD, and then the host CPU will load the subgraph from the SSD to continue the random walk. At the same time, the subgraph scheduling core continues to select the four subgraphs with the largest number of walkers from the remaining subgraphs for the next step of dynamic subgraph construction, thereby ensuring that the host CPU and Smart SSD can work in parallel.

Images