BFS graph traversal optimization method oriented to heterogeneous graph processing system

Abstract

The invention discloses a BFS graph traversal optimization method oriented to a heterogeneous graph processing system, which comprises the following steps of: S1, dividing graph data, sorting vertexes of a graph according to key attributes, and then distributing the sorted vertexes into different partitions; s2, sorting the vertexes, and carrying out descending sorting on the vertexes in the dictionary according to out-degree; and S3, distributing thread counts according to the out-degree of each vertex so as to process the data through threads. By combining the advantages of the two methods, the algorithm can process various complex conditions more effectively, and potential errors and conflicts are reduced.

Description

Technical Field

[0001] The present invention relates to the field of graph data processing, and in particular to a BFS graph traversal optimization method for a heterogeneous graph processing system. Background Art

[0002] The Graphics Processing Unit (GPU) is a general-purpose many-core accelerator that provides powerful parallel computing capabilities, which makes the GPU have obvious advantages over the Central Processing Unit (CPU) in parallel computing. The CPU is suitable for logic-intensive control tasks, but lacks in computing-intensive tasks; the GPU is equipped with many computing resources with consistent structure, which can process multiple tasks in parallel at the same time, making the graph traversal faster and suitable for traversing large amounts of data. The CPU and GPU are used to jointly process graph traversal tasks. When the graph is sparse, the tasks will be directly scheduled to the CPU for processing, reducing expensive data transmission overhead; when the graph is dense, the CPU and GPU will dynamically select subgraph tasks for processing, and synchronize vertex data on both ends in a fine-grained manner during the iteration process. [9] We can use a CPU-GPU heterogeneous system to support large-scale graph traversal tasks. The system can store large real graphs in CSR format on the CPU side, and transfer the data to the GPU side through CUDA functions or PCI-E when needed. Then, the GPU side uses CUDA code to convert it into processable graph data before executing the traversal. The advantages of CPU-GPU heterogeneous system graph computing include: (1) High parallelism: GPU is very suitable for processing large data sets and can handle multiple tasks at the same time, greatly improving computing efficiency and throughput. (2) Good flexibility: CPU-GPU heterogeneous system can flexibly allocate computing resources according to task requirements, improving the adaptability and scalability of the system. (3) High efficiency: By optimizing the data transmission path and algorithm implementation, the CPU-GPU heterogeneous system can significantly improve the computing speed and reduce the computing time. However, the explosive growth of graphics today has posed a huge challenge to the data transmission between the CPU and GPU. Due to the huge amount of data transmission between the CPU and GPU, the performance of graphics processing usually drops sharply. At the same time, as the graph data increases, the problem of unbalanced thread load on the GPU side also emerges. Therefore, how to make full use of the computing power of this heterogeneous system and improve the system energy efficiency is an urgent problem to be solved.

[0003] In recent years, many scholars have proposed many CPU-GPU heterogeneous graph processing systems. For example, CuSha overcomes the shortcomings associated with CSR representation (irregular memory access and insufficient GPU utilization), places the edges and vertices required for computing subsets in memory continuously, and introduces a graph representation that effectively maps slices to various GPU subcomponents. At the same time, a new graph representation is proposed to eliminate the insufficient utilization of GPU for large sparse graphs. Tigr is committed to converting irregular graphs into more regular graphs, that is, adding a virtual layer on top of the original irregular graph to make it "look more regular". In essence, virtualization separates programming abstraction from physical graphs. This separation allows computing tasks to be scheduled at the virtual layer, while actual value propagation is performed at the physical layer. Subway is committed to solving the problem of data transmission between CPU and GPU in the case of excess memory for large real graphs. It generates multiple subgraphs with active points and transfers them asynchronously to GPU memory. In each iteration, the size of the loaded graph can be significantly reduced, accelerating the data transmission between CPU and GPU for large real graphs.

[0004] At the same time, graph traversal algorithms have also received a lot of attention. The BFS (Breadth First Search) algorithm is an algorithm for traversing or searching a graph or tree data structure. Given a graph G = (V, E), an initial root vertex s, V = {v1, v2, v3, ...}, E = {(v1, v2), (v1, v5), (v2, v4), ...}, V is a vertex set, and E is an edge set. The top-down BFS will visit all reachable vertices starting from s, and then generate the final BFS spanning tree. At the beginning, all data structures are initialized, a random root vertex s is generated and put into the current queue. When the current queue is not empty, all neighbors of each vertex u will traverse the current queue. The traversal process is to mark the status of unvisited neighbors as visited, map their parent vertices and put these vertices into the next queue. The traditional BFS algorithm starts from a node in the graph, first visits all adjacent nodes, and then for each adjacent node, visits their unvisited adjacent nodes until all nodes have been visited. This may lead to repeated visits to a large number of nodes, resulting in redundancy. The bottom-up BFS algorithm proposed by scholars is an improved BFS algorithm. Compared with the top-down method, the bottom-up method is the opposite. When the current queue is large, it starts from all unvisited vertices and checks whether their parent vertices are in the current queue. As the current queue increases, the iterative graph traversal will end earlier. It starts from the bottom of the graph and traverses upward layer by layer until it reaches the top layer. Unlike the traditional top-down BFS algorithm, the bottom-up BFS algorithm only visits the nodes of the current layer at each layer, and does not go back to the previous layer. The bottom-up BFS algorithm can avoid the problems of redundant edge retrieval and update conflicts, but it is not very effective for graphs with low out-degree. Therefore, how to process larger-scale graph data and how to further improve the efficiency of the algorithm are the current problems that need to be solved.

[0005] In summary, although there have been significant achievements in the field of CPU-GPU heterogeneous system graph traversal, there are still some shortcomings. For example, the graph traversal algorithm has not been optimized. For large-scale graph data, optimizing the graph traversal algorithm can significantly improve computing efficiency, reduce computing time and resource consumption, and thus speed up data processing and analysis. Therefore, how to optimize the traversal algorithm for CPU-GPU heterogeneous system graph calculation has become a work focus. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a BFS graph traversal optimization method for heterogeneous graph processing systems, optimize the traditional BFS algorithm into a fused BFS algorithm, improve the traversal speed, and achieve the goal of improving efficiency and reducing resource consumption when processing large-scale data graphs.

[0007] In order to achieve the above object, the technical solution adopted by the present invention is: a BFS graph traversal optimization method for a heterogeneous graph processing system, comprising:

[0008] S1. Partition the graph data, sort the graph vertices according to key attributes, and then assign the sorted vertices to different partitions;

[0009] S2. Sort each vertex and sort the vertices in the dictionary in descending order according to their out-degree;

[0010] S3. Allocate the number of threads according to the out-degree of each vertex to process the data through the threads.

[0011] In step S1, the graph data is partitioned using 1-D data partitioning, where the 1-D data partitioning is to divide the data into one-dimensional linear order.

[0012] The step S1 also includes:

[0013] Step 1: Select 1-D data as the basis for division according to the attributes of the nodes in the graph data, and obtain the degree of each vertex in the graph data;

[0014] Step 2: Sort the selected 1-D data so that it can be divided in the subsequent steps and determine the number of divisions;

[0015] Step 3: Divide the 1-D data into a corresponding number of intervals according to the determined number of divisions;

[0016] Step 4: Assign each node in the graph data to the corresponding partition interval.

[0017] Step 4 in step S1 includes: allocating nodes to corresponding partition intervals according to the corresponding relationship between node attributes and 1-D data.

[0018] Step S2 includes: when sorting the vertices, using the sorted() function to sort the vertices in the dictionary in descending order according to the out-degree; using the lambda function as a keyword parameter for sorting, so as to sort according to the out-degree and select the first Z% as the central vertices.

[0019] The step S2 further comprises:

[0020] (1) Use the BFS algorithm to traverse the graph data, define a vertex queue to record the vertices of each traversed level, and store new vertices in the vertex queue when they are found. Then compare the same values ​​in the queue with those in the central vertex queue, convert the queue into a set, obtain the same values ​​in the two sets, and return the number of the same values.

[0021] (2): Repeat the above step (1) to obtain the number of central vertices at each traversal level, and finally obtain the value of α.

[0022] The transformation parameter α is defined as:

[0023]

[0024] Where Fh is the number of central vertices traversed in the current queue, and Th represents the total number of central vertices.

[0025] The edges are classified into multiple queues according to their out-degree, which refers to the number of edges adjacent to a vertex, and different numbers of threads are assigned to each queue to balance the workload among threads.

[0026] A set range is selected to process vertices with different out-degrees, and four queues are defined, including: Small-Queue, Middle-Queue, Large-Queue and Extreme-Queue. In the next level of operation, four cores with different numbers of threads are assigned to work on different boundary queues to balance the workload between threads. At the same time, through asynchronous processing operations, after a thread completes a task, it does not need to wait for the completion of this batch of tasks and directly starts processing the next stage of work tasks to reduce thread waiting time.

[0027] The advantages of the present invention are: improving search efficiency: the top-down method traverses layer by layer through a single queue, while the bottom-up method avoids redundant edge retrieval and update conflict problems through double queue optimization. This fusion method combines the advantages of both methods, thereby improving the overall search efficiency. At the same time, it also enhances the robustness of the algorithm. By combining the advantages of the two methods, the algorithm can more effectively handle various complex situations and reduce potential errors and conflicts. In addition, it also reduces thread resource loss, achieving a balance between resource consumption and performance processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The following is a brief description of the contents expressed in the drawings of the present invention and the symbols in the drawings:

[0029] Figure 1 It is a conversion diagram of the BFS algorithm of the present invention;

[0030] Figure 2 This is a simplified schematic diagram of the BFS traversal of the present invention;

[0031] Figure 3 It is a flow chart of the traversal method of the present invention;

[0032] Figure 4 This is a flowchart of the implementation of the BFS algorithm integrated with the present invention;

[0033] Figure 5 This is a schematic diagram of thread allocation according to the present invention. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention will be further explained in detail below by describing the optimal embodiment with reference to the accompanying drawings.

[0035] This work designs a method to fuse the BFS algorithm when traversing the CPU-GPU heterogeneous system graph, optimizes the traditional BFS algorithm into a fused BFS algorithm, improves the traversal speed, and enables it to achieve the goal of improving efficiency and reducing resource consumption when processing large-scale data graphs.

[0036] The traditional BFS algorithm refers to a top-down BFS algorithm, which starts from the starting node, first visits all its directly adjacent nodes (i.e., nodes with a distance of 1), and then visits the neighbor nodes of these adjacent nodes layer by layer (i.e., nodes with a distance of 2), and so on, until the target node is visited or the entire graph is traversed.

[0037] The bottom-up BFS algorithm starts from the starting node, first visits the node closest to the starting node, and then expands outward layer by layer until the target node is found or all nodes are traversed. During the search process, BFS uses a queue to manage the nodes to be visited, ensuring that each visit is the node that was added to the queue the earliest, thereby achieving hierarchical traversal.

[0038] The traditional BFS algorithm is prone to redundant edge retrieval and update conflicts; and the bottom-up BFS algorithm is not effective for graphs with low out-degree. Therefore, we can combine the two BFS algorithms organically. The two algorithms complement each other. When the out-degree of the layer is high, the traditional BFS algorithm is skipped. On the contrary, when the out-degree of the layer decreases, it is more suitable for the traditional BFS algorithm, which will greatly improve the traversal speed. For this, we define a conversion parameter to determine whether the out-degree of the current layer needs to be converted, and control the mutual conversion of the two algorithms. When traversing each layer, the number of central vertices of the layer is collected and then compared with the number of vertices of the layer. When the proportion of central vertices is higher than α, it is judged that the out-degree of the current processing queue is high, and the bottom-up BFS algorithm is used for traversal. The effect is better. If the proportion is less than α, it means that the current processing queue can use the top-down traditional BFS algorithm.

[0039] First, we need to partition the graph data using 1-D data partitioning, which is a simple data partitioning method that divides the data into one-dimensional linear order. In graph partitioning, this usually means sorting the vertices of the graph according to a key attribute (such as vertex ID) and then assigning these vertices to different partitions. The steps are as follows:

[0040] Step 1: According to the attributes of the nodes in the graph data (such as the degree and weight of the nodes), select a suitable 1-D data as the basis for division, and use node:len(list(G.neighbors(node)) to obtain the degree of each vertex in the graph data. G.neighbors(node) is usually used to obtain the neighbor nodes of a node. Specifically, it returns all nodes directly connected to the specified node. list(G.neighbors(node)) converts the iterator of this neighbor node into a list. Therefore, len(list(G.neighbors(node))) calculates the number of neighbor nodes directly connected to the specified node node. That is, it represents the degree of the node, that is, the number of edges directly connected to the node in the graph.

[0041] Step 2: Sort this 1-D data so that it can be partitioned in the subsequent steps. Determine the number of partitions. This can be based on the size, structure, or other factors of the graph data.

[0042] Step 3: Divide the 1-D data into a corresponding number of intervals according to the determined number of partitions. For example, if the graph data needs to be divided into k parts, the 1-D data can be divided into k equally spaced intervals.

[0043] Step 4: Assign each node in the graph data to the corresponding partition interval. Specifically, assign the node to the corresponding partition interval according to the corresponding relationship between the node attribute and the 1-D data.

[0044] Then sort the vertices, and use the sorted() function to sort the vertices in the dictionary (a dictionary is a mutable data structure for storing key-value pairs. It allows you to quickly access the corresponding value by key. It is an unordered set of key-value pairs, where each key must be unique and immutable (such as a string, number, or tuple), and the value can be of any data type. The dictionary is defined by curly braces {}, with keys and values ​​separated by colons, and multiple key-value pairs separated by commas) in descending order according to the out-degree. Here we use the lambda function as the keyword parameter for sorting, so that we can sort by out-degree and select the top 5% as the central vertices. This article defines the conversion parameter α as:

[0045]

[0046] Where Fh is the number of central vertices in the current queue (collected at each layer), and Th represents the total number of central vertices. The current queue traversed refers to the data processing queue during graph data processing.

[0047] The sorted() function is a powerful built-in function for sorting iterable objects. It returns a new list instead of modifying the original list.

[0048] The lambda function is a concise way to define a function. It does not require the def keyword, but directly defines an anonymous function through the lambda keyword. Its basic syntax is lambda arguments:expression, where arguments represents the function's parameter list, and expression is the result expression returned by the function.

[0049] Step 1: Use R-MAT to generate experimental graphs. Import the networkx library and define a function called generate_rmat that accepts five parameters: the number of nodes (n), the number of edges (m), the probability of connecting existing nodes (a), the probability of connecting new nodes (b), and the probability of connecting to new nodes (c). Generate some experimental graphs and then process the experimental graphs. The R-MAT graph generator is a tool for generating graphs with scale-free distribution characteristics. Designed based on the principle of recursive matrix partitioning, this design makes the generated graph exhibit scale-free distribution characteristics in terms of vertex degree. The R-MAT graph generator provides multiple template classes that allow users to customize the graph generation process as needed, including parameters such as the number of vertices, the number of edges, the probability distribution of edges, and whether to permute vertex indices.

[0050] NetworkX is a powerful Python library for creating, manipulating, and studying complex networks. It can create graphs in a variety of ways, including directly creating nodes and edges, building graphs from existing data, and generating graphs from graph algorithms. You can perform various graph operations, such as adding, deleting, and modifying nodes and edges, getting a list of nodes and edges of a graph, and calculating the degree and neighbors of a graph. It has built-in commonly used graph and complex network analysis algorithms, such as the shortest path problem, connectivity problem, centrality problem, community detection problem, etc. These algorithms can help users solve various problems, such as finding the shortest path between two nodes, determining whether a graph is connected, finding the most important nodes in a graph, or discovering the community structure in a graph. It also provides a wealth of visualization capabilities to help users better understand and display graph structures. Users can use different drawing methods to draw graphs of nodes and edges, draw according to the properties of nodes and edges, and use different layout algorithms to lay out graphs.

[0051] The generate_rmat function is a C++ template class function used to generate an R-MAT graph. It is mainly used to generate an R-MAT graph with specific properties. This graph has a scale-free distribution in terms of vertex degree and is implemented using recursive matrix partitioning. The parameters include: RandomGenerator: This is a random number generator type used to generate probability values ​​for edges. Graph: This is the type of graph, usually a graph type based on an adjacency list or other graph structure. EdgePredicate: This is an optional edge predicate type used to specify which edges to keep. By default, it keeps all edges.

[0052] The experimental graph data is obtained through experiments in order to verify this solution. However, when the graph data is actually traversed and processed, the graph data is already available during use and does not need to be obtained experimentally. The following step 2 is directly executed on the graph data. For ease of understanding, the graph data to be processed is referred to as experimental graph data here.

[0053] Step 2: Use the traditional BFS algorithm for preprocessing to obtain the number of central vertices, traverse the experimental graph, define a vertex queue to record the vertices of each traversed level, and store new vertices in the vertex queue when they are found. Then compare the same values ​​of this queue with the central vertex queue, use set1.intersection to convert the queue into a set, and then use the compare_queues function to obtain the same values ​​of the two sets and return the number of identical values.

[0054] Step 3: Repeat the above steps to get the number of central vertices at each traversal level, and finally get the value of α.

[0055] After getting the value of α, the whole traversal process can be started, and the large graph data can be processed. The threads are allocated during the traversal process. The corresponding number of threads is allocated according to the out-degree of each vertex. The working thread load is adjusted during the graph traversal on the GPU side. Different numbers of threads are allocated for processing according to the boundaries of different out-degrees. At the same time, asynchronous processing is used to avoid waiting of threads at the same level and directly enter the traversal of the next level. This not only improves the traversal speed, but also reduces the waste caused by idle threads. At the same time, it can better adapt to larger-scale data processing tasks, thereby meeting the needs of more application scenarios.

[0056] Finally, the number of threads can be allocated according to the out-degree of each vertex to improve the utilization of thread resources. The out-degree of each point in the graph is different. If a boundary queue has more edges, then the GPU thread assigned to it naturally needs to perform more operations. In order to alleviate the imbalance of workload between threads, the boundaries can be classified into multiple queues according to the out-degree (the number of edges between adjacent vertices), and different numbers of threads are allocated to each queue to balance the workload between threads. Select a suitable range to process vertices with different out-degrees, and define four queues, namely Small-Queue, Middle-Queue, Large-Queue and Extreme-Queue. In the next level of operation, four cores (Thread, Warp, Block and Grid) with different numbers of threads will be assigned to work on different boundary queues to balance the workload between threads. At the same time, through asynchronous processing operations, after a thread completes a task, it does not need to wait for the completion of this batch of tasks and directly starts to process the next stage of work tasks, reducing thread waiting time.

[0057] Obviously, the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concept and technical solution of the present invention, they are all within the protection scope of the present invention.

Claims

1. A BFS graph traversal optimization method for a heterogeneous graph processing system, characterized by: include: S1. Partition the graph data, sort the graph vertices according to key attributes, and then assign the sorted vertices to different partitions; S2. Sort each vertex and sort the vertices in the dictionary in descending order according to their out-degree; S3. Allocate the number of threads according to the out-degree of each vertex to process the data through the threads.

2. A BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 1, characterized in that: In step S1, the graph data is partitioned using 1-D data partitioning, where the 1-D data partitioning is to divide the data into one-dimensional linear order.

3. A BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 2, characterized in that: The step S1 also includes: Step 1: Select 1-D data as the basis for division according to the attributes of the nodes in the graph data, and obtain the degree of each vertex in the graph data; Step 2: Sort the selected 1-D data so that it can be divided in the subsequent steps and determine the number of divisions; Step 3: Divide the 1-D data into a corresponding number of intervals according to the determined number of divisions; Step 4: Assign each node in the graph data to the corresponding partition interval.

4. A BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 3, characterized in that: Step 4 in step S1 includes: allocating nodes to corresponding partition intervals according to the corresponding relationship between node attributes and 1-D data.

5. The BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 1, characterized in that: Step S2 includes: when sorting the vertices, using the sorted() function to sort the vertices in the dictionary in descending order according to the out-degree; using the lambda function as a keyword parameter for sorting, so as to sort according to the out-degree and select the first Z% as the central vertices.

6. A BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 5, characterized in that: The step S2 further comprises: (1) Use the BFS algorithm to traverse the graph data, define a vertex queue to record the vertices of each traversed level, and store new vertices in the vertex queue when they are found. Then compare the same values ​​in the queue with those in the central vertex queue, convert the queue into a set, obtain the same values ​​in the two sets, and return the number of the same values. (2): Repeat the above step (1) to obtain the number of central vertices at each traversal level, and finally obtain the value of α.

7. A BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 6, characterized in that: The transformation parameter α is defined as: Where Fh is the number of central vertices traversed in the current queue, and Th represents the total number of central vertices.

8. The BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 1, characterized in that: The edges are classified into multiple queues according to their out-degree, which refers to the number of edges adjacent to a vertex, and different numbers of threads are assigned to each queue to balance the workload among threads.

9. A BFS graph traversal optimization method for a heterogeneous graph processing system according to claim 8, characterized in that: A set range is selected to process vertices with different out-degrees, and four queues are defined, including: Small-Queue, Middle-Queue, Large-Queue and Extreme-Queue. In the next level of operation, four cores with different numbers of threads are assigned to work on different boundary queues to balance the workload between threads. At the same time, through asynchronous processing operations, after a thread completes a task, it does not need to wait for the completion of this batch of tasks and directly starts processing the next stage of work tasks to reduce thread waiting time.

Images