Power law graph flow division method in heterogeneous computing environment
By adopting the flow-based heterogeneous perceptual graph division method (SHAP) in a heterogeneous computing environment, using the power law distribution feature heuristic to select vertices, the performance imbalance of the graph division method and low information synchronization efficiency in the heterogeneous computing environment are solved. The problem of this achieves more efficient graph data processing and analysis.
Patent Information
- Application Number
- CN202411590114.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-11-08
- Publication Date
- 2025-02-14
AI Technical Summary
In a heterogeneous computing environment, existing graph division methods are difficult to effectively handle large-scale dynamic graphs, resulting in performance imbalance and low information synchronization efficiency.
A flow-based heterogeneous perceptual graph division method (SHAP) is proposed to select vertices through the power law distribution feature heuristic of graph data, reduce the graph processing time cost of each node, and effectively divide the power law in a heterogeneous computing environment picture.
SHAP effectively solves the problem of performance imbalance between nodes, improves the efficiency and performance of graph data processing and analysis, maintains a low replication coefficient in a heterogeneous computing environment, and improves the efficiency of graph processing algorithms.
Smart Images
Figure CN119441544A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of distributed graph computing and relates to a power-law graph flow partitioning method in a heterogeneous computing environment. Background Art
[0002] Graph analysis has become a widely used technology in the fields of biomedicine, social networks, transportation, etc. The human brain contains more than 80 billion neurons, each of which is connected to about 10,000 other neurons on average, forming a complex information transmission system. The development of the social field has also promoted the application of graph analysis technology. For example, the number of registered users of Facebook has exceeded 2 billion, forming a huge social network. The growing scale of graph data in these fields has promoted the development of many distributed graph computing systems. Therefore, it is crucial to partition the graph when executing graph processing algorithms in parallel.
[0003] In the field of graph data processing, traditional edge partitioning algorithms focus on minimizing the number of vertex cuts, but have limitations in achieving scalability, especially in minimizing the number of vertex replications. This approach may not effectively guarantee the efficiency of information synchronization between replicated nodes. In addition, existing edge partitioning methods fail to fully utilize cluster structure information to guide partitioning strategies, resulting in poor performance when processing large-scale dynamic graphs [1]. In recent years, researchers have begun to focus on flow-based graph partitioning techniques. This technique partitions incoming elements based only on the attributes of the current element and the information previously assigned to the partition without having to understand the global information of the entire input graph. Stream-based methods have become an ideal solution when considering the scale of input data and the constraints of available resources. However, stream partitioning methods are still challenging in heterogeneous environments. Therefore, this paper proposes a stream-based heterogeneity-aware graph partitioning method (SHAP) to effectively partition large graph data and improve the efficiency and performance of graph data processing and analysis.
[0004] [1] Hu, Weisong, Chao Tian, Xiaowei Liu, Hongwei Qi, Li Zha, Huaming Liao, Yuezhuo Zhang, and Jie Zhang. "Multiple-job optimization in mapreduce for heterogeneous workloads." In 2010 Sixth International Conference on Semantics, Knowledge and Grids, pp. 135-140. IEEE, 2010.
[0006] [2] A major factor affecting graph partitioning methods is the physical environment in which the partitions are located. Cluster nodes differ in processor architecture, network bandwidth, memory storage, and other aspects. Most of the existing edge partitioning algorithms assume an ideal homogeneous computing environment, and few algorithms take into account the impact of the hardware environment. When graph computing systems are built on various hardware platforms, these assumptions do not always hold true for partitioning in a homogeneous computing environment. Cluster environments are mainly affected by the following three main factors: hardware heterogeneity, virtualization technology, and diversified application requirements. Hardware heterogeneity refers to the fact that different types of computer hardware (such as CPU, memory, storage, etc.) have different performance characteristics and configurations. The widespread use of virtualization technology allows multiple virtual machine instances to run on the same physical server, each with a different operating system and configuration. Virtualization technology brings flexibility and resource sharing features, but it also adds heterogeneity to the cluster environment. Even though some research has been done on the heterogeneity of cluster environments, these static graph partitioning algorithms are usually only applicable to small-scale data and have poor scalability, making it difficult to adapt to the needs of large-scale dynamic graphs. Summary of the invention
[0007] Graph partitioning takes into account factors such as load balancing and communication minimization. However, the computing environment in which the graph computing system is located is complex. The physical cluster architecture has an impact on the distributed environment, including the heterogeneity of node processors and network bandwidth. In order to solve the problem of unbalanced performance between nodes, a flow-based heterogeneity-aware graph partitioning method (SHAP) is proposed. SHAP heuristically selects vertices based on the power-law distribution characteristics of graph data, reducing the expected time cost of graph processing for each node in the cluster. SHAP effectively partitions the power-law graph in a heterogeneous computing environment.
[0008] The technical solution adopted by the present invention is:
[0009] A power-law graph flow partitioning method in a heterogeneous computing environment. The overall process is as follows: Figure 1 As shown, the specific steps include:
[0010] Step 1: Stream loading mechanism
[0011] Step 1.1: Read the vertices in the data graph G and serialize them in a certain order.
[0012] Step 1.2: When loading the edge set of the graph data, use the streamingLoader() function to load it. This function loads one edge set E at a time l , until the memory limit CacheSize is reached or the edge set is empty.
[0013] Step 1.3: Load edge set E in each round l Then, use the while loop to traverse the currently loaded edge set El , until the edge set E l is empty or reaches CacheSize. In the loop, select the next edge e ∈ E l , and assign it to the partition Pi, while processing the edge using the CheckEdge(e, i) function, which may assign additional edges to other partitions. Update the edge set E l , until E l is empty or reaches CacheSize.
[0014] Step 2: Heuristic Selection
[0015] To obtain quick results during the processing and ensure that the partitioning quality is within an acceptable range, a heterogeneous-aware flow-based heuristic rule can be adopted, and the specific process is as Figure 2 shown.
[0016] Step 2.1: Introduce the core set and the boundary set
[0017] Introduce the core set C s and the boundary set B s , and adopt a heuristic method based on vertex characteristics.
[0018] Step 2.2: Process of checking edges
[0019] For the currently loaded edge set E l , check each edge e = {x, y} that satisfies the following conditions:
[0020] If e is covered by other partitions E j (i.e., e ∈ E j ), or e is touched by the boundary set C j , and satisfies |e ∩ C j | = 1 (i.e., max{dx, dy} ≤ 2m / n), then assign the edge e to the partition E j .
[0021] Step 2.3: Processing of the CheckEdge function
[0022] The CheckEdge() function adds additional unassigned edges to the edge set that satisfies the conditions j < i and E j ≤ α i ·m / p.
[0023] Step 2.4: Select the edge set
[0024] The selected edge set E i comes from the non-complete information of the graph, that is, it is selected from the currently loaded edge set E l .
[0025] Step 3: Termination condition and quality assessment
[0026] In the process of heuristic selection, an explicit judgment condition is also required. The specific process is as follows: Figure 3 shown.
[0027] Step 3.1: When the edge set E l Stop execution when it is empty or reaches CacheSize.
[0028] Step 3.2: Perform quality evaluation after each round of partitioning to check whether the computational cost of each partition meets the termination conditions.
[0029] The computational cost of each partition is defined as follows:
[0030] f p (E i )
[0031] =|E i |
[0032] When all partitions are divided, the computation cost of each partition has the following termination condition:
[0033]
[0034] The imbalance factor λ ≥ 0 is a constant parameter. i The relative isomerism factor is as follows:
[0035]
[0036] Therefore, we get the partition P for each round i The judgment termination condition of the partition can be expressed as follows:
[0037]
[0038] Beneficial effects of the present invention:
[0039] Solving the problem of unbalanced performance: To address the challenge of unbalanced performance among nodes in distributed graph computing systems, SHAP reduces the expected time cost of graph processing for each node in the cluster by heuristically selecting vertices, effectively solving the problem caused by unbalanced node performance.
[0040] Considering environmental complexity: SHAP takes into account the impact of the physical cluster architecture on the distributed environment, including the heterogeneity of node processors, network bandwidth, etc., so that it can better adapt to complex computing environments and improve the overall performance of the system.
[0041] Performance Superiority: In experiments, SHAP maintains a low replication factor in heterogeneous computing environments, improving the efficiency of graph processing algorithms and increasing runtime by 1.3 times to 11.37% compared to traditional methods.
[0042] Adaptability to complex environments: SHAP takes into account the impact of the physical cluster architecture on the distributed environment, including the heterogeneity of node processors, network bandwidth, etc., so that it can better adapt to complex computing environments and reduce the performance bottleneck of the entire system.
[0043] Effectiveness and Scalability: SHAP is compared with other state-of-the-art stream partitioning methods in different environments. The results show that it still maintains a low replication factor in heterogeneous computing environments, while having good scalability and can effectively process large-scale power-law graph data. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] Figure 1 This is the overall flow chart of the power-law graph flow partitioning method in a heterogeneous computing environment;
[0045] Figure 2 A flow chart for selecting vertices for the heuristic of the present invention;
[0046] Figure 3 Flowchart of judgment conditions for heterogeneous perception stream division. DETAILED DESCRIPTION
[0047] A power-law graph flow partitioning method in a heterogeneous computing environment. The overall process is as follows: Figure 1 As shown, the specific steps include:
[0048] Step 1: Stream loading mechanism
[0049] Step 1.1: Read the vertices in the data graph G and serialize them in a certain order.
[0050] Step 1.2: When loading the edge set of the graph data, use the streamingLoader() function to load it. This function loads one edge set E at a time l , until the memory limit CacheSize is reached or the edge set is empty.
[0051] Step 1.3: Load edge set E in each round l Then, use the while loop to traverse the currently loaded edge set E l , until the edge set E l Is empty or reaches CacheSize. In the loop, select the next edge e∈E l , and assign it to partition Pi, and use the CheckEdge(e,i) function to process the edge, possibly assigning additional edges to other partitions. Update edge set El , until E l is empty or reaches CacheSize.
[0052] Step 2: Heuristic Selection
[0053] In order to obtain quick results during the processing and ensure that the partitioning quality is within an acceptable range, a heterogeneous-aware flow-based heuristic rule can be adopted. The specific process is as Figure 2 shown.
[0054] Step 2.1: Introduce the core set and the boundary set
[0055] Introduce the core set C s and the boundary set B s , and adopt a heuristic method based on vertex characteristics.
[0056] Step 2.2: Process of checking edges
[0057] For the currently loaded edge set E l , check each edge e = {x, y} that satisfies the following conditions:
[0058] If e is covered by another partition E j (i.e., e ∈ E j ), or e is touched by the boundary set C j , and satisfies |e ∩ C j | = 1 (i.e., max{dx, dy} ≤ 2m / n), then assign the edge e to the partition E j .
[0059] Step 2.3: Processing of the CheckEdge function
[0060] The CheckEdge() function adds additional unassigned edges to the edge set that satisfies the conditions j < i and E j ≤ α i ·m / p.
[0061] Step 2.4: Select the edge set
[0062] The selected edge set E i comes from the non-complete information of the graph, that is, it is selected from the currently loaded edge set E l .
[0063] Step 3: Termination condition and quality evaluation
[0064] During the heuristic selection process, there also needs to be an explicit judgment condition. The specific process is as Figure 3 shown.
[0065] Step 3.1: When the edge set E lStop execution when it is empty or reaches CacheSize.
[0066] Step 3.2: Perform quality evaluation after each round of partitioning to check whether the computational cost of each partition meets the termination conditions.
[0067] The computational cost of each partition is defined as follows:
[0068] f p (E i )=|E i |
[0070] When all partitions are divided, the computation cost of each partition has the following termination condition:
[0071]
[0072] The imbalance factor λ ≥ 0 is a constant parameter. i The relative isomerism factor is as follows:
[0073]
[0074] Therefore, we get the partition P for each round i The judgment termination condition of the partition can be expressed as follows:
[0075]
Claims
1. A power-law graph flow partitioning method in a heterogeneous computing environment, characterized in that: Here are the steps: Step 1: Stream loading mechanism Step 1.1: Read the vertices in the data graph G and serialize them in a certain order; Step 1.2: When loading the edge set of the graph data, use the streamingLoader() function to load it. This function loads one edge set E at a time l , until the memory limit CacheSize is reached or the edge set is empty; Step 1.3: Load edge set E in each round l Then, use the while loop to traverse the currently loaded edge set E l , until the edge set E l Empty or reaches CacheSize; Step 2: Heuristic Selection Step 2.1: Introducing core and boundary sets Introducing the core set C s and the boundary set B s ,adopting a heuristic approach based on vertex features; Step 2.2: Check edge process For the currently loaded edge set E l , check that each edge e={x,y} satisfies the following conditions: If e is divided by other j Coverage (i.e. e∈E j ), or e is bounded by the boundary set C j Touch, and satisfy |e∩C j |=1, edge e is assigned to partition E j ; Step 2.3: CheckEdge function processing The CheckEdge() function adds additional unassigned edges to the edge set that satisfies the conditions j < i and E j ≤ α i · m / p Step 2.4: Select edge set Selected edge set E i From the incomplete information of the graph, that is, from the currently loaded edge set E l Select from; Step 3: Termination conditions and quality assessment Step 3.1: When the edge set E l Stop execution when it is empty or reaches CacheSize; Step 3.2: Perform quality evaluation after each round of partitioning to check whether the computational cost of each partition meets the termination conditions.
2. The method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 1, characterized in that: In step 1.3, in the loop, select the next edge e∈E l , and assign it to partition Pi, and use the CheckEdge(e,i) function to process the edges and assign additional edges to other partitions; update the edge set E l Until E l Is empty or CacheSize is reached.
3. A method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 1 or 2, characterized in that: The specific operation of step 2.1 is as follows: introduce the core set C s and the boundary set B s , a heuristic method based on vertex features is adopted.
4. A method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 1 or 2, characterized in that: In step 3.2, the specific operations are as follows: The computational cost of each partition is defined as follows: f p (AND i )=|And i | When all partitions are divided, the computation cost of each partition has the following termination condition: Each partition α i The relative isomerism factor is as follows: Therefore, we get the partition P for each round i The judgment termination condition of the partition is expressed as follows:
5. The method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 3, characterized in that: In the step 3.2, the specific operations are as follows: The computational cost of each partition is defined as follows: f p (AND i )=|And i | When all partitions are divided, the computation cost of each partition has the following termination condition: Each partition α i The relative isomerism factor is as follows: Therefore, we get the partition P for each round i The judgment termination condition of the partition is expressed as follows:
6. The method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 4, characterized in that: The imbalance factor λ≥0.
7. The method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 5, characterized in that: The imbalance factor λ≥0.
8. The method for partitioning a power-law graph flow in a heterogeneous computing environment according to claim 1, characterized in that: In step 2.2, |e∩C j |=1, that is, max{dx,dy}≤2m / n.