Adaptive Graph Partitioning Method for Heterogeneous Fusion Processors in Big Data

Abstract

The invention relates to an adaptive graph division method for a heterogeneous fusion processor in big data, which is characterized by comprising the following steps of: carrying out fine-grained division on a general graph load, and distributing loads with different irregular degrees to different equipment; analyzing dynamic graph load, designing adaptive graph division, and automatically identifying whether multi-device hybrid operation is needed or not; and for multi-device processing of a large-scale graph load, operating processing data by using a pipeline mode. Based on a CPU and GPU integrated architecture, an automatic programming framework system capable of meeting high-performance requirements in a heterogeneous and dynamic environment is constructed for a graph calculation program, and efficient fine-grained graph division is researched for new calculation characteristics of a large graph and a real-time dynamic graph.

Description

technical field [0001] The invention relates to an adaptive graph division method for heterogeneous fusion processors in big data, and relates to the field of heterogeneous computing. Background technique [0002] In the era of big data, it is difficult to use traditional CPU processors to cope with the processing requirements of large-scale loads, and the emergence of GPUs has brought new ideas for big data processing. More and more big data applications are processed by GPU. Graph computing is a representative of big data applications. Due to the complex and irregular relationship between points in the graph, it has always been a hot topic in big data research. At present, more and more researchers are paying attention to how to use GPU to accelerate big data graph computing applications. The traditional discrete GPU and CPU are not on the same chip, communication needs to pass through PCIe, and the data transmission efficiency is low, so the integrated processor of CPU ...

AI Technical Summary

Problems solved by technology

[0003] At present, many scholars are studying how to use the integrated architecture to accelerate the irregular program of graph calculation. The heterogeneous characteristics and load irregularity on the integrated architecture pose great challenges to the efficiency of data division. The existing technology has not solved it well. The main reasons are: First, many current works only perform coarse-grained data division, without considering the fine-grained interaction between CPU and GPU

Second, although some research has achieved fine-grained data partitioning, it is usually only applicable to specific applications, such as Hash Join and MapReduce in the database field, and cannot provide general automatic conversion for irregular programs based on graphs or sparse matrices

[0004] In addition, in order to make full use of the resources on the integrated architecture, a new division of program tasks is usually required. The current common division method is static division, which is only applicable to the first round of iterations. This division is not necessarily efficient after the graph is changed.

However, there is an overhead to repartitioning the graph after each iteration

In addition, for the integrated architecture, when the input graph data exceeds a certain scale in the big data environment, the program cannot be executed. The current technology does not solve this problem, and an additional segmentation design is required for the big data load.

Images