A Design Method for Parallel Processing Framework Supporting Large-Scale Dynamic Graph Data Query

Abstract

The present invention provides a parallel processing framework and design method supporting large-scale dynamic graph data query. The design method is as follows: the parallel processing framework combines the MapReduce framework and the BSP framework, takes the Hadoop framework as the underlying foundation, and adopts the "MapReduce+BSP" mode To meet the needs of large amount of data and multiple iterations of graph calculation, at the same time, the parallel processing framework realizes real-time monitoring function to check the dynamic changes of graph files by building iteration control module, real-time monitoring module, Map stage function design, and Reduce stage function design . The framework designed by the invention combines the advantages of the MapReduce framework and the BSP framework into one, and avoids the disadvantage that the MapReduce framework cannot perform multiple iteration calculations while solving the problem that the BSP framework cannot realize high-throughput calculations.

Description

technical field [0001] The invention belongs to the technical field of large-scale graphs, and in particular relates to a design of a graph data processing framework, in particular to a design of a parallel processing frame supporting large-scale dynamic graph data query. Background technique [0002] With the rapid development of "big data" and "cloud computing" technologies, a new generation of information and communication technologies has emerged. Intelligent transportation, social networking and other fields have been difficult to use traditional Internet technology to meet the needs of the public. How to deal with the complex graph structure behind these fields has become the focus of more and more research institutions. In an environment where the amount of information grows and the relationship between data is complex, the processing technology of large-scale dynamic graphs is changing with each passing day. Research institutions represented by Google, Facebook, Apa...

AI Technical Summary

Problems solved by technology

Comparing the two basic frameworks, the MapReduce framework can perform batch processing on large blocks of files, such as figure 1 As shown, the framework has a relatively mature programming interface, is easy to program and implements, and has a high degree of versatility and abstraction. However, for iterative calculations, jobs can only be started multiple times in a chain, resulting in high network transmission overhead and greatly reducing It improves processing efficiency, so it is not suitable for iterative calculations and calculations with high real-time requirements; while the BSP framework introduces the concept of "super step", which is a state framework, such as figure 2 As shown, this framework can better solve the problem of iterative computing, and is suitable for iterative computing and matrix computing. However, because it is a framework based on memory computing, it has high requirements for the memory performance of each node in the cluster, and the existing implementation framework In terms of version stability and various mechanisms, etc., it is not mature enough, and the data calculation throughput is weak

Images