Unlock instant, AI-driven research and patent intelligence for your innovation.

Parallel processing framework supporting large-scale dynamic data query and design method

A parallel processing and data query technology, which is used in electrical digital data processing, special data processing applications, relational databases, etc.

Active Publication Date: 2018-03-16
LIAONING UNIVERSITY
View PDF10 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] In order to solve the shortcomings of the existing large-scale graph processing framework, the present invention provides a parallel processing framework and design method that supports large-scale dynamic graph data query, which can effectively support the processing operations of large-scale dynamic graphs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Parallel processing framework supporting large-scale dynamic data query and design method
  • Parallel processing framework supporting large-scale dynamic data query and design method
  • Parallel processing framework supporting large-scale dynamic data query and design method

Examples

Experimental program
Comparison scheme
Effect test

example

[0076] (1) A small cluster is constructed, which consists of three identical PCs, one of which is used as the Master node, and the other two are used as Slave nodes. Use the classic Dijkstra single-source shortest path algorithm to process graph files. In the single-source shortest path algorithm, a given graph G=(V, E, W), where V is the set of vertices, E is the set of directed edges, and W is the set of non-negative weights. Select a vertex v in V as the source, and calculate the shortest path length from v to other vertices, that is, find the minimum value of the sum of the weights of each edge.

[0077] (2) Dijkstra's algorithm generates the shortest path from the source point to each vertex according to the increasing order of the weights between each vertex in the vertex set and the source point. Its algorithm is similar to the breadth-first search traversal algorithm of the graph, that is, to find the shortest path with the smallest weight first, and then refer to it ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a parallel processing framework supporting large-scale dynamic data query and a design method. The design method comprises the steps that the parallel processing framework combines a MapReduce framework and a BSP framework, a Hadoop framework is used as a bottom foundation, a "MapReduce + BSP" mode is adopted to meet the demands of a large size and multiple iterations of graphic calculation data, and meanwhile the parallel processing framework achieves a real-time monitoring function to check dynamic changes of a graphic file through iteration control module construction, real-time monitoring module construction, Map stage function design and Reduce stage function design. According to the designed framework, advantages of the MapReduce framework and the BSP frameworkare integrated, and the defect that the MapReduce framework cannot perform multiple iterative calculations is avoided while the problem that the BSP framework cannot realize high-throughput calculation is solved.

Description

technical field [0001] The invention belongs to the technical field of large-scale graphs, and in particular relates to a design of a graph data processing framework, in particular to a design of a parallel processing frame supporting large-scale dynamic graph data query. Background technique [0002] With the rapid development of "big data" and "cloud computing" technologies, a new generation of information and communication technologies has emerged. Intelligent transportation, social networking and other fields have been difficult to use traditional Internet technology to meet the needs of the public. How to deal with the complex graph structure behind these fields has become the focus of more and more research institutions. In an environment where the amount of information grows and the relationship between data is complex, the processing technology of large-scale dynamic graphs is changing with each passing day. Research institutions represented by Google, Facebook, Apa...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F17/30
CPCG06F16/27G06F16/284
Inventor 宋宝燕王俊陆单晓欢丁琳琳张毅隆尹东晓
Owner LIAONING UNIVERSITY