Large-scale graph data processing method

A processing method and technology of graph data, applied in the field of data processing, can solve problems such as low efficiency, and achieve the effects of reducing communication cost, reducing the number of global iterations, and reducing communication volume

Inactive Publication Date: 2014-07-09
NORTHWESTERN POLYTECHNICAL UNIV
View PDF1 Cites 9 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] In order to solve the technical problem of low efficiency of existing large-scale graph d

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Large-scale graph data processing method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] refer to figure 1 .

[0023] There is one master node and multiple worker nodes in this method. The master node is responsible for the management of each work node; assigns and schedules tasks to all work nodes; controls the global synchronization. After the fence synchronization is successful, the master node will instruct the work nodes to perform the next global iteration. Work nodes are used to perform specific tasks and are responsible for sending and receiving messages; work nodes periodically send "heartbeat" information to the master node, which includes the status of each work node, available system memory, etc.

[0024] This method adopts vertex-centered programming, and requires the user to rewrite the compute method of the vertex, and the compute method of each vertex is the same. For different algorithms, the compute method of vertices is not the same. However, it is generally divided into three major steps: (1) receiving the message sent by the previous...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a large-scale graph data processing method used for solving the technical problem that an existing large-scale graph data processing method is low in efficiency. According to the technical scheme, each parallel task is converted into a series of global iterative tasks, and iteration at every time is divided into three ordered stages, namely computation, global communication and barrier synchronization. The computation is divided into global communication and local communication, and the local computation includes a series of continuous internal iterations. The global communication stage includes that each working node sends a message of a current global iteration to a working node of a next global iteration. The barrier synchronization includes that a master node waits for that message passing of all the working nodes is completed, and then starts the next global iteration. Since multiple internal iterations are adopted during processing, times of the global iterations are decreased. Iteration times needed for highway data processing single-source shortest paths in the northeast of United States are decreased from more than 3800 times in the background art to 25 times; execution time is shortened from 1200s in the background art to 60s.

Description

technical field [0001] The invention relates to a data processing method, in particular to a large-scale graph data processing method. Background technique [0002] With the rapid growth of emerging applications such as social network analysis, bioinformatics network analysis, and machine learning, the demand for processing capabilities of large-scale graphs with hundreds of millions of vertices is becoming more and more urgent. The existing large-scale graph data processing methods mainly include methods based on MapReduce model and BSP model. [0003] The document "Pregel: A System for Large-Scale Graph Processing, SIGMOD, 2010, p135-145" discloses a method based on the BSP model. The method consists of a series of global iterations. During each iteration, each graph vertex receives the message sent by the previous iteration, then modifies its own value, and finally sends a message to the adjacent vertices. This method can effectively solve the processing problem of larg...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30
CPCG06F9/46G06F9/54
Inventor 陈群白松李战怀勾志营索勃潘巍
Owner NORTHWESTERN POLYTECHNICAL UNIV
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap