A real-time graph data processing system and method based on bsp model

A data processing system and data processing technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of not meeting real-time requirements, low statistical query efficiency, unreasonable storage structure, etc., to speed up access performance, access efficiency, the effect of preserving locality

Active Publication Date: 2017-12-15
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF2 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0009] The technical problem to be solved by the present invention is to provide a real-time graph data processing system and method based on the BSP model, which is used to solve the problems of unreasonable storage structure, low statistical query efficiency, and failure to meet real-time requirements in the existing graph data processing technology and low efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A real-time graph data processing system and method based on bsp model
  • A real-time graph data processing system and method based on bsp model
  • A real-time graph data processing system and method based on bsp model

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0067] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0068] Existing graph data processing systems generally include three levels:

[0069] The first level is the data storage layer, which is mainly responsible for storing graph data, and at the same time provides an efficient concurrent access interface, providing powerful storage support for graph data processing.

[0070] The second layer is the graph data statistics query layer. This layer is mainly responsible for responding to user query statistics requests. These jobs are characterized by only accessing the graph data once, but the total amount of data accessed is directly related to the size of the job. Therefore, when the cluster system runs multiple jobs, it involves the load balancing problem of the entire ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present invention relates to a real-time graph data processing system and method based on a BSP model. The system includes: a data storage unit for preprocessing graph data, and according to "memory storage-distributed memory storage-distributed file system" Three-tier storage structure storage, generating jobs based on graph data; the graph data query and statistics unit is used to query and count graph data, decompose the jobs generated by the data storage unit into multiple tasks, and distribute them to corresponding computing nodes in a balanced manner. Then count the calculation results of each task, and combine the calculation results of all tasks as the final result to return to the user; the graph data analysis and processing unit is used to make each calculation node execute the decomposed tasks through iterative calculations, and realize each task through message passing. Synchronize iterative calculations and output the calculation results of tasks. The method realizes real-time graph data processing based on the system, and has the advantages of high access efficiency, maintaining cluster load balance, and accelerating BSP model execution efficiency.

Description

technical field [0001] The invention relates to the field of large-scale graph data processing, in particular to a real-time graph data processing system and method based on a BSP model. Background technique [0002] In recent years, with the rapid development and popularization of the SNS (Social Network Service) platform, graph data, which is the data representation of the platform, is also in a state of information expansion. In order to express more information, the expression form of graph data is becoming more and more complex, and the amount of data is also increasing. [0003] At the same time, the number of data items in graph data will be larger, and the relationship between data and data will be more complex, and data does not exist in isolation. Therefore, the storage of graph data will face greater challenges. In addition, how to process such large-scale graph data to achieve the purpose of mining hidden information is also a problem that graph data processing...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
Inventor 周薇韩冀中戴娇张章
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap