System and method for real-time graph data processing based on BSP (Board Support Package) model

A data processing system and data processing technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of not meeting real-time requirements, unreasonable storage structure, and low efficiency of statistical query, so as to speed up access Performance, access efficiency, and speed-up effects

CN103336808AActive Publication Date: 2013-10-02INST OF INFORMATION ENG CHINESE ACAD OF SCI
2 Cites 33 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Publication Date
2013-10-02

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention relates to a system and method for real-time graph data processing based on a BSP (Board Support Package) model. The system comprises a data storage unit, a graph data inquiring and counting unit and a graph data analyzing and processing unit, wherein the data storage unit is used for preprocessing graph data, storing the graph data according to a memory storage-distributed memory storage-distributed file system three-layer storage structure and generating an operation based on the graph data, the graph data inquiring and counting unit is used for inquiring and counting the graph data, decomposing the operation generated by the data storage unit into a plurality of tasks, distributing the tasks to corresponding calculating nodes in an equalized manner, carrying out statistics on all task calculation results, combining all the task calculation results and returning the combined task calculation results serving as a final result to users, and the graph data analyzing and processing unit is used for enabling all the calculating nodes to execute the decomposed tasks through iterative calculation, realizing the synchronization of iterative calculation of each time through message passing and outputting the task calculation results. The method is used for realizing the real-time graph data processing based on the system. The system and the method have the advantages that the access efficiency is high, the cluster load balancing is kept, the execution efficiency of the BSP model is accelerated, and the like.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] The invention relates to the field of large-scale graph data processing, in particular to a real-time graph data processing system and method based on a BSP model. Background technique

[0002] In recent years, with the rapid development and popularization of the SNS (Social Network Service) platform, graph data, which is the data representation of the platform, is also in a state of information expansion. In order to express more information, the expression form of graph data is becoming more and more complex, and the amount of data is also increasing.

[0003] At the same time, the number of data items in graph data will be larger, and the relationship between data and data will be more complex, and data does not exist in isolation. Therefore, the storage of graph data will face greater challenges. In addition, how to process such large-scale graph data to achieve the purpose of mining hidden information is also a problem that graph data processing...

Examples

Embodiment Construction

[0067] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.

[0068] Existing graph data processing systems generally include three levels:

[0069] The first level is the data storage layer, which is mainly responsible for storing graph data, and at the same time provides an efficient concurrent access interface, providing powerful storage support for graph data processing.

[0070] The second layer is the graph data statistics query layer. This layer is mainly responsible for responding to user query statistics requests. These jobs are characterized by only accessing the graph data once, but the total amount of data accessed is directly related to the size of the job. Therefore, when the cluster system runs multiple jobs, it involves the load balancing problem of the entire ...