System and method for real-time graph data processing based on BSP (Board Support Package) model
A data processing system and data processing technology, applied in the direction of electrical digital data processing, special data processing applications, instruments, etc., can solve the problems of not meeting real-time requirements, unreasonable storage structure, and low efficiency of statistical query, so as to speed up access Performance, access efficiency, and speed-up effects
Active Publication Date: 2013-10-02
INST OF INFORMATION ENG CHINESE ACAD OF SCI
View PDF2 Cites 33 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0009] The technical problem to be solved by the present invention is to provide a real-time graph data processing system and method based on the BSP model, which is used to solve the problems of unreasonable storage structure, low statistical query efficiency, and failure to meet real-time requirements in the existing graph data processing technology and low efficiency
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0067] The principles and features of the present invention are described below in conjunction with the accompanying drawings, and the examples given are only used to explain the present invention, and are not intended to limit the scope of the present invention.
[0068] Existing graph data processing systems generally include three levels:
[0069] The first level is the data storage layer, which is mainly responsible for storing graph data, and at the same time provides an efficient concurrent access interface, providing powerful storage support for graph data processing.
[0070] The second layer is the graph data statistics query layer. This layer is mainly responsible for responding to user query statistics requests. These jobs are characterized by only accessing the graph data once, but the total amount of data accessed is directly related to the size of the job. Therefore, when the cluster system runs multiple jobs, it involves the load balancing problem of the entire ...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more PUM
Login to view more Abstract
The invention relates to a system and method for real-time graph data processing based on a BSP (Board Support Package) model. The system comprises a data storage unit, a graph data inquiring and counting unit and a graph data analyzing and processing unit, wherein the data storage unit is used for preprocessing graph data, storing the graph data according to a memory storage-distributed memory storage-distributed file system three-layer storage structure and generating an operation based on the graph data, the graph data inquiring and counting unit is used for inquiring and counting the graph data, decomposing the operation generated by the data storage unit into a plurality of tasks, distributing the tasks to corresponding calculating nodes in an equalized manner, carrying out statistics on all task calculation results, combining all the task calculation results and returning the combined task calculation results serving as a final result to users, and the graph data analyzing and processing unit is used for enabling all the calculating nodes to execute the decomposed tasks through iterative calculation, realizing the synchronization of iterative calculation of each time through message passing and outputting the task calculation results. The method is used for realizing the real-time graph data processing based on the system. The system and the method have the advantages that the access efficiency is high, the cluster load balancing is kept, the execution efficiency of the BSP model is accelerated, and the like.
Description
technical field [0001] The invention relates to the field of large-scale graph data processing, in particular to a real-time graph data processing system and method based on a BSP model. Background technique [0002] In recent years, with the rapid development and popularization of the SNS (Social Network Service) platform, graph data, which is the data representation of the platform, is also in a state of information expansion. In order to express more information, the expression form of graph data is becoming more and more complex, and the amount of data is also increasing. [0003] At the same time, the number of data items in graph data will be larger, and the relationship between data and data will be more complex, and data does not exist in isolation. Therefore, the storage of graph data will face greater challenges. In addition, how to process such large-scale graph data to achieve the purpose of mining hidden information is also a problem that graph data processing...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more Application Information
Patent Timeline
Login to view more Patent Type & Authority Applications(China)
IPC IPC(8): G06F17/30
Inventor 周薇韩冀中戴娇张章
Owner INST OF INFORMATION ENG CHINESE ACAD OF SCI
Who we serve
- R&D Engineer
- R&D Manager
- IP Professional
Why Eureka
- Industry Leading Data Capabilities
- Powerful AI technology
- Patent DNA Extraction
Social media
Try Eureka
Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.
© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap