Operation system based on hierarchical large-scale graph data

Abstract

The invention discloses an operation system based on hierarchical large-scale graph data. The system comprises a graph data collection unit, a graph data analysis unit and a graph data management unit, wherein the graph data analysis unit comprises a graph data partitioning module, a statistical module and a graph data merging module; the graph data management unit comprises an operation module, acomparison module and an alarm module; and the graph data collection unit is used for collecting the large-scale graph data, performing noise filtering processing on the graph data through median filtering and then transmitting processed graph data to the graph data analysis unit and the graph data management unit. According to the system, the graph data is partitioned according to adjacent nodesof the graph data after being preprocessed, then the partitioned graph data is integrated, boundary node collection is performed on the preprocessed graph data to obtain an original boundary, meanwhile, the original boundary is compared with the integrated data, the precision of the partitioned data is judged, and therefore the accuracy of the graph data is ensured.

Description

technical field [0001] The invention belongs to the field of large-scale graph data processing, and relates to a computing system based on hierarchical large-scale graph data. Background technique [0002] In the era of big data mining, graphs can not only directly describe many practical applications in the fields of computer science, chemistry, and bioinformatics, such as social networks, web (web) graphs, chemical substances, and biological structures, but also can be used to describe various A data mining algorithm, such as matrix factorization or shortest path, etc. Among them, the graph includes a plurality of nodes and the edges connecting each node, the graph data includes the node data of each node and the edge data of the edges connecting each node, and the edge data of an edge includes the source node, the destination node and the The weight of the edge. In a stand-alone graph computing processing platform (that is, a processing platform that uses a single compu...

AI Technical Summary

Problems solved by technology

For example, in GraphChi (a stand-alone graph computing and processing platform), since the computing mode is centered on the destination node during graph computing, the computer sorts the graph data according to the ID (identification) of the destination node from small to large. The edge data in GraphChi is divided into multiple edge data blocks (called shards in GraphChi), all the edge data corresponding to the same destination node are divided into one edge data block, but the edge data blocks obtained by different segmentation rules are different, resulting in The data obtained when the final merge is less accurate

Images