Graph data processing method, device and system

Abstract

The invention discloses a graph data processing method applied to a distributed system which comprises a master control node and multiple work nodes. The method includes the steps that the master control node acquires graph data and segments the graph data to obtain P fragments, wherein P is a positive integer, and the P fragments include the first fragment and the second fragment; at least two edge sets are determined from each fragment; the edge sets included in the first fragment of the P fragments are scheduled to at least two work nodes for processing; incidence edge sets included in thesecond fragment are scheduled to the work nodes for processing, wherein the incidence edge sets are edge sets including outgoing edges of a target vertex corresponding to the first fragment. By meansof the graph data processing method in the embodiment, a great deal of memory resources of the distributed system can be utilized for processing graph data, the overall throughput capacity of the system is increased, and the IO overhead can be reduced.

Description

technical field [0001] The present invention relates to the technical field of the Internet, in particular to a method, device and system for image data processing. Background technique [0002] With the rapid development of the Internet, the amount of data in the network is growing explosively. Through the analysis of these massive data, some valuable information can be mined to facilitate the discovery of new market opportunities or business models. However, the amount of data in the network is very large. In order to improve the efficiency of data analysis, it is necessary to model the data, that is, to express the data through a specific model. Since the graph (Graph) has a strong model expression ability, it can intuitively express the entities (Entities) with interconnections and the relationship between them. Therefore, in many big data analysis scenarios, the data to be processed is represented in the form of a graph, and then the graph is analyzed and processed to ...

AI Technical Summary

Problems solved by technology

[0004] Due to the rapid growth of the scale of graph data, the total memory of the graph processing system may also be smaller than the scale of the graph data to be processed, resulting in the failure of the graph processing task

At the same time, due to the strong dependence of graph data, if only part of the graph data is loaded into the memory for processing each time, the adjacency relationship between vertices in the graph data will often be destroyed, resulting in poor data locality. During the graph calculation process, the data associated with the currently calculated data needs to be frequently read from the disk, resulting in a large amount of IO overhead

Images