Data processing method, data processing system, computer equipment and storage medium

Abstract

The invention discloses a data processing method, a data processing system, computer equipment and a storage medium, and belongs to the technical field of graph calculation. The embodiment of the invention provides a method. A data processing system with a distributed architecture is adopted; partial graph data is stored by each computing node in the data processing system. In a coding phase, primary aggregation is carried out at each computing node, interaction of vertex IDs between the computing nodes is carried out to complete second aggregation;; through the two aggregation processes, allthe computing nodes can store full-amount vertex IDs, decoding can be completed without communication in the decoding stage, the overall communication traffic can be reduced, and the algorithm execution efficiency can be improved.

Description

technical field [0001] The present application relates to the technical field of graph computing, and in particular to a data processing method, a data processing system, computer equipment, and a storage medium. Background technique [0002] The encoding and decoding of graph data plays an important role in graph computing. Generally speaking, in the commonly used graph data processing process, the process of encoding and decoding graph data is usually to encode the vertex ID (Identity, number) of the graph data first, then execute the graph algorithm in the specific business scenario, and finally convert the The output data is decoded. Specifically, encoding and decoding is to convert the vertex ID of the graph data into a data representation related to the number of vertices, so as to narrow the numerical range corresponding to the vertex ID, thereby improving the execution efficiency of the algorithm. [0003] However, the encoding and decoding in the process of graph ...

AI Technical Summary

Problems solved by technology

[0003] However, the encoding and decoding in the process of graph data processing mainly has the following disadvantages: in large-scale graph scenarios, since more computing resources are required, and each computing resource is usually performed by a single computing resource when encoding, in the decoding process It is necessary to communicate between various computing resources to obtain the original vertex ID corresponding to each coded vertex ID, resulting in a relatively large amount of communication between computing resources, which ultimately leads to low algorithm execution efficiency

Images