Data processing method, data processing system, computer equipment and storage medium

Abstract

The application discloses a data processing method, a data processing system, computer equipment, and a storage medium, which belong to the technical field of graph computing. In the method provided by the embodiment of the present application, by using a data processing system with a distributed architecture, each computing node in the data processing system stores part of the graph data, and in the encoding stage, each computing node performs an aggregation first, and then calculates Interaction of vertex IDs between nodes to complete the second aggregation. Through the above two aggregation processes, each computing node can store a full amount of vertex IDs. In the decoding stage, decoding can be completed without communication, which can reduce The overall communication volume improves the efficiency of algorithm execution.

Description

technical field [0001] The present application relates to the technical field of graph computing, and in particular to a data processing method, a data processing system, computer equipment, and a storage medium. Background technique [0002] The encoding and decoding of graph data plays an important role in graph computing. Generally speaking, in the commonly used graph data processing process, the process of encoding and decoding graph data is usually to encode the vertex ID (Identity, number) of the graph data first, then execute the graph algorithm in the specific business scenario, and finally convert the The output data is decoded. Specifically, encoding and decoding is to convert the vertex ID of the graph data into a data representation related to the number of vertices, so as to narrow the numerical range corresponding to the vertex ID, thereby improving the execution efficiency of the algorithm. [0003] However, the encoding and decoding in the process of graph ...

AI Technical Summary

Problems solved by technology

[0003] However, the encoding and decoding in the process of graph data processing mainly has the following disadvantages: in large-scale graph scenarios, since more computing resources are required, and each computing resource is usually performed by a single computing resource when encoding, in the decoding process It is necessary to communicate between various computing resources to obtain the original vertex ID corresponding to each coded vertex ID, resulting in a relatively large amount of communication between computing resources, which ultimately leads to low algorithm execution efficiency

Images