Data compression method, data decompression method, data compression system, and related devices for graph data

Abstract

The invention provides a graph data compression method. The method comprises the steps of obtaining original graph data and determining a node pair of each edge in the original graph data; grouping the same source nodes in the node pairs according to a preset grouping rule, wherein one source node is reserved in each group; storing the data into the corresponding FIFO according to the group type of each group; adding labels to the source node data of the source node and the destination node data of the destination node; and polling and storing the FIFO to obtain compressed graph data. According to the method, different compression ratios are adopted for the source nodes with different numbers corresponding to the destination nodes, only one source node is reserved in the same group, and repeated source nodes do not need to be stored; and large-scale parallel decompression of the graph data can be supported, and the processing bandwidth of the graph data is greatly increased. The invention further provides a data compression system of a graph calculation system, a computer readable storage medium and a data compression terminal, which have the above beneficial effects.

Description

technical field [0001] The present application relates to the field of data storage, and in particular to a method for compressing and decompressing image data, a compression system and related devices. Background technique [0002] In the era of big data, a large amount of data interacts with each other to form a large-scale graph, which is widely used in various algorithms such as PageRank. At present, the scale of the graph often reaches the order of tens of millions to hundreds of millions of nodes, and the edge (connection between nodes) information of the graph reaches the scale of billions. Usually, Internet graph data is in the form of a sparse matrix, and the storage methods that can be used include COO and CSR. COO uses the triplet format of <source node, destination node, weight>. Considering that a node is represented by 4 bytes, a Edges require 12 bytes of storage. CSR considers that the same row node has the same address, and only stores the same row of...

AI Technical Summary

Problems solved by technology

Since the length of column nodes is randomly distributed, it will be very difficult to replicate multiple node information in parallel on hardware

For example, in a graph data with a parallel processing capacity of 128 nodes, the input row nodes correspond to several column nodes, and copying at this time requires huge decoding logic and time-consuming

Images