Lossless image data compression method based on improved image abstract algorithm

Abstract

The application discloses a lossless graph data compression method based on an improved graph abstract algorithm, comprising the following steps: after grouping nodes, screening groups with a node number greater than the maximum node number of the groups; based on the node merging stage calculation amount and execution time obtained by the node number of the different groups, the node number threshold is re-determined, and the screened groups are further divided to form new groups; between the multiple iterations of the node grouping stage and the node merging stage, the in-group nodes that can be merged are screened by using a Bloom filter, and the nodes meeting the merging benefit condition are merged. The method of the application limits the size of each group by taking the calculation amount as the measurement standard after grouping the nodes, identifies and filters redundant calculation by using a Bloom filter between the multiple iterations of the node grouping stage and the node merging stage, and accelerates the encoding calculation by using a multi-level hash structure in the edge structure encoding stage, so that the running efficiency of graph calculation can be improved and the storage cost of graph data can be reduced.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a lossless graph data compression method based on an improved graph summarization algorithm. Background Technology

[0002] With the development of the internet and society, graph models have been widely applied in various fields. Mining the structure of graphs and extracting useful information has rich research significance and practical value, such as conducting social network analysis, statistical analysis of financial networks, and detecting abnormal transactions. However, applying graph models and mining algorithms to research is becoming increasingly difficult. On the one hand, many graph mining and analysis algorithms have high time complexity and are easily affected by data growth in efficiency; on the other hand, the scale of graph data in the real world is growing larger and larger at an alarming rate. Therefore, how to reduce the scale of graph data and perform graph mining and analysis efficiently has become a major research hotspot.

[0003] Graph summarization is an efficient and scalable data compression algorithm that not only reduces data size and saves storage space, but also enables efficient graph querying and information recovery, helping to accelerate the execution of various graph mining and analysis algorithms on large-scale graph data. Lossless graph summarization algorithms compress data while ensuring that the vertices and edges of the graph can be fully recovered when needed, without losing any structural information (Navlakha, S., Rastogi, R., Shrivastava, N.: Graph summarization with bounded error. In: Proceedings of the ACM SIGMOD International Conference on Management of Data. pp. 419–432, 2008). Graph summarization algorithms are based on the Minimum Description Length (MDL) principle. This algorithm takes the original graph as input and outputs a concise summary graph and a correction set. The summary graph is a simplified version of the original graph. Its nodes are called supernodes, which represent a set of similar vertices in the original graph; its edges are called superedges, which represent all edges between two associated supernodes.

[0004] The general execution flow of graph summarization algorithms includes three stages: node grouping, node merging, and edge structure encoding (Shin, K., Ghoting, A., Kim, M., Raghavan, H.: Sweg: lossless and lossy summarization of web-scale graphs. In: The World Wide Web Conference.pp. 1679–1690 2019). However, graph summarization algorithms still suffer from insufficient execution efficiency and poor compression performance when applied to large-scale graph data, which urgently need to be addressed. Specifically, when calculating the optimal candidate node using node grouping, there is an unsuitable search space, leading to high time overhead and insufficient compression performance.

[0005] To address the issues encountered during the execution of the graph summarization algorithm, it is necessary to design optimization techniques to improve each stage, ensuring the compression performance of graph data while accelerating the algorithm's execution efficiency, thus meeting the need for efficient data compression in large-scale graph data scenarios. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the existing technology and provide a lossless graph data compression method based on an improved graph summarization algorithm.

[0007] To achieve the above technical objectives, the present invention adopts the following technical solution:

[0008] A lossless graph data compression method based on an improved graph summarization algorithm includes a node grouping stage, a node merging stage, and an edge structure encoding stage, and also includes:

[0009] The computational load and execution time of the node merging stage are obtained according to the number of nodes in different groups, and the maximum acceptable number of nodes in the initial group is determined according to the preset execution time threshold.

[0010] After grouping the nodes, select groups with more nodes than the maximum number of nodes in the selected groups. Based on the computational cost and execution time of the node merging stage obtained from the number of nodes in the different groups, redetermine the node number threshold and further divide the selected groups to form new groups.

[0011] Between multiple iterations of the node grouping and node merging phases, a Bloom filter is used to screen for mergeable nodes within a group, and nodes that meet the merging benefit criteria are merged.

[0012] As a preferred implementation, the threshold number of nodes in a group is determined based on the following formula. :

[0013]

[0014] in, It is a group The number of nodes, It is the group number. It is a predefined number of group nodes. Indicates the number of group nodes. Execution time required for the merge phase;

[0015] Based on the node number threshold Divide the groups into new groups.

[0016] As a preferred implementation, the groups are divided by multiple samplings, from the groups Perform multiple samplings, each sampling Nodes, generating A new group.

[0017] As a preferred embodiment, the step of using a Bloom filter to filter mergeable intra-group nodes includes:

[0018] Randomly select node A within the group, and then select the optimal candidate node B from the remaining nodes based on an approximate index. Use a Bloom filter to determine whether node A and node B are among the top nodes. The round of iterations has been calculated;

[0019] If yes, then remove node A from the group and randomly select node A within the group again for judgment, until the Bloom filter judgment result is no. At this time, calculate the merge benefit of nodes A and B and merge the nodes that meet the merge benefit conditions.

[0020] As a preferred implementation, for nodes that do not meet the conditions for merging benefits, it is further determined whether the merging benefits of nodes A and B are less than the conditions for merging in the next round.

[0021] If it is not less than, then remove node A from the group and randomly select node A from the group again for judgment; otherwise, calculate to satisfy the condition. The smallest Value, will the node pair and Insert the first A Bloom filter for the layer is used in the node merging phase of the next iteration;

[0022] in, The combined revenue of node A and node B, , Each is before and after The threshold for round iterations and when going forward The threshold for round iteration.

[0023] As a preferred implementation, the node with the largest Jacobian approximation index value among the remaining nodes in the group is selected as the optimal candidate node B.

[0024] In a preferred implementation, a unique value is calculated for nodes A and B through data transformation and used as a unique identifier for the node pair. This unique identifier is then used as the input to the Bloom filter to determine whether the node pair has been calculated.

[0025] As a preferred implementation, node A and node B are converted into binary values ​​and then concatenated sequentially to form a new value. , serving as a unique identifier for node pairs.

[0026] As a preferred implementation, in the node grouping stage, nodes are initially grouped based on whether their shingle values ​​are equal.

[0027] In a preferred implementation, after the node grouping stage and node merging stage are iterated a specified number of times, the edge encoding stage is entered. Super edges are generated between super nodes to form a summary graph and a corrected edge set. The corrected edge set is used to supplement the missing edge information in the summary graph and correct the redundant edge information introduced into the summary graph.

[0028] The method of this invention limits the size of each group by computational cost after node grouping. It uses a Bloom filter to identify and filter redundant computations between multiple iterations in the node grouping and node merging stages. It also uses a multi-level hash structure to accelerate encoding computation in the edge structure encoding stage. This can improve the running efficiency of graph computation and reduce the storage cost of graph data. Attached Figure Description

[0029] Figure 1 This is a framework diagram of a lossless graph data compression method based on an improved graph summarization algorithm.

[0030] Figure 2 A table for selecting criteria for node repartitioning based on computational load. Detailed Implementation

[0031] like Figure 1 As shown, the lossless graph data compression method based on the improved graph summarization algorithm includes three stages: node grouping, node merging, and edge structure encoding, as well as a redundancy calculation identification module. The redundancy calculation identification module is mainly used to identify and filter redundant calculations in the node merging stage to improve the execution efficiency of the algorithm. The operation steps of each stage / module are as follows:

[0032] 1. In the node grouping phase, nodes are divided into multiple groups based on their structural and connection similarities. This means that nodes falling into the same group imply a high degree of structural similarity, i.e., a high degree of overlap in their neighbor sets. The specific execution steps are as follows:

[0033] a) Generate a random mapping function Randomly number the nodes;

[0034] b) For each node Through formula compute nodes The shingle value is used to group two nodes together. If two nodes have the same shingle value, they are placed in the same group. .

[0035] When the nodes are grouped using the above steps, the following results are obtained: Subsequently, nodes falling within the same group have high structural and connectivity similarity. Merging nodes within the same group at this point has a greater chance of yielding higher merging benefits, i.e., reducing storage costs. However, this node grouping method has high randomness, and the generated groups are unevenly sized, meaning some groups have an excessively large number of nodes, reducing algorithm efficiency. Therefore, a computationally oriented node repartitioning method is used to limit the size of each group. The specific execution steps are as follows:

[0036] a1) For each group If the number of nodes is greater than 46, then we will base it on... Figure 2 The table and formulas shown Determine the threshold number of nodes for this group. , in the formula yes Figure 2 The second row of the table shown contains the predefined number of group nodes. express Figure 2 The table shown in row 3 contains the number of group nodes. The execution time required for the merge phase, It is a group The number of nodes, Figure 2 The first row of the table shown represents the number of group nodes. The amount of computation required for the merging phase;

[0037] b1) Based on the node number threshold Grouping To divide, that is, from grouping Perform multiple samplings;

[0038] c1) Each sampling Nodes, ultimately generating A new group.

[0039] 2. The node merging phase is executed after the node grouping phase. The node grouping phase generates groups using structural similarity function-based grouping and computationally oriented node repartitioning methods. Within each group, a node merging phase will be performed; nodes between groups will not be merged. The specific execution steps are as follows:

[0040] a) Traversing and processing the groups generated during the node grouping phase ;

[0041] b) For grouping For example, randomly selecting nodes within the group and calculate the nodes The Jacobian approximation index value is calculated using the formula for approximating all remaining nodes in the group. ,in Represents a node With super nodes The number of adjacent nodes in the middle. Supernode The number of nodes;

[0042] c) Select the node with the largest Jacobian approximation index value among all nodes. As the optimal candidate node, and calculate the node and The combined profit is calculated using the following formula: The denominator represents the node. and The numerator represents the storage cost required before merging, and the denominator represents the storage cost after merging. This revenue formula measures the storage cost of the merging nodes. and The amount of storage cost that can be reduced before and after merging. A higher benefit value indicates a better merging of nodes. and This can reduce storage costs significantly. Nodes The formula for calculating storage cost is as follows: ,in Represents a node and The cost of super-edge storage, among which Represents a node and The actual set of edges, Represents a node and All node pairs between them;

[0043] d) If node and Between the profit values Greater than the threshold of the current iteration (in , (This is the specified total number of iterations), i.e. Then merge nodes and ;

[0044] e) If node and The profit value between them is less than the threshold, that is... If so, the nodes will not be merged. and and nodes From grouping Delete it, return to step b) and select a new node. Until grouping All nodes have been processed.

[0045] During the node grouping phase between consecutive iterations, the structural similarity of nodes can lead to multiple nodes being repeatedly placed into the same group, even though the merging condition is not met. Calculating the profit for these nodes that repeatedly fall into the same group reduces the algorithm's computational efficiency, resulting in redundant computation. Therefore, this invention introduces a redundant computation identification module to identify and filter redundant computations. The specific steps are as follows:

[0046] a1) In the node merging phase, step c) calculates the optimal candidate node as... At this time, do not calculate the nodes first. and Instead of relying on merged revenue, it uses a Bloom filter to determine node performance. and If the node has been computed in the previous κ iterations, skip it. and The merging directly involves the nodes. From grouping Step b) is deleted and returned to the merge phase.

[0047] Specifically, first, pair the nodes. and After calculating a unique value through data transformation, it is used as the input to the Bloom filter, serving as a unique identifier for each node pair. In this embodiment, it is... and The value is converted to a binary value, and then concatenated in order to form a new value. As input to the Bloom filter, this unique identifier is used to determine node pairs. and Has it been calculated?

[0048] In each iteration (including the node grouping phase and the node merging phase), in the node merging phase, the Bloom filter is first used to determine whether the current node pair has been calculated. If it has, the merging is skipped; otherwise, proceed to b1.

[0049] b1) If the Bloom filter determines the node and Not in front The round of iterations has been calculated; now we begin calculating the nodes. and The combined benefits and the determination of nodes. and Does the consolidated revenue meet the consolidation criteria, i.e. If the conditions are met, then merge the nodes. and ;

[0050] If c1) is not satisfied, then continue to check the node. and Does the merger gain meet the condition of being less than the next round of merger, i.e. If the condition is not met, then directly remove the node. From grouping Step b) is deleted and returned to the merge phase.

[0051] d1) If satisfied, then calculate the minimum. Make Finally, the node pair and Insert into the A Bloom filter for the layer is used in the node merging phase of the next iteration. and That is, when going forward and backward The threshold of the wheel and when going forward and backward The threshold of the wheel.

[0052] 3. After the node grouping and node merging phases have been iterated a specified number of times, a set of supernodes is generated. The edge structure encoding stage is then performed. This stage generates super edges between super nodes, thus forming the summary graph. and corrected edge set To achieve the goal of not losing any original image structural information, the specific execution steps are as follows:

[0053] a) The set of edges traversed in the graph The neighbor set is sorted according to the node number, and then the sorted neighbor nodes are aggregated into a hyperedge structure, and then the formula is used to... Determine if it is in the section and Generate hyperedges between ;

[0054] b) If Then no hyperedge will be generated. Instead, All edges in the set are added to the correction set. In the middle; otherwise, generate a superedge. and put The redundant edges introduced in the process are added to the correction set. middle;

[0055] For the self-loop edge of supernode A The formula for judgment is Then no self-loop edges will be generated. Instead, All edges in the set are added to the correction set. Otherwise, generate a loop edge. and put The redundant edges introduced in the process are added to the correction set. middle.

[0056] On multiple datasets, compared with the SWeG algorithm (unmodified graph summarization algorithm), this embodiment achieves an average execution time speedup of 3.53 times and an average data compression improvement of 0.53%.

Claims

1. A lossless graph data compression method based on an improved graph summarization algorithm, comprising a node grouping stage, a node merging stage, and an edge structure encoding stage, characterized in that, Also includes: The computational load and execution time of the node merging stage are obtained according to the number of nodes in different groups, and the maximum acceptable number of nodes in the initial group is determined according to the preset execution time threshold. After grouping the nodes, select groups with more nodes than the maximum number of nodes in the selected group, and determine the node number threshold for each group based on the following formula. : ; in, It is a group The number of nodes, It is the group number. It is a predefined number of group nodes. Indicates the number of group nodes. Execution time required for the merge phase; Based on the node number threshold Divide the groups into new groups; Between multiple iterations of the node grouping and node merging phases, a Bloom filter is used to screen for mergeable nodes within a group, and nodes that meet the merging benefit criteria are merged, as follows: Randomly select node A within the group, and then select the optimal candidate node B from the remaining nodes based on an approximate index. Use a Bloom filter to determine whether node A and node B are among the top nodes. The round of iterations has been calculated; If yes, then remove node A from the group and randomly select node A within the group again for judgment, until the Bloom filter judgment result is no. At this time, calculate the merge benefit of node A and B and merge the nodes that meet the merge benefit conditions. For nodes that do not meet the conditions for merging benefits, continue to determine whether the merging benefits of nodes A and B are less than the conditions for merging in the next round. If it is not less than, then remove node A from the group and randomly select node A from the group again for judgment; otherwise, calculate to satisfy the condition. The smallest Value, will the node pair and Insert the first A Bloom filter for the layer is used in the node merging phase of the next iteration; in, The combined revenue of node A and node B, , Each is before and after The threshold for round iterations and when going forward The threshold for round iteration.

2. The method according to claim 1, characterized in that, The groups were divided by multiple samplings. Perform multiple samplings, each sampling Each node generates A new group.

3. The method according to claim 1, characterized in that, The node with the largest Jacobian approximation index value among the remaining nodes in the group is selected as the optimal candidate node B.

4. The method according to claim 1, characterized in that, A unique value is calculated for nodes A and B through data transformation and used as a unique identifier for the node pair. This unique identifier is then used as the input to the Bloom filter to determine whether the node pair has been calculated.

5. The method according to claim 4, characterized in that, Convert nodes A and B to binary values ​​and then concatenate them in order to form a new value. , serving as a unique identifier for node pairs.

6. The method according to claim 1, characterized in that, In the node grouping phase, nodes are initially grouped based on whether their shingle values ​​are equal.

7. The method according to claim 1, characterized in that, After the node grouping and node merging phases are iterated a specified number of times, the edge structure encoding phase begins. Super edges are generated between super nodes to form a summary graph and a corrected edge set. The corrected edge set is used to supplement the missing edge information in the summary graph and correct the redundant edge information introduced into the summary graph.

Images