Data compression method, device, equipment and storage medium

Abstract

The embodiment of the present application discloses a data compression method and related devices. The method of the embodiment of the present application includes: calculating the similarity between each data unit by traversing each data unit in the data to be compressed under each group; determining the dependency relationship between each data unit based on the similarity between each data unit; generating a dependency tree based on the dependency relationship between each data unit; generating the sorting results of each data unit for the dependency tree using a graph traversal algorithm; determining the arrangement order of each data unit in each group based on the sorting results of each data unit, and compressing the data to be compressed under each group. The present application can mine the potential correlation between data by calculating the similarity between data units and generating a dependency tree; optimizing the arrangement order of data units by using the sorting results generated by the graph traversal algorithm for the dependency tree, which can further compress redundant information and improve the data compression rate.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to data compression methods, devices, equipment, and storage media. Background Art

[0002] With the explosive growth of data volumes, data compression technology has been widely used in databases, file systems, object storage, and other fields. Data compression can significantly reduce storage space usage, lower transmission costs, and improve system performance.

[0003] Current data compression technologies primarily rely on traditional encoding methods (such as dictionary compression, run-length encoding, and Huffman coding) to analyze data characteristics and achieve efficient compression. Furthermore, compression can be further improved by grouping identical or similar data units based on their content characteristics.

[0004] However, existing data compression methods often fail to fully exploit the data's potential compressibility when dealing with complex data structures or highly repetitive data, resulting in limited compression rates. Furthermore, when processing large-scale or multi-dimensional data, existing data compression methods may suffer from insufficient compression speed and decompression efficiency, making them unable to meet the efficiency requirements of practical applications. Summary of the Invention

[0005] Based on the above problems, the embodiments of the present application provide data compression methods, devices, equipment and storage media, the purpose of which is to improve the data compression rate, speed up the compression and decompression process, and improve the overall compression efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data compression method, comprising:

[0007] By traversing each data unit in the data to be compressed under each group, calculating the similarity between each data unit;

[0008] Determining the dependency relationship between the data units based on the similarities between the data units;

[0009] Generate a dependency tree based on the dependency relationships between the data units;

[0010] Generating a sorting result of each of the data units using a graph traversal algorithm on the dependency tree;

[0011] Based on the sorting results of the data units, the arrangement order of the data units in each group is determined, and data compression is performed on the data to be compressed in each group.

[0012] In one embodiment, before traversing the similarities of the data units in the data to be compressed in each cluster and determining the dependency relationship between the data units, the method further includes:

[0013] The target data set is grouped according to a preset grouping type to obtain data to be compressed under at least one group; wherein the grouping types include: grouping by page, grouping by data container, and grouping by similarity between data units.

[0014] In one embodiment, generating a dependency tree according to the dependency relationships between the data units includes:

[0015] generating a directed graph corresponding to the dependency relationship according to the dependency relationship between the data units;

[0016] If a circular reference relationship exists in the directed graph, determining the pruning position of the circular reference relationship according to the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1;

[0017] Performing a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph;

[0018] Convert the directed acyclic graph into a dependency tree.

[0019] In one embodiment, determining the trimming position of the circular reference relationship based on the path length from each node to each fork node in the circular reference relationship includes:

[0020] Selecting a target path from a plurality of candidate paths formed by each node to each fork node according to the path length from each node to each fork node in the circular reference relationship;

[0021] The starting node and the ending node of the target path are determined, and the directed edge from the ending node to the starting node in the circular reference relationship is determined as the clipping position of the circular reference relationship.

[0022] In one embodiment, selecting a target path from a plurality of candidate paths from each node to each fork node according to the path length from each node to each fork node in the circular reference relationship includes:

[0023] According to the path length from each node to each fork node in the circular reference relationship, determining a candidate path with the longest path length from multiple candidate paths formed from each node to each fork node;

[0024] If there are multiple candidate paths with the longest path length, the target path is selected according to the out-degree of the starting node in each candidate path with the longest path length.

[0025] In one embodiment, the graph traversal algorithm includes a depth-first traversal algorithm and a breadth-first traversal algorithm.

[0026] In one embodiment, determining the arrangement order of the data units in each group based on the sorting results of the data units includes:

[0027] For the data to be compressed in each group, calculating a first data compression ratio of each data unit of the data to be compressed before the order of arrangement is adjusted based on the sorting result and a second data compression ratio of each data unit of the data to be compressed after the order of arrangement is adjusted based on the sorting result;

[0028] If the second data compression rate is higher than the first data compression rate, adjusting the arrangement order of each of the data units based on the sorting result;

[0029] If the second data compression rate is lower than the first data compression rate, the data order of each of the data units is not adjusted.

[0030] In a second aspect, an embodiment of the present application further provides a data compression device, comprising:

[0031] A calculation unit, configured to calculate similarities between the data units by traversing the data units in the data to be compressed under each group;

[0032] a determining unit, configured to determine a dependency relationship between the data units based on similarities between the data units;

[0033] A generating unit, configured to generate a dependency tree according to the dependency relationships between the data units;

[0034] The generating unit is further configured to generate a sorting result of each of the data units using a graph traversal algorithm on the dependency tree;

[0035] The determining unit is further configured to determine an arrangement order of the data units in each of the groups based on the sorting results of the data units, and perform data compression on the data to be compressed in each group.

[0036] In a third aspect, an embodiment of the present application further provides a computer device, including:

[0037] CPU, memory, input and output interfaces;

[0038] The memory is a short-term storage memory or a persistent storage memory;

[0039] The central processing unit is configured to communicate with the memory and execute instructions in the memory to perform any one of the above-mentioned data compression methods.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on a computer, the computer executes any one of the above-mentioned data compression methods.

[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0042] By calculating the similarity between data units and generating a dependency tree, the embodiments of the present application can effectively explore the potential correlations between data and reduce the dispersion of duplicate data. Based on the dependency tree, a graph traversal algorithm is used to generate the sorting results of the data units, optimizing the arrangement order between data units, thereby further compressing the redundant information in the data. The optimized arrangement enables the compression algorithm to process more continuous duplicate data at a time, reducing repeated calculations and significantly improving the compression speed and compression ratio.

[0043] Furthermore, the embodiments of the present application only adjust the order of data units within a group, without changing the number of groups, the grouping criteria, or the dependencies between groups. Therefore, disruption to existing systems is minimal, enabling rapid compatibility and adaptation to various group compression technologies. Furthermore, the optimized data arrangement makes the reference relationships between data units clearer during the decompression phase, further improving decompression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0045] Figure 1 A schematic diagram of data unit sorting provided in an embodiment of the present application;

[0046] Figure 2 A flowchart of a data compression method provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of data unit reordering provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a data compression device provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the computer device structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0051] With the explosive growth of data volumes, data compression technology has been widely used in databases, file systems, object storage, and other fields. Data compression can significantly reduce storage space usage, lower transmission costs, and improve system performance.

[0052] Current data compression technologies primarily rely on traditional encoding methods (such as dictionary compression, run-length encoding, and Huffman coding) to analyze data characteristics and achieve efficient compression. Furthermore, compression can be further improved by grouping identical or similar data units based on their content characteristics.

[0053] However, existing data compression methods often fail to fully exploit the data's potential compressibility when dealing with complex data structures or highly repetitive data, resulting in limited compression rates. Furthermore, when processing large-scale or multi-dimensional data, existing data compression methods may suffer from insufficient compression speed and decompression efficiency, making them unable to meet the efficiency requirements of practical applications.

[0054] Based on this, in each embodiment of the present application, a dependency tree is generated by calculating the similarity between data units, and the arrangement order of data units is optimized using a graph traversal algorithm, and the potential correlation and duplication between data units are explored, thereby improving data compression rate and compression efficiency.

[0055] Before further elaborating on the data compression method provided in the embodiments of the present application, the technical background that may be involved in the embodiments of the present application is first introduced.

[0056] During packet compression, the order of data units significantly impacts the compression ratio, primarily due to the nature of the compression algorithm. Most compression algorithms typically traverse the data from front to back, eliminating duplicates as they go. Therefore, the arrangement of the data directly determines the amount of redundant information the compression algorithm can identify and eliminate.

[0057] Please refer to Figure 1 , Figure 1A schematic diagram of sorting data units. Figure 1 As shown in the figure, there are three data blocks, some of which are repeated (the same color represents the repeated data part; purple represents the common overlapping part of the red data part and the blue data part, that is, the part that the two data segments are the same). Figure 1 In the upper half of the data, the data are arranged in a specific order. In this case, the compression algorithm can only identify and eliminate partially overlapping redundant data (i.e. Figure 1 The part covered by the double-headed arrow in the upper part is Figure 1 The blue, purple, and red areas on the far right of the top half ultimately compresses only about 1.5 data blocks of redundant content. It's important to note that "1.5 data blocks" isn't a fixed unit of measurement, but rather a visual representation of the data unit in the diagram.

[0058] However, if the order of the data units is adjusted to Figure 1 As shown in the lower part, the red and blue data blocks can be completely identified as repeated parts, and a total of 2 data blocks can be compressed, which is much higher than Figure 1 The upper half can be compressed by 1.5 data blocks, thus achieving a higher compression ratio. Figure 1 The optimized data sorting and compression effects obtained in the lower half can be achieved through the embodiments of the present application. The embodiments of the present application can enable the compression algorithm to fully utilize the potential repeatability of the data, thereby significantly improving the compression effect.

[0059] The following is a further detailed description of the various embodiments of the present application in conjunction with the accompanying drawings.

[0060] The present application embodiment provides a data compression method, such as Figure 2 As shown, the method includes steps S201-S205.

[0061] S201: traverse each data unit in the data to be compressed under each group and calculate the similarity between each data unit.

[0062] In the embodiments of the present application, a data unit refers to the minimum data granularity involved in the compression process, which may vary depending on the application scenario and is not limited in the embodiments of the present application. For example, in a file system, a data unit may be a file, a file block, or a storage block (e.g., a 4KB storage block). In a database, a data unit may be a single record, a column of data, a row of data, or an index entry. In object storage, a data unit may be an object or a portion of an object (e.g., a shard or a data block).

[0063] The method for calculating the similarity between each data unit also depends on the characteristics of the data unit and the application scenario. For example, similarity calculation can be achieved by calculating the Euclidean distance, cosine similarity, Pearson correlation coefficient, etc. between each data unit. The specific method of similarity calculation is not limited in the embodiments of this application.

[0064] By calculating the similarity between each data unit, it is possible to discover the repeated or related parts between the data units in each group, providing a basis for subsequent dependency analysis and compression operations, thereby more efficiently removing duplicate data during compression and improving the compression rate.

[0065] S202: Determine the dependency relationship between the data units based on the similarity between the data units.

[0066] S203: Generate a dependency tree based on the dependency relationships between the various data units.

[0067] By calculating the similarity between data units, we analyze whether there are certain associations or dependencies between them. Based on these associations, we construct a structure describing the dependencies, such as a dependency tree. A dependency tree is a special form of a directed acyclic graph (DAG). Nodes in the tree represent data units, and edges represent dependencies between data units. The dependency tree can reflect the relative positions, content associations, or reference relationships between data units.

[0068] Specifically, a similarity threshold (e.g., 0.8) can be set. When the similarity between two data units exceeds the threshold, a dependency relationship is considered to exist between them, and one data unit is marked as dependent on the other. Finally, the data units are connected through directed edges based on the similarity, forming a dependency tree.

[0069] S204: Generate a sorting result of each data unit using a graph traversal algorithm on the dependency tree.

[0070] The graph traversal algorithm is used to access all nodes in the dependency tree in a certain order, thereby generating a sorted result of the data units. Optionally, the graph traversal algorithm includes a depth-first search algorithm (DFS) and a breadth-first search algorithm (BFS).

[0071] Please refer to Figure 3 , Figure 3 A schematic diagram of data unit reordering. Figure 3The upper part on the right is the order of the data units in a group in the initial state (i.e., "Seg1-Seg2-Seg3-Seg4-Seg5-Seg6"). By calculating the similarity between each data unit to determine the dependency relationship between the data units, we can get the following: Figure 3 The directed acyclic graph in the left half can be converted into a dependency tree. For example, using BFS on the dependency tree, we can get the sorting results of each data unit (such as Figure 3 The lower right side shows the arrangement order of "Seg6-Seg2-Seg4-Seg3-Seg1-Seg5").

[0072] It is understandable that the ordering results of each data unit generated by the graph traversal algorithm for the dependency tree are not unique, because there are multiple nodes in a certain layer of the dependency tree, and the order between these nodes is usually not strictly dependent (that is, there is no fixed rule for the relative order between them). Therefore, the order of accessing these nodes may be different. For example, for Figure 3 Nodes Seg3, Seg1, and Seg5 at the same layer can be accessed in the order of Seg3→Seg1→Seg5, Seg1→Seg3→Seg5, or Seg5→Seg3→Seg1 based on the BFS algorithm. Figure 3 In the example, node Seg6 has two child nodes, Seg2 and Seg4. Based on the DFS algorithm, we can first visit Seg2, then visit the child nodes under Seg2, then visit Seg4, and then visit the child nodes of Seg4. Alternatively, DFS can first visit Seg4, then visit the child nodes under Seg4, then visit Seg2, and then visit the child nodes of Seg2. Of course, if there is a requirement for uniqueness of the sorting results, fixed rules can be added to the graph traversal algorithm (for example, sorting by node name lexicographically or by specific weight priority) to ensure that the output of the graph traversal algorithm is unique.

[0073] By using a graph traversal algorithm to generate a sorted result for the dependency tree, the order of data units can be quickly determined, reducing the computational overhead of complex sorting algorithms. Furthermore, this dependency-based grouping compression method can speed up the compression and decompression process, improving overall efficiency.

[0074] S105: Based on the sorting results of the data units, determine the arrangement order of the data units in each group, and perform data compression on the data to be compressed in each group.

[0075] By sorting the data units within each group using the BFS or DFS algorithm, an optimized permutation order is generated and applied to the group. This optimized permutation ensures that similar or duplicate data units are arranged in close proximity, making it easier for subsequent compression algorithms to efficiently identify and eliminate redundant data. Further compression of the grouped data using a data compression algorithm (such as dictionary compression, run-length encoding, or the Lempel-Ziv 1977 / 1978 algorithm) can improve both efficiency and compression ratio.

[0076] Compared with the existing technology that only focuses on the aggregation of similar data units, there is a lack of specific research on the sorting of each data unit within the group, which may lead to the problem of insufficient compression rate. The embodiment of the present application calculates the similarity between data units and generates a dependency tree, which can effectively mine the potential correlation between data and reduce the dispersion of duplicate data. Based on the dependency tree, a graph traversal algorithm is used to generate the sorting results of data units, and the arrangement order between data units is optimized, so that redundant information in the data can be further compressed, so that the compression algorithm can process more continuous duplicate data at one time, reduce repeated calculations, and significantly improve the compression speed and compression rate.

[0077] Furthermore, the embodiments of the present application only adjust the order of data units within a group, without changing the number of groups, the grouping criteria, or the dependencies between groups. Therefore, interference with existing compression systems is minimal, enabling rapid compatibility and adaptation to various group compression technologies. Furthermore, the optimized data arrangement makes the reference relationships between data units clearer during the decompression phase, further improving decompression efficiency.

[0078] In order to further improve the data compression effect, in one embodiment, before traversing the similarity of each data unit in the data to be compressed in each cluster and determining the dependency relationship between the data units, the method also includes: grouping the target data set according to a preset grouping type to obtain the data to be compressed under at least one group; wherein the grouping types include: grouping by page, grouping by data container, and grouping by the similarity between data units.

[0079] Before calculating the similarity of the data to be compressed within each cluster and determining the dependencies between data units, the target dataset needs to be grouped. The purpose of grouping is to divide the data into smaller, more manageable units, thereby optimizing the subsequent similarity calculation and compression process.

[0080] Methods for grouping target datasets include but are not limited to:

[0081] (1) Group by page:

[0082] The target dataset is divided into groups according to page size or logical page. For example, in a storage system, each page may correspond to a storage unit of a certain size, which is suitable for paging management of stored data.

[0083] (2) Grouping by data container:

[0084] The target dataset is grouped by container (such as table, folder, or database partition). This approach is suitable for datasets that have already been pre-organized and categorized according to a logical structure. For example, in a database, different "tables" store employee information, order information, product information, and so on. Each table can be considered an independent logical unit (container).

[0085] (3) Grouping based on the similarity between data units:

[0086] Data units are grouped based on their similarity, placing similar or highly correlated data into the same group. Because similar data is often more easily compressed, this also facilitates subsequent data compression.

[0087] This grouping method divides the target dataset into smaller units (similar to clusters), with each group being considered a separate, smaller dataset. This grouping allows similarity calculations and dependency analysis to be performed within a smaller scope, significantly reducing computational complexity. Furthermore, because data units within a group are more likely to be highly similar or repetitive, compression algorithms can more efficiently identify and eliminate redundant data, thereby improving overall compression rates.

[0088] In order to obtain more accurate sorting results, in one embodiment, a dependency tree is generated based on the dependency relationships between the various data units, including: generating a directed graph corresponding to the dependency relationships based on the dependency relationships between the various data units; if there is a circular reference relationship in the directed graph, determining the clipping position of the circular reference relationship based on the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1; performing a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph; and converting the directed acyclic graph into a dependency tree.

[0089] In an embodiment of the present application, nodes and edges can be added one by one according to the dependency relationship between data units to construct a corresponding dependency directed graph. The nodes of the graph represent data units, and the edges represent the dependency relationship between nodes. If there are one or more loops (i.e., circular reference relationships) in the dependency directed graph, it will not be possible to directly construct a dependency tree. A loop is a special path. For example, in the path Seg1→Seg3→Seg2→...→Seg6→Seg1, starting from a node Seg1, passing through other nodes, and finally returning to the starting node Seg1, a closed loop is formed.

[0090] To resolve the circular reference problem, you can determine the clipping position of the ring by the following method:

[0091] (1) Calculate the path length from each node in the ring to each fork node, and give priority to the ring with the longest path for pruning. A fork node is a node with an in-degree greater than 0 and an out-degree greater than 1, and the outgoing edge of the fork node is connected to the node outside the circular reference relationship (that is, the terminal node of the outgoing edge). It should be understood that in fact, the path length between each node in the ring and the terminal node of each outgoing path can also be calculated, and the ring with the longest path can be given priority for pruning. However, in fact, the path length from each node in the ring to the terminal node is equal to the path length from the node to the fork node plus the path length from the fork node to the terminal node. For each node in the ring, the path length from the fork node to the terminal node is fixed. And the embodiment of the present application mainly focuses on the problem of eliminating circular reference relationships. Therefore, it is only necessary to pay attention to the path length from the node in the ring to the fork node to determine the pruning position of the circular reference relationship, thereby avoiding redundant calculations and effectively solving the circular reference problem.

[0092] (2) According to the weight of the dependency (such as similarity), the edges corresponding to the dependency with lower weight are removed first.

[0093] At the selected cropping position, the corresponding directed edges are deleted to break the ring structure and generate a directed acyclic graph (DAG).

[0094] The steps to convert a directed acyclic graph into a dependency tree are as follows:

[0095] (1) Select a root node from the DAG. The root node is usually selected as a node with an in-degree of 0 and a non-zero out-degree.

[0096] (2) Starting from the root node, traverse the DAG layer by layer and build the tree structure in sequence according to the direction relationship of the edges.

[0097] (3) The child nodes of each node are determined by the edges in the DAG, and a dependency tree is finally generated.

[0098] The embodiments of the present application can flexibly determine the pruning location through path length or weight priority strategies when the dependency relationship is more complex or there are many rings, thereby maximizing the retention of important dependency relationships and reducing damage to the original data structure. The DAG and dependency tree generated after pruning the ring are easier to process, significantly reducing the computational complexity of subsequent algorithms (such as sorting, compression, etc.). At the same time, the reasonable selection of fork nodes avoids the accidental deletion of key node dependencies and improves the accuracy of dependency modeling.

[0099] In one embodiment, a clipping position of a circular reference relationship is determined based on the path length from each node to each fork node in the circular reference relationship, including: selecting a target path from multiple candidate paths from each node to each fork node based on the path length from each node to each fork node in the circular reference relationship; determining the starting node and ending node of the target path, and determining the directed edge from the ending node to the starting node in the circular reference relationship as the clipping position of the circular reference relationship.

[0100] For example, consider a directed graph containing a circular reference relationship B→C→D→E→F→B, an incoming path A→B, and an outgoing path E→G. In this directed graph, node F is farthest from the fork node E (the node's unidirectional reference relationship requires the most hops), with a hop count of 4. Therefore, the directed edge E→F is broken to eliminate the loop.

[0101] In one embodiment, a target path is selected from multiple candidate paths formed by each node to each fork node based on the path length from each node to each fork node in a circular reference relationship, including: determining a candidate path with the longest path length from multiple candidate paths formed by each node to each fork node based on the path length from each node to each fork node in the circular reference relationship; if there are multiple candidate paths with the longest path lengths, selecting the target path based on the out-degree of the starting node in each candidate path with the longest path length.

[0102] For example, there is a directed relationship graph containing a circular reference relationship of B→C→D→E→F→B, an incoming path A→B, an outgoing path E→G, and another outgoing path D→I.

[0103] In this directed graph, the candidate path (F→B→C→D→E) has the longest path length from node F to the bifurcation node E, with a hop count of 4. The candidate path (E→F→B→C→D) has the longest path length from node E to the bifurcation node D, also with a hop count of 4. Since the two paths have the same length, we need to compare the out-degrees of their starting nodes: Node E has an out-degree of 2 (pointing to both F and G). Node F has an out-degree of 1 (pointing only to E). We select the candidate path with the smaller out-degree starting node (F→B→C→D→E) as the target path and determine the directed edge E→F as the pruning location. We eliminate the loop by deleting the directed edge E→F.

[0104] When pruning circular reference relationships, the embodiments of the present application give priority to selecting starting nodes with smaller out-degrees, which can minimize interference with other dependency relationships in the graph, avoid large-scale adjustments to the entire graph structure, ensure the locality of the pruning operation, and avoid damaging the integrity of global dependency relationships.

[0105] In one embodiment, based on the sorting results of each data unit, the arrangement order of each data unit in each group is determined, including: for the data to be compressed in each group, calculating the first data compression rate of each data unit of the data to be compressed before the arrangement order is adjusted based on the sorting result and the second data compression rate after the arrangement order is adjusted based on the sorting result; if the second data compression rate is higher than the first data compression rate, adjusting the arrangement order of each data unit based on the sorting result; if the second data compression rate is lower than the first data compression rate, not adjusting the data sorting of each data unit.

[0106] Exemplarily, assume that there is a group of data units D1, D2, D3, and D4 to be compressed, and the data contents are as follows: D1 = "aaa", D2 = "abc", D3 = "aaa", and D4 = "bbb".

[0107] The first step is to calculate the first data compression ratio of the current arrangement:

[0108] The original sequence is D1, D2, D3, D4. Assuming run-length encoding (RLE) is used as the compression algorithm, repeated adjacent data will be compressed, but the compression effect is limited. The original sequence is compressed to: ["aaa", "abc", "aaa", "bbb"]. Assuming the original data size is 16 bytes and the compressed data size is 12 bytes, the calculated compression ratio before adjustment is: (1 - (12 / 16)) × 100% = 25%.

[0109] The second step is to adjust the sort order based on the sort results:

[0110] After sorting, the arrangement is adjusted to D1, D3, D2, D4 (similar data are clustered together).

[0111] The compression result after adjustment is: ["aaa", "aaa", "abc", "bbb"]. "aaa" is aggregated. Assuming the original data size is 16 bytes and the compressed data size is 8 bytes, the compression ratio before adjustment is calculated as: (1-(8 / 16))×100%=50%.

[0112] The third step is to compare the compression ratio:

[0113] The compression ratio before adjustment is 25%, and the compression ratio after adjustment is 50%.

[0114] Because the compression rate after adjustment is higher, the adjusted sorting D1, D3, D2, D4 is finally adopted.

[0115] The embodiment of the present application effectively avoids invalid sorting adjustments by comparing the compression rates before and after sorting, while achieving better data arrangement within the data group, maximizing compression efficiency and effect, and is suitable for scenarios with uneven data distribution (such as big data processing or complex file compression).

[0116] In order to implement the data compression method of the embodiment of the present application, the embodiment of the present application also provides a data compression device, such as Figure 4 As shown, the device includes:

[0117] The calculation unit 401 is configured to calculate the similarity between each data unit by traversing each data unit in the data to be compressed under each group;

[0118] A determining unit 402 is configured to determine a dependency relationship between the data units based on similarities between the data units;

[0119] A generating unit 403 is configured to generate a dependency tree according to the dependency relationships between the data units;

[0120] The generating unit 403 is further configured to generate a sorting result of each of the data units using a graph traversal algorithm on the dependency tree;

[0121] The determining unit 402 is further configured to determine an arrangement order of the data units in each of the groups based on the sorting results of the data units, and perform data compression on the data to be compressed in each group.

[0122] In one embodiment, the device further includes: a processing unit; the processing unit is used to group the target data set according to a preset grouping type to obtain data to be compressed under at least one group; wherein the grouping type includes: grouping by page, grouping by data container, and grouping by similarity between data units.

[0123] In one embodiment, the generating unit 403 is further configured to generate a directed graph corresponding to the dependency relationship according to the dependency relationship between the data units;

[0124] The determining unit 402 is further configured to, if a circular reference relationship exists in the directed graph, determine a pruning position of the circular reference relationship based on a path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1;

[0125] The processing unit is further configured to perform a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph;

[0126] The processing unit is further configured to convert the directed acyclic graph into a dependency tree.

[0127] In one embodiment, the processing unit is further configured to select a target path from a plurality of candidate paths formed from each node to each fork node according to the path length from each node to each fork node in the circular reference relationship;

[0128] The determining unit 402 is further configured to determine a start node and an end node of the target path, and determine a directed edge from the end node to the start node in the circular reference relationship as a clipping position of the circular reference relationship.

[0129] In one embodiment, the determining unit 402 is further configured to determine, based on the path length from each node to each fork node in the circular reference relationship, a candidate path with the longest path length from multiple candidate paths formed from each node to each fork node;

[0130] The processing unit is further configured to select a target path according to the out-degree of a starting node in each of the candidate paths with the longest path length if there are multiple candidate paths with the longest path length.

[0131] In one embodiment, the graph traversal algorithm includes a depth-first traversal algorithm and a breadth-first traversal algorithm.

[0132] In one embodiment, the calculation unit 401 is further configured to calculate, for the data to be compressed in each group, a first data compression ratio of each data unit of the data to be compressed before the order of the data units to be compressed is adjusted based on the sorting result, and a second data compression ratio of each data unit of the data to be compressed after the order of the data units to be compressed is adjusted based on the sorting result.

[0133] The processing unit is further configured to adjust the arrangement order of each of the data units based on the sorting result if the second data compression rate is higher than the first data compression rate;

[0134] The processing unit is further configured to not adjust the data order of each of the data units if the second data compression rate is lower than the first data compression rate.

[0135] In actual application, the processing unit can be implemented by a processor in a computer device in combination with a communication interface, and the calculation unit 401, the determination unit 402, and the generation unit 403 can be implemented by a communication interface in a data compression device.

[0136] It should be noted that the above embodiments provide a data compression device, using the division of the aforementioned program modules as an example only. In actual applications, the aforementioned processing can be assigned to different program modules as needed, i.e., the internal structure of the device can be divided into different program modules to complete all or part of the aforementioned processing. Furthermore, the data compression device and the data compression method embodiments provided in the above embodiments share the same concept. The specific implementation process is detailed in the method embodiments and will not be further described here.

[0137] Based on the hardware implementation of the above program modules, and in order to implement a data compression method provided in an embodiment of the present application, an embodiment of the present application further provides a computer device, such as Figure 5 As shown, the computer device 500 includes:

[0138] CPU 501, memory 502 and input / output interface 503;

[0139] The memory 502 is a temporary storage memory or a permanent storage memory;

[0140] The central processing unit 501 is configured to communicate with the memory 502 and execute instructions in the memory 502 to perform any one of the above-mentioned data compression methods.

[0141] Of course, in actual application, the various components in the computer device 500 are coupled together through the bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0142] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the computer device 500. Examples of such data include: any computer program used to operate on the computer device 500.

[0143] It is understandable that when the processor in the computer device described above executes the computer program, it can also implement the functions of the various units in the corresponding device embodiments described above, which will not be repeated here. For example, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory and executed by the processor to complete the various embodiments of the present application. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device. For example, the computer program can be divided into the various units in the above-mentioned computer device, and each unit can implement the specific functions described in the above-mentioned corresponding computer device.

[0144] A computer device may be a desktop computer, laptop, PDA, cloud server, or other computing device. A computer device may include, but is not limited to, a processor and memory. Those skilled in the art will appreciate that processors and memory are merely examples of computer devices and do not constitute a limitation of computer devices. Computer devices may include more or fewer components, or combinations of certain components, or different components. For example, a computer device may also include input / output devices, network access devices, buses, and the like.

[0145] The processor can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the computer device and connects the various parts of the entire computer device using various interfaces and lines.

[0146] The memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card, at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0147] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is run on a computer, the computer is enabled to execute any one of the above-mentioned data compression methods.

[0148] An embodiment of the present application also provides a computer program product having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, it is used to implement the data compression method described in the first aspect of the embodiment of the present application or any specific implementation of the first aspect.

[0149] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0152] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data compression method, characterized in that: include: By traversing each data unit in the data to be compressed under each group, calculating the similarity between each data unit; Determining the dependency relationship between the data units based on the similarities between the data units; Generate a dependency tree based on the dependency relationships between the data units; Generating a sorting result of each of the data units using a graph traversal algorithm on the dependency tree; Based on the sorting results of the data units, determining the arrangement order of the data units in each group, and performing data compression on the data to be compressed in each group; Generating a dependency tree according to the dependency relationships between the data units includes: generating a directed graph corresponding to the dependency relationship according to the dependency relationship between the data units; If a circular reference relationship exists in the directed graph, determining the pruning position of the circular reference relationship according to the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1; Performing a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph; Converting the directed acyclic graph into a dependency tree; The determining, based on the sorting results of the data units, the arrangement order of the data units in each of the groups includes: For the data to be compressed in each group, calculating a first data compression ratio of each data unit of the data to be compressed before the order of arrangement is adjusted based on the sorting result and a second data compression ratio of each data unit of the data to be compressed after the order of arrangement is adjusted based on the sorting result; If the second data compression rate is higher than the first data compression rate, adjusting the arrangement order of each of the data units based on the sorting result; If the second data compression rate is lower than the first data compression rate, the data order of each of the data units is not adjusted.

2. The data compression method according to claim 1, wherein: Before traversing the similarities of the data units in the data to be compressed of each cluster and determining the dependency relationship between the data units, the method further includes: The target data set is grouped according to a preset grouping type to obtain data to be compressed under at least one group; wherein the grouping types include: grouping by page, grouping by data container, and grouping by similarity between data units.

3. The data compression method according to claim 1, wherein: The determining of the clipping position of the circular reference relationship according to the path length from each node to each fork node in the circular reference relationship includes: Selecting a target path from a plurality of candidate paths formed by each node to each fork node according to the path length from each node to each fork node in the circular reference relationship; The starting node and the ending node of the target path are determined, and the directed edge from the ending node to the starting node in the circular reference relationship is determined as the clipping position of the circular reference relationship.

4. The data compression method according to claim 3, wherein: The step of selecting a target path from a plurality of candidate paths from each node to each of the forked nodes according to the path length from each node to each of the forked nodes in the circular reference relationship includes: According to the path length from each node to each fork node in the circular reference relationship, determining a candidate path with the longest path length from multiple candidate paths formed from each node to each fork node; If there are multiple candidate paths with the longest path length, the target path is selected according to the out-degree of the starting node in each candidate path with the longest path length.

5. The data compression method according to claim 1, wherein: The graph traversal algorithm includes a depth-first traversal algorithm and a breadth-first traversal algorithm.

6. A data compression device, characterized in that: include: A calculation unit, configured to calculate similarities between the data units by traversing the data units in the data to be compressed under each group; a determining unit, configured to determine a dependency relationship between the data units based on similarities between the data units; A generating unit, configured to generate a dependency tree according to the dependency relationships between the data units; The generating unit is further configured to generate a sorting result of each of the data units using a graph traversal algorithm on the dependency tree; The determining unit is further configured to determine an arrangement order of the data units in each of the groups based on the sorting results of the data units, and perform data compression on the data to be compressed in each group; The generating unit is specifically configured to: generate a directed graph corresponding to the dependency relationship based on the dependency relationship between each of the data units; if a circular reference relationship exists in the directed graph, determine a clipping position of the circular reference relationship based on the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1; perform a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph; and convert the directed acyclic graph into a dependency tree; The determination unit is specifically used to: calculate, for the data to be compressed in each group, a first data compression rate of each data unit of the data to be compressed before the arrangement order is adjusted based on the sorting result, and a second data compression rate after the arrangement order is adjusted based on the sorting result; if the second data compression rate is higher than the first data compression rate, adjust the arrangement order of each data unit based on the sorting result; if the second data compression rate is lower than the first data compression rate, do not adjust the data sorting of each data unit.

7. A computer device, characterized in that: include: CPU, memory and input / output interfaces; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the data compression method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run on a computer, the data compression method according to any one of claims 1 to 5 is executed.

9. A computer program product having a computer program / instructions stored thereon, characterized in that: When the computer program / instruction is executed by a processor, it is used to implement the data compression method according to any one of claims 1 to 5.

Images