Data compression method and device, equipment and storage medium

Abstract

The embodiment of the invention discloses a data compression method and a related device. The method provided by the embodiment of the invention comprises the following steps of: traversing each data unit in to-be-compressed data under each group, and calculating the similarity among the data units; determining a dependency relationship among the data units according to the similarity among the data units; generating a dependency relationship tree according to the dependency relationship among the data units; generating a sorting result of each data unit for the dependency relationship tree by using a graph traversal algorithm; and based on the sorting result of each data unit, determining the sorting sequence of each data unit in each group, and performing data compression on the to-be-compressed data in each group. According to the method, the similarity between the data units is calculated and the dependency tree is generated, so that the potential relevance between the data can be mined; the arrangement sequence of the data units is optimized through the sorting result generated by the graph traversal algorithm for the dependency relationship tree, redundant information can be further compressed, and the data compression rate is increased.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of data processing, and in particular to a data compression method, apparatus, device and storage medium. Background Art

[0002] With the explosive growth of data volume, data compression technology has been widely used in databases, file systems, object storage and other fields. Through data compression, storage space usage can be significantly reduced, transmission costs can be reduced, and system performance can be improved.

[0003] Current data compression technology mainly analyzes data features based on traditional coding methods (such as dictionary compression, run-length coding, and Huffman coding) to achieve efficient compression. In addition, the same or similar data units can be grouped and compressed based on the content characteristics of the data units, thereby further improving the compression rate.

[0004] However, when dealing with complex data structures or highly repetitive data, existing data compression methods often fail to fully exploit the potential compressibility of data, resulting in limited compression rates. At the same time, when dealing with large-scale or multi-dimensional data, existing data compression methods may have problems with insufficient compression speed and decompression efficiency, making it difficult to meet the efficiency requirements in practical applications. Summary of the invention

[0005] Based on the above problems, the embodiments of the present application provide a data compression method, apparatus, device and storage medium, the purpose of which is to improve the data compression rate, speed up the compression and decompression process, and improve the overall compression efficiency.

[0006] In a first aspect, an embodiment of the present application provides a data compression method, comprising:

[0007] By traversing each data unit in the data to be compressed under each group, calculating the similarity between each data unit;

[0008] Determining the dependency relationship between the data units according to the similarities between the data units;

[0009] Generate a dependency tree according to the dependency relationships between the data units;

[0010] Generating a sorting result of each of the data units using a graph traversal algorithm on the dependency tree;

[0011] Based on the sorting results of the data units, the arrangement order of the data units in each group is determined, and data compression is performed on the data to be compressed in each group.

[0012] In one embodiment, before traversing the similarities of the data units in the data to be compressed of each cluster to determine the dependency relationship between the data units, the method further includes:

[0013] The target data set is grouped according to a preset grouping type to obtain data to be compressed under at least one group; wherein the grouping types include: grouping by page, grouping by data container, and grouping by similarity between data units.

[0014] In one embodiment, generating a dependency tree according to the dependency relationships between the data units includes:

[0015] According to the dependency relationship between each of the data units, generating a directed graph corresponding to the dependency relationship;

[0016] If there is a circular reference relationship in the directed graph, the clipping position of the circular reference relationship is determined according to the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1;

[0017] Performing a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph;

[0018] Convert the directed acyclic graph into a dependency tree.

[0019] In one embodiment, determining the clipping position of the circular reference relationship according to the path length from each node to each fork node in the circular reference relationship includes:

[0020] According to the path length from each node to each fork node in the circular reference relationship, select a target path from a plurality of candidate paths formed from each node to each fork node;

[0021] The starting node and the ending node of the target path are determined, and the directed edge from the ending node to the starting node in the circular reference relationship is determined as the clipping position of the circular reference relationship.

[0022] In one embodiment, selecting a target path from a plurality of candidate paths consisting of each node to each of the forked nodes according to the path length from each node to each forked node in the circular reference relationship includes:

[0023] According to the path length from each node to each fork node in the circular reference relationship, determine a candidate path with the longest path length from multiple candidate paths formed from each node to each fork node;

[0024] If there are multiple candidate paths with the longest path length, the target path is selected according to the out-degree of the starting node in each candidate path with the longest path length.

[0025] In one embodiment, the graph traversal algorithm includes a depth-first traversal algorithm and a breadth-first traversal algorithm.

[0026] In one embodiment, determining the arrangement order of each of the data units in each of the groups based on the sorting results of each of the data units includes:

[0027] For the data to be compressed in each group, calculate a first data compression rate of each data unit of the data to be compressed before the arrangement order is adjusted based on the sorting result and a second data compression rate of each data unit of the data to be compressed after the arrangement order is adjusted based on the sorting result;

[0028] If the second data compression rate is higher than the first data compression rate, adjusting the arrangement order of each of the data units based on the sorting result;

[0029] If the second data compression rate is lower than the first data compression rate, the data order of each of the data units is not adjusted.

[0030] In a second aspect, an embodiment of the present application further provides a data compression device, including:

[0031] A calculation unit, used for calculating the similarity between each data unit by traversing each data unit in the data to be compressed under each group;

[0032] A determination unit, configured to determine the dependency relationship between the data units according to the similarities between the data units;

[0033] A generating unit, configured to generate a dependency tree according to the dependency relationships between the data units;

[0034] The generating unit is further used to generate a sorting result of each of the data units using a graph traversal algorithm on the dependency tree;

[0035] The determining unit is further configured to determine the arrangement order of the data units in each of the groups based on the sorting results of the data units, and to perform data compression on the data to be compressed in each group.

[0036] In a third aspect, an embodiment of the present application further provides a computer device, including:

[0037] CPU, memory, input and output interfaces;

[0038] The memory is a short-term storage memory or a persistent storage memory;

[0039] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform any one of the above-mentioned data compression methods.

[0040] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is run on a computer, the computer executes any one of the above-mentioned data compression methods.

[0041] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0042] The embodiment of the present application can effectively mine the potential correlation between data and reduce the dispersion of duplicate data by calculating the similarity between data units and generating a dependency tree. Based on the dependency tree, a graph traversal algorithm is used to generate the sorting results of data units, and the arrangement order between data units is optimized, so that redundant information in the data can be further compressed. The optimized arrangement enables the compression algorithm to process more continuous duplicate data at one time, reduce repeated calculations, and significantly improve the compression speed and compression rate.

[0043] At the same time, the embodiment of the present application only adjusts the arrangement order of the data units within the group, without changing the number of groups, the grouping standard, and the dependencies between groups. Therefore, the interference to the existing system is minimal, and various group compression technologies can be quickly compatible and adapted. In addition, the optimized data arrangement makes the reference relationship between data units clearer during the decompression stage, thereby further improving the decompression efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0045] Figure 1 A schematic diagram of data unit sorting provided in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a data compression method flow chart provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of data unit reordering provided in an embodiment of the present application;

[0048] Figure 4 A schematic diagram of the structure of a data compression device provided in an embodiment of the present application;

[0049] Figure 5 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0051] With the explosive growth of data volume, data compression technology has been widely used in databases, file systems, object storage and other fields. Through data compression, storage space usage can be significantly reduced, transmission costs can be reduced, and system performance can be improved.

[0052] Current data compression technology mainly analyzes data features based on traditional coding methods (such as dictionary compression, run-length coding, and Huffman coding) to achieve efficient compression. In addition, the same or similar data units can be grouped and compressed based on the content characteristics of the data units, thereby further improving the compression rate.

[0053] However, when dealing with complex data structures or highly repetitive data, existing data compression methods often fail to fully exploit the potential compressibility of data, resulting in limited compression rates. At the same time, when dealing with large-scale or multi-dimensional data, existing data compression methods may have problems with insufficient compression speed and decompression efficiency, making it difficult to meet the efficiency requirements in practical applications.

[0054] Based on this, in various embodiments of the present application, a dependency tree is generated by calculating the similarity between data units, and a graph traversal algorithm is used to optimize the arrangement order of data units, and to explore the potential correlation and duplication between data units, thereby improving data compression rate and compression efficiency.

[0055] Before further elaborating on the data compression method provided in the embodiments of the present application, the technical background that may be involved in the embodiments of the present application is first introduced.

[0056] During the packet compression process, the order of data units has a significant impact on the compression rate, which is mainly due to the characteristics of the compression algorithm. Most compression algorithms usually traverse the data one by one from front to back and eliminate duplicate parts during the traversal. Therefore, the arrangement of data will directly determine the amount of redundant information that the compression algorithm can identify and eliminate.

[0057] Please refer to Figure 1 , Figure 1A schematic diagram of a data unit sorting. Figure 1 As shown in the figure, there are three data blocks, some of which are repeated (the same color represents the repeated data part; purple represents the common overlapping part of the red data part and the blue data part, that is, the part that the two data segments are the same). Figure 1 In the upper part of the image, the data is arranged in a specific order. In this case, the compression algorithm can only identify and eliminate partially overlapping redundant data (i.e. Figure 1 The part covered by the double-headed arrow in the upper part is Figure 1 The blue + purple + red part on the far right of the upper half) can only compress about 1.5 data blocks of redundant content. It should be noted that "1.5 data blocks" does not refer to a fixed unit of measurement, but a visual representation of the data unit in the figure.

[0058] However, if the order of the data units is adjusted to Figure 1 As shown in the lower part, the red and blue data blocks can be completely identified as repeated parts, and a total of 2 data blocks can be compressed, which is much higher than Figure 1 The upper part can be compressed by 1.5 blocks, thus achieving a higher compression ratio. Figure 1 The optimized data sorting and compression effects obtained in the lower part can be achieved through the embodiments of the present application. The embodiments of the present application can enable the compression algorithm to fully utilize the potential repeatability of the data, thereby significantly improving the compression effect.

[0059] The various embodiments of the present application are described in further detail below in conjunction with the accompanying drawings.

[0060] The present application embodiment provides a data compression method, such as Figure 2 As shown, the method includes steps S201-S205.

[0061] S201: traverse each data unit in the data to be compressed in each group, and calculate the similarity between each data unit.

[0062] In the embodiments of the present application, a data unit refers to the minimum data granularity involved in the compression process, which may vary depending on the application scenario, and the embodiments of the present application are not limited here. For example, in a file system, a data unit may be a file, a file block, or a storage block (e.g., a 4KB storage block). In a database, a data unit may be a single record, a column of data, a row of data, or an index entry. In object storage, a data unit may be an object or a part of an object (e.g., a shard or a data block).

[0063] The method of calculating the similarity between each data unit also depends on the characteristics of the data unit and the application scenario. For example, similarity calculation can be achieved by calculating the Euclidean distance, cosine similarity, Pearson correlation coefficient, etc. between each data unit. The specific method of similarity calculation is not limited in the embodiment of the present application.

[0064] By calculating the similarity between each data unit, it is possible to discover the repeated or related parts between the data units in each group, providing a basis for subsequent dependency analysis and compression operations, thereby more efficiently removing duplicate data during compression and improving the compression rate.

[0065] S202: Determine the dependency relationship between the data units according to the similarity between the data units.

[0066] S203: Generate a dependency tree according to the dependency relationships between the various data units.

[0067] By calculating the similarity between data units, we analyze whether there is a certain correlation or dependency between the data units, and build a structure describing the dependency relationship based on this correlation, such as a dependency tree. The dependency tree is a special form of a directed acyclic graph (DAG). The nodes in the tree represent data units, and the edges represent the dependency relationships between data units. The dependency tree can reflect the relative positions, content associations, or reference relationships between data units.

[0068] Specifically, a similarity threshold (such as 0.8) can be set. When the similarity between two data units is greater than the threshold, it is considered that there is a dependency relationship between the two, and one of the data units is marked as dependent on the other data unit. Finally, the data units are connected through directed edges according to the similarity to form a dependency tree.

[0069] S204: Generate a sorting result of each data unit using a graph traversal algorithm on the dependency tree.

[0070] The graph traversal algorithm is used to access all nodes in the dependency tree in a certain order, thereby generating a sorted result of the data unit. Optionally, the graph traversal algorithm includes a depth-first traversal algorithm (DFS, Depth-First Search) and a breadth-first traversal algorithm (BFS, Breadth-First Search).

[0071] Please refer to Figure 3 , Figure 3 A schematic diagram of data unit reordering. Figure 3The upper part on the right is the arrangement order of each data unit in a certain group in the initial state (i.e., "Seg1-Seg2-Seg3-Seg4-Seg5-Seg6"). By calculating the similarity between each data unit to determine the dependency relationship between each data unit, we can get the following: Figure 3 The directed acyclic graph in the left half can be converted into a dependency tree. For example, using BFS on the dependency tree, we can get the sorting results of each data unit (such as Figure 3 The lower right side shows the arrangement order of “Seg6-Seg2-Seg4-Seg3-Seg1-Seg5”).

[0072] It is understandable that the ordering results of each data unit generated by the graph traversal algorithm for the dependency tree are not unique, because there are multiple nodes in a layer of the dependency tree, and the order between these nodes is usually not strictly dependent (that is, there is no fixed rule for their relative order). Therefore, the order of accessing these nodes may be different. For example, for Figure 3 Nodes Seg3, Seg1, and Seg5 at the same layer can be accessed in the order of Seg3→Seg1→Seg5, Seg1→Seg3→Seg5, or Seg5→Seg3→Seg1 based on the BFS algorithm. Figure 3 There are two child nodes Seg2 and Seg4 under the Seg6 node. Based on the DFS algorithm, Seg2 can be visited first, then the child nodes under Seg2, then Seg4 and then the child nodes of Seg4. Alternatively, DFS can also visit Seg4 first, then the child nodes under Seg4, then Seg2 and then the child nodes of Seg2. Of course, if there is a requirement for uniqueness of the sorting results, fixed rules can be added to the graph traversal algorithm (for example, sorting by node name lexicographic order or by specific weight priority) to ensure that the output result of the graph traversal algorithm is unique.

[0073] By using the graph traversal algorithm to generate sorting results for the dependency tree, the order of data units can be quickly determined, reducing the computational overhead of complex sorting algorithms. At the same time, this dependency-based grouping compression method can speed up the compression and decompression process and improve overall efficiency.

[0074] S105: Based on the sorting results of the data units, determine the arrangement order of the data units in each group, and perform data compression on the data to be compressed in each group.

[0075] By using the BFS or DFS algorithm to sort the data units in each group, an optimized arrangement order is generated and applied to the group. The optimized arrangement method can ensure that similar or repeated data units are arranged in close proximity, so that the subsequent compression algorithm can efficiently identify and eliminate redundant data. On this basis, the grouped data is further compressed using a data compression algorithm (such as dictionary compression, run-length encoding, or Lempel-Ziv 1977 / 1978 algorithm), which can improve the compression efficiency and compression rate.

[0076] Compared with the prior art which only focuses on the aggregation of similar data units, there is a lack of specific research on the sorting of each data unit within the group, which may lead to the problem of insufficient compression rate. The embodiment of the present application can effectively mine the potential correlation between data and reduce the dispersion of duplicate data by calculating the similarity between data units and generating a dependency tree. Based on the dependency tree, a graph traversal algorithm is used to generate the sorting results of data units, and the arrangement order between data units is optimized, so that the redundant information in the data can be further compressed, so that the compression algorithm can process more continuous duplicate data at one time, reduce repeated calculations, and significantly improve the compression speed and compression rate.

[0077] At the same time, the embodiment of the present application only adjusts the arrangement order of the data units within the group, without changing the number of groups, the grouping standard, and the dependencies between groups. Therefore, the interference to the existing compression system is minimal, and various group compression technologies can be quickly compatible and adapted. In addition, the optimized data arrangement makes the reference relationship between data units clearer during the decompression stage, thereby further improving the decompression efficiency.

[0078] In order to further improve the data compression effect, in one embodiment, before traversing the similarities of each data unit in the data to be compressed of each cluster and determining the dependency relationship between the data units, the method also includes: grouping the target data set according to a preset grouping type to obtain the data to be compressed under at least one group; wherein the grouping types include: grouping by page, grouping by data container, and grouping by the similarity between data units.

[0079] Before calculating the similarity of the data to be compressed in each cluster and determining the dependencies between data units, the target data set needs to be grouped. It can be understood that the purpose of grouping is to divide the data into a smaller and more easily processed set of units, thereby optimizing the subsequent similarity calculation and compression process.

[0080] Ways to group the target dataset include but are not limited to:

[0081] (1) Grouping by page:

[0082] The target data set is divided into groups according to page size or logical page. For example, in a storage system, each page may correspond to a storage unit of a certain size, which is suitable for paging management of stored data.

[0083] (2) Grouping by data container:

[0084] The target data set is grouped according to containers (such as tables, folders, and database partitions), which is suitable for data sets that have been pre-classified and organized according to logical structures. For example, in a database, different "tables" store employee information, order information, product information, etc., and each table can be regarded as an independent logical unit (container).

[0085] (3) Grouping based on the similarity between data units:

[0086] Data units are grouped according to their similarity, and similar or highly correlated data are grouped together. Since similar data is usually easier to compress quickly, it also facilitates subsequent data compression.

[0087] Through the above grouping method, the target data set can be divided into a smaller set of units (similar to clusters), and each group can be regarded as an independent small data set. After grouping, similarity calculation and dependency analysis only need to be performed within a smaller range, greatly reducing the computational complexity. Moreover, since the data units within the group are more likely to have high similarity or repetitiveness, the compression algorithm can more efficiently identify and eliminate redundant data when compressing the data, thereby improving the overall compression rate.

[0088] In order to obtain more accurate sorting results, in one embodiment, a dependency tree is generated according to the dependencies between the data units, including: according to the dependencies between the data units, a directed graph corresponding to the dependencies is generated; if there is a circular reference relationship in the directed graph, the clipping position of the circular reference relationship is determined according to the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1; a clipping operation is performed on the circular reference relationship at the clipping position to obtain a directed acyclic graph; and the directed acyclic graph is converted into a dependency tree.

[0089] In an embodiment of the present application, nodes and edges can be added one by one according to the dependencies between data units to construct a corresponding dependency directed graph. The nodes of the graph represent data units, and the edges represent the dependencies between nodes. If there are one or more loops (i.e., circular reference relationships) in the dependency directed graph, the dependency tree cannot be directly constructed. A loop is a special path. For example, in the path Seg1→Seg3→Seg2→...→Seg6→Seg1, starting from a node Seg1, passing through other nodes, it finally returns to the starting node Seg1, forming a closed loop.

[0090] To solve the circular reference problem, the following method can be used to determine the clipping position of the ring:

[0091] (1) Calculate the path length from each node in the ring to each fork node, and give priority to the ring with the longest path for pruning. A fork node is a node with an in-degree greater than 0 and an out-degree greater than 1, and the out-edge of the fork node is connected to the node outside the circular reference relationship (that is, the terminal node of the out-edge). It should be understood that in fact, the path length between each node in the ring and the terminal node of each outgoing path can also be calculated, and the ring with the longest path can be given priority for pruning. But in fact, the path length from each node in the ring to the terminal node is equal to the path length from the node to the fork node plus the path length from the fork node to the terminal node. For each node in the ring, the path length from the fork node to the terminal node is fixed. And the embodiment of the present application mainly focuses on the problem of eliminating circular reference relationships, so it is only necessary to pay attention to the path length from the node in the ring to the fork node to determine the pruning position of the circular reference relationship, thereby avoiding redundant calculations and effectively solving the circular reference problem.

[0092] (2) According to the weight of the dependency (such as similarity), the edges corresponding to the dependency with lower weight are removed first.

[0093] At the selected cropping position, the corresponding directed edges are deleted to break the ring structure and generate a directed acyclic graph DAG.

[0094] The steps to convert a directed acyclic graph into a dependency tree are as follows:

[0095] (1) Select a root node from the DAG. The root node is usually selected as a node with an in-degree of 0 and a non-zero out-degree.

[0096] (2) Starting from the root node, traverse the DAG layer by layer and build the tree structure in sequence according to the direction relationship of the edges.

[0097] (3) The child nodes of each node are determined by the edges in the DAG, and finally a dependency tree is generated.

[0098] The embodiments of the present application can flexibly determine the pruning position through path length or weight priority strategy when the dependency relationship is more complex or there are more rings, so as to retain important dependencies to the maximum extent and reduce the damage to the original data structure. The DAG and dependency tree generated after pruning the ring are easier to handle, which significantly reduces the computational complexity of subsequent algorithms (such as sorting, compression, etc.). At the same time, the reasonable selection of fork nodes avoids the accidental deletion of key node dependencies and improves the accuracy of dependency modeling.

[0099] In one embodiment, the clipping position of the circular reference relationship is determined according to the path length from each node to each fork node in the circular reference relationship, including: selecting a target path from multiple candidate paths consisting of each node to each fork node according to the path length from each node to each fork node in the circular reference relationship; determining the starting node and the ending node of the target path, and determining the directed edge from the ending node to the starting node in the circular reference relationship as the clipping position of the circular reference relationship.

[0100] For example, there is a directed relationship graph, which contains a circular reference relationship of B→C→D→E→F→B, and there is also an inbound path A→B and an outbound path E→G. In this directed relationship graph, node F is farthest from the fork node E (according to the unidirectional reference relationship of the node, the number of hops required is the largest), and its hop count is 4. Therefore, the directed edge E→F is chosen to be broken to eliminate the loop.

[0101] In one embodiment, a target path is selected from multiple candidate paths formed by each node to each fork node according to the path length from each node to each fork node in a circular reference relationship, including: determining a candidate path with the longest path length from multiple candidate paths formed by each node to each fork node according to the path length from each node to each fork node in the circular reference relationship; if there are multiple candidate paths with the longest path lengths, selecting the target path according to the out-degree of the starting node in each candidate path with the longest path length.

[0102] Exemplarily, there is a directed relationship graph, which contains a circular reference relationship of B→C→D→E→F→B, and also has an inbound path A→B, an outbound path E→G, and another outbound path D→I.

[0103] In this directed relationship graph, the path length of node F from the fork node E in the candidate path (F→B→C→D→E) is the farthest, and its hop count is 4. The distance length of node E from the fork node D in the candidate path (E→F→B→C→D) is the farthest, and its number of paths is also 4. Since the two paths have the same length, the out-degree of their starting nodes needs to be compared: the out-degree of node E is 2 (pointing to F and G). The out-degree of node F is 1 (pointing only to E). Select the candidate path (F→B→C→D→E) with the smaller out-degree of the starting node as the target path, and determine the directed edge E→F as the clipping position. Eliminate the loop by deleting the directed edge E→F.

[0104] When pruning circular reference relationships, the embodiment of the present application gives priority to selecting starting nodes with smaller out-degrees, which can minimize interference with other dependency relationships in the graph, avoid large-scale adjustments to the entire graph structure, ensure the locality of the pruning operation, and avoid damaging the integrity of global dependency relationships.

[0105] In one embodiment, based on the sorting results of each data unit, the arrangement order of each data unit in each group is determined, including: for the data to be compressed in each group, calculating the first data compression rate of each data unit of the data to be compressed before the arrangement order is adjusted based on the sorting result and the second data compression rate after the arrangement order is adjusted based on the sorting result; if the second data compression rate is higher than the first data compression rate, adjusting the arrangement order of each data unit based on the sorting result; if the second data compression rate is lower than the first data compression rate, not adjusting the data sorting of each data unit.

[0106] Exemplarily, assume that there is a group of data units D1, D2, D3, D4 to be compressed, and the data contents are as follows: D1 = "aaa", D2 = "abc", D3 = "aaa", D4 = "bbb".

[0107] The first step is to calculate the compression ratio of the first data in the current arrangement:

[0108] The original arrangement order is D1, D2, D3, D4. Assuming that run-length encoding (RLE) is used as the compression algorithm, repeated adjacent data will be compressed but the compression effect is limited. The original order compression result: ["aaa", "abc", "aaa", "bbb"]. Assuming the original data size: 16 bytes; the compressed data size: 12 bytes. The compression ratio before adjustment is calculated: (1-(12 / 16))×100%=25%.

[0109] The second step is to adjust the sorting order based on the sorting results:

[0110] After sorting, the arrangement is adjusted to D1, D3, D2, D4 (similar data are grouped together).

[0111] The compression result after adjustment is: ["aaa", "aaa", "abc", "bbb"]. "aaa" is aggregated, assuming the original data size is 16 bytes; the compressed data size is 8 bytes. The compression ratio before adjustment is calculated as: (1-(8 / 16))×100%=50%.

[0112] The third step is to compare the compression ratio:

[0113] The compression ratio before adjustment is 25%, and the compression ratio after adjustment is 50%.

[0114] Because the compression rate after adjustment is higher, the adjusted sorting D1, D3, D2, D4 is finally adopted.

[0115] The embodiment of the present application effectively avoids invalid sorting adjustments by comparing the compression rates before and after sorting, while achieving better data arrangement within the data grouping, maximizing compression efficiency and effect, and is suitable for scenarios with uneven data distribution (such as big data processing or complex file compression).

[0116] In order to implement the data compression method of the embodiment of the present application, the embodiment of the present application also provides a data compression device, such as Figure 4 As shown, the device comprises:

[0117] A calculation unit 401 is used to calculate the similarity between each data unit by traversing each data unit in the data to be compressed under each group;

[0118] A determination unit 402, configured to determine the dependency relationship between the data units according to the similarities between the data units;

[0119] A generating unit 403, configured to generate a dependency tree according to the dependency relationships between the data units;

[0120] The generating unit 403 is further configured to generate a sorting result of each of the data units using a graph traversal algorithm on the dependency tree;

[0121] The determining unit 402 is further configured to determine the arrangement order of the data units in each of the groups based on the sorting results of the data units, and perform data compression on the data to be compressed in each group.

[0122] In one embodiment, the device also includes: a processing unit; the processing unit is used to group the target data set according to a preset grouping type to obtain the data to be compressed under at least one group; wherein the grouping type includes: grouping by page, grouping by data container and grouping by similarity between data units.

[0123] In one embodiment, the generating unit 403 is further configured to generate a directed graph corresponding to the dependency relationship according to the dependency relationship between the data units;

[0124] The determining unit 402 is further configured to determine a pruning position of a circular reference relationship according to a path length from each node to each fork node in the circular reference relationship if there is a circular reference relationship in the directed graph; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1;

[0125] The processing unit is further used to perform a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph;

[0126] The processing unit is further used to convert the directed acyclic graph into a dependency tree.

[0127] In one embodiment, the processing unit is further configured to select a target path from a plurality of candidate paths formed from each node to each fork node according to the path length from each node to each fork node in the circular reference relationship;

[0128] The determining unit 402 is further configured to determine a start node and an end node of the target path, and determine a directed edge from the end node to the start node in the circular reference relationship as a clipping position of the circular reference relationship.

[0129] In one embodiment, the determining unit 402 is further configured to determine a candidate path with the longest path length from a plurality of candidate paths formed from each of the nodes to each of the forked nodes according to the path length from each of the nodes to each of the forked nodes in the circular reference relationship;

[0130] The processing unit is further configured to select a target path according to the out-degree of a starting node in each of the candidate paths with the longest path length if there are multiple candidate paths with the longest path length.

[0131] In one embodiment, the graph traversal algorithm includes a depth-first traversal algorithm and a breadth-first traversal algorithm.

[0132] In one embodiment, the calculation unit 401 is further used to calculate, for the data to be compressed in each group, a first data compression rate of each data unit of the data to be compressed before the arrangement order is adjusted based on the sorting result and a second data compression rate of each data unit of the data to be compressed after the arrangement order is adjusted based on the sorting result;

[0133] The processing unit is further configured to adjust the arrangement order of each of the data units based on the sorting result if the second data compression rate is higher than the first data compression rate;

[0134] The processing unit is further configured to not adjust the data order of each of the data units if the second data compression rate is lower than the first data compression rate.

[0135] In actual application, the processing unit can be implemented by a processor in a computer device in combination with a communication interface, and the calculation unit 401, the determination unit 402, and the generation unit 403 can be implemented by a communication interface in a data compression device.

[0136] It should be noted that: the above embodiment provides a data compression device, and only uses the division of the above program modules as an example to illustrate when performing data compression. In actual applications, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the above-described processing. In addition, the data compression device provided in the above embodiment and the data compression method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0137] Based on the hardware implementation of the above program modules, and in order to implement a data compression method provided in an embodiment of the present application, an embodiment of the present application further provides a computer device, such as Figure 5 As shown, the computer device 500 includes:

[0138] CPU 501, memory 502 and input / output interface 503;

[0139] The memory 502 is a temporary storage memory or a permanent storage memory;

[0140] The central processor 501 is configured to communicate with the memory 502 and execute instructions in the memory 502 to perform any one of the above data compression methods.

[0141] Of course, in actual application, the various components in the computer device 500 are coupled together through the bus system 504. It can be understood that the bus system 504 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 504 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 504 .

[0142] The memory 502 in the embodiment of the present application is used to store various types of data to support the operation of the computer device 500. Examples of such data include: any computer program used to operate on the computer device 500.

[0143] It is understandable that when the processor in the computer device described above executes the computer program, it can also implement the functions of the various units in the above-mentioned corresponding device embodiments, which will not be repeated here. Exemplarily, the computer program can be divided into one or more modules / units, one or more modules / units are stored in the memory, and executed by the processor to complete the various embodiments of the present application. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the computer device. For example, the computer program can be divided into the various units in the above-mentioned computer device, and each unit can implement the specific functions described in the above-mentioned corresponding computer device.

[0144] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The computer device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the processor and the memory are merely examples of computer devices and do not constitute a limitation on the computer device. The computer device may include more or fewer components, or a combination of certain components, or different components. For example, the computer device may also include input and output devices, network access devices, buses, etc.

[0145] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor, etc. The processor is the control center of the computer device, and uses various interfaces and lines to connect various parts of the entire computer device.

[0146] The memory can be used to store computer programs and / or modules. The processor implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0147] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed on a computer, the computer executes any one of the above-mentioned data compression methods.

[0148] An embodiment of the present application also provides a computer program product having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, it is used to implement the data compression method described in the first aspect of the embodiment of the present application or any specific implementation method of the first aspect.

[0149] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0150] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0151] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0152] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0153] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A data compression method, characterized in that: include: By traversing each data unit in the data to be compressed under each group, calculating the similarity between each data unit; Determining the dependency relationship between the data units according to the similarities between the data units; Generate a dependency tree according to the dependency relationships between the data units; Generating a sorting result of each of the data units using a graph traversal algorithm on the dependency tree; Based on the sorting results of the data units, the arrangement order of the data units in each group is determined, and data compression is performed on the data to be compressed in each group.

2. The data compression method according to claim 1, characterized in that: Before traversing the similarities of the data units in the data to be compressed of each cluster and determining the dependency relationship between the data units, the method further includes: The target data set is grouped according to a preset grouping type to obtain data to be compressed under at least one group; wherein the grouping types include: grouping by page, grouping by data container, and grouping by similarity between data units.

3. The data compression method according to claim 1, characterized in that: The step of generating a dependency tree according to the dependency relationships between the data units includes: According to the dependency relationship between each of the data units, generating a directed graph corresponding to the dependency relationship; If there is a circular reference relationship in the directed graph, the clipping position of the circular reference relationship is determined according to the path length from each node to each fork node in the circular reference relationship; wherein the fork node is a node with an in-degree greater than 0 and an out-degree greater than 1; Performing a clipping operation on the circular reference relationship at the clipping position to obtain a directed acyclic graph; Convert the directed acyclic graph into a dependency tree.

4. The data compression method according to claim 3, characterized in that: The step of determining the clipping position of the circular reference relationship according to the path length from each node to each fork node in the circular reference relationship includes: According to the path length from each node to each fork node in the circular reference relationship, select a target path from a plurality of candidate paths formed from each node to each fork node; The starting node and the ending node of the target path are determined, and the directed edge from the ending node to the starting node in the circular reference relationship is determined as the clipping position of the circular reference relationship.

5. The data compression method according to claim 4, characterized in that: The step of selecting a target path from a plurality of candidate paths formed from each node to each of the forked nodes according to the path length from each node to each of the forked nodes in the circular reference relationship comprises: According to the path length from each node to each fork node in the circular reference relationship, determine a candidate path with the longest path length from multiple candidate paths formed from each node to each fork node; If there are multiple candidate paths with the longest path length, the target path is selected according to the out-degree of the starting node in each candidate path with the longest path length.

6. The data compression method according to claim 1, characterized in that: The graph traversal algorithm includes a depth-first traversal algorithm and a breadth-first traversal algorithm.

7. The data compression method according to claim 1, characterized in that: The step of determining the arrangement order of the data units in each of the groups based on the sorting results of the data units includes: For the data to be compressed in each group, calculate a first data compression rate of each data unit of the data to be compressed before the arrangement order is adjusted based on the sorting result and a second data compression rate of each data unit of the data to be compressed after the arrangement order is adjusted based on the sorting result; If the second data compression rate is higher than the first data compression rate, adjusting the arrangement order of each of the data units based on the sorting result; If the second data compression rate is lower than the first data compression rate, the data order of each of the data units is not adjusted.

8. A data compression device, characterized in that: include: A calculation unit, used for calculating the similarity between each data unit by traversing each data unit in the data to be compressed under each group; A determination unit, configured to determine the dependency relationship between the data units according to the similarities between the data units; A generating unit, configured to generate a dependency tree according to the dependency relationships between the data units; The generating unit is further used to generate a sorting result of each of the data units using a graph traversal algorithm on the dependency tree; The determining unit is further configured to determine the arrangement order of the data units in each of the groups based on the sorting results of the data units, and to perform data compression on the data to be compressed in each group.

9. A computer device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the data compression method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is run on a computer, the data compression method according to any one of claims 1 to 7 is executed.

Images