Data compression method and apparatus

Abstract

Provided in the present application are a data compression method and apparatus. The method comprises: performing block division on data included in a first file, and obtaining at least one first file block on the basis of each first file; performing deduplication on first file blocks having the same data; if a plurality of first file blocks are obtained after deduplication, performing similarity aggregation on the plurality of first file blocks, so as to obtain at least one second file block, wherein the similarity aggregation refers to aggregating, into a second file block on the basis of a hash value, at least one of the first file blocks which conforms to a preset rule; and compressing each second file block, so as to correspondingly obtain a first compression block. The compression ratio of data compression can be improved.

Description

Data compression method and device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 14, 2023, with application number 202310708752.2 and application name “Data Compression Method and Device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing, and in particular to a data compression method and device. Background Art

[0003] With the development of communication technology, data compression has been widely used in various fields, such as system file compression. In the actual application of system file compression, such as using Squashfs read-only images to store various component applications and script files, the read-only file system occupies a large amount of storage space, placing high storage requirements on the device, and some devices may have insufficient storage space.

[0004] How to further improve the compression ratio of data compression and store more data in a limited storage space has become an urgent problem to be solved.

[0005] Summary of the Invention

[0006] The present application provides a data compression method and device, which can further improve the compression ratio of data compression and store more data in a limited storage space.

[0007] In a first aspect, the present application provides a method for data compression, comprising: dividing the data contained in a first file into blocks, and obtaining at least one first file block based on each first file; deduplicating the first file blocks with the same data; if multiple first file blocks are obtained after deduplication, performing similarity aggregation on the multiple first file blocks to obtain at least one second file block, wherein the similarity aggregation refers to aggregating at least one first file block that meets preset rules into a second file block based on a hash value; and compressing each second file block to obtain a corresponding first compressed block.

[0008] If multiple first file blocks are obtained after deduplication, if the deduplicated first file blocks are directly compressed, the data to be compressed is large and the compression ratio is not high. Therefore, several first file blocks with high similarity can be aggregated by similarity to obtain a second file block. Since the multiple first file blocks aggregated in a second file block are highly similar, that is, the first file blocks aggregated into the same second file block include more identical data, based on this, reference can be made to methods such as compression dictionaries to take a value for the identical parts and compress them. In this way, more first file blocks can be aggregated and compressed in a second file block, thereby improving the compression ratio.

[0009] Optionally, each first file may be divided into first file blocks with the same data size, or each first file block may be divided into first file blocks with different data sizes.

[0010] In one possible implementation, the similarity aggregation includes determining the similarity of each of the first file blocks based on a preset algorithm, and performing aggregation based on the amount of data that each of the second file blocks can contain. For example, an algorithm based on a hash value is used to calculate all the first file blocks after deduplication to obtain the hash value of each first file block, and the first file blocks are sorted from high to low according to the similarity. The first file blocks aggregated in a second file block are determined based on the amount of data that a second file block can contain (i.e., the size of the second file block). The size of the second file block can be preset.

[0011] In one possible implementation, the similarity aggregation includes determining the similarity of each first file block based on the preset algorithm and performing aggregation based on a preset minimum threshold for similarity. For example, if a minimum threshold for similarity is preset, the first file blocks are sorted from highest to lowest until the similarity of the first file blocks is no less than the preset minimum threshold. All first file blocks with similarities exceeding the minimum threshold are then aggregated to obtain at least one second file block.

[0012] In one possible implementation, the similarity aggregation includes aggregation based on the output result of the preset algorithm. For example, a locality-sensitive hashing (LSH) algorithm can be adopted. Based on the characteristics of LSH, the first file blocks that can be output to the same bucket are file blocks with high similarity. Therefore, it can be considered that after calculation by the LSH algorithm, the similarity between the first file blocks output together is high, meets the preset rules, and can be similarly aggregated into the same second file block.

[0013] Through the above-mentioned different preset rules, after similarity aggregation, the probability that the data of each first file block in each second file block is the same is high. The same part is saved only once and pointed to each first file block, or the same part is replaced by encoding and pointed to each first file block before compression, which can greatly improve the compression ratio.

[0014] In one possible implementation, the data contained in the second file is divided into blocks according to a preset data length, and at least one third file block is obtained based on each second file, wherein the second file is a file to be compressed, and the data contained in the second file is used to perform the first operation; the second file blocks with the same data are deduplicated; and each of the deduplicated second file blocks is compressed to obtain a corresponding second compressed block.

[0015] In a possible implementation, the first file is a file to be compressed, and data contained in the first file is used to perform operations other than the first operation.

[0016] Based on some first operations, only file blocks that are segmented and compressed according to a preset data length can be identified. It is possible to first determine whether the data contained in the file to be compressed is used to perform the first operation, then determine the file to be compressed required for performing the first operation as the second file, and determine the file to be compressed for performing operations other than the first operation as the first file, and compress them in different ways. Dividing the file to be compressed into different blocks based on different operations and then compressing them is compatible with operations that recognize different compression block sizes, making network device compression more compatible.

[0017] In one possible implementation, the method further includes: storing the first compressed block in a mirror file; wherein the mirror file includes an extended super block, a data block, a bucket and a bucket table, the extended super block is used to store information about the data block and the bucket, each data block is used to store at least one first compressed block, the bucket is used to indicate the data block corresponding to each first compressed block, and the offset of each first compressed block in the data block, and the bucket table is used to store the correspondence between each first file and the bucket.

[0018] In one possible implementation, the first operation is a version upgrade of Squashfs. The version upgrade of Squashfs can only recognize Squashfs images, that is, it can only upgrade files that are divided into 128k blocks and address them according to the addressing method of the existing version. In this case, if the file blocks are deduplicated and similarity aggregated according to other block division rules, and the addressing is performed according to the addressing method corresponding to the second file block after similarity aggregation, it will not be recognized, and the old version will not be able to be upgraded to the new version, causing a generation gap and affecting the version upgrade of the network device. Therefore, compressing the files used for the version upgrade of Squashfs in a way that it can recognize can effectively prevent the occurrence of generation gaps.

[0019] In a second aspect, the present application provides a device, comprising: a first blocking module, used to block and divide the data contained in a first file, and obtain at least one first file block based on each first file; a first deduplication module, used to deduplicate all first file blocks with the same data; a similarity aggregation module, used to perform similarity aggregation on the multiple first file blocks if multiple first file blocks are obtained after deduplication by the deduplication module, to obtain at least one second file block, wherein the similarity aggregation refers to aggregating at least one first file block that meets preset rules into a second file block based on a hash value; and a first compression module, used to compress each second file block to obtain a corresponding first compressed block.

[0020] In one possible implementation, it also includes: a second blocking module, used to block the data contained in the second file according to a preset data length, and obtain at least one third file block based on each second file, wherein the second file is a file to be compressed, and the data contained in the second file is used to perform the first operation; a second deduplication module, used to deduplicate all the second file blocks with the same data; and a second compression module, used to compress each of the second file blocks after deduplication to obtain a corresponding second compressed block.

[0021] In a possible implementation, the first file is a file to be compressed, and data contained in the first file is used to perform operations other than the first operation.

[0022] In one possible implementation, the similarity aggregation includes determining the similarity of each first file block based on a preset algorithm, and performing aggregation based on the amount of data that each second file block can contain; or, the similarity aggregation includes determining the similarity of each first file block based on the preset algorithm, and performing aggregation based on a preset minimum threshold of similarity; or, the similarity aggregation includes performing aggregation based on the output result of the preset algorithm; wherein the preset algorithm is associated with the hash value.

[0023] In one possible implementation, a storage module is used to store the first compressed block in a mirror file; wherein the mirror file includes an extended super block, a data block, a bucket and a bucket table, the extended super block is used to store information about the data block and the bucket, each of the data blocks is used to store at least one first compressed block, the bucket is used to indicate the data block corresponding to each first compressed block, and the offset of each first compressed block in the data block, and the bucket table is used to store the correspondence between each first file and the bucket.

[0024] In a possible implementation, the first operation is a version upgrade of Squashfs.

[0025] In a third aspect, the present application provides a device comprising at least one processor, a transceiver, and a memory. The at least one processor is coupled to the transceiver, and when the at least one transceiver executes a program or instruction, the control device implements part or all of the operations of the first aspect and any possible implementation of the first aspect. The first device may be a terminal device or a network device, or a chip in the terminal device or the network device.

[0026] In a fourth aspect, the present application provides a device, comprising a communication port and a processor, wherein the communication port is used to perform the sending and receiving operations involved in the method described in any of the aforementioned aspects and any possible implementation of any of the aforementioned aspects, and the processor is used to perform other operations other than the sending and receiving operations involved in the method described in any of the aforementioned aspects and any possible implementation of any of the aforementioned aspects. For example, when the device is used as an apparatus to perform the method described in the first aspect, the processor is used to divide the data contained in the first file into blocks, and obtain at least one first file block based on each first file; deduplicate the first file blocks with the same data; if multiple first file blocks are obtained after deduplication, perform similarity aggregation on the multiple first file blocks to obtain at least one second file block, wherein the similarity aggregation refers to aggregating at least one first file block that meets the preset rules into a second file block based on a hash value; and compressing each second file block to obtain a corresponding first compressed block.

[0027] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a processor, it implements part or all of the operations included in the method described in any of the foregoing aspects and any possible implementation of any of the foregoing aspects.

[0028] In a sixth aspect, the present application provides a computer program product comprising instructions which, when executed on a processor, implement the method described in any of the foregoing aspects and part or all of the operations included in any possible implementation of any of the foregoing aspects.

[0029] In a seventh aspect, the present application provides a chip comprising: a port circuit and a processor. The port circuit is connected to the processor, and the processor is configured to cause the chip to perform some or all of the operations included in the method described in any of the aforementioned aspects and any possible implementation of any of the aforementioned aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] FIG1 is a flow chart of a data compression method provided in an embodiment of the present application;

[0032] FIG2 is a flow chart of another data compression method provided in an embodiment of the present application;

[0033] FIG3 is a schematic diagram of a field composition structure of a Squashfs image file provided in an embodiment of the present application;

[0034] FIG4 is a schematic diagram of a compression process provided by an embodiment of the present application;

[0035] FIG5 is a schematic structural diagram of a device provided in an embodiment of the present application;

[0036] FIG6 is a schematic structural diagram of another device provided in an embodiment of the present application;

[0037] FIG7 is a schematic structural diagram of another device provided in an embodiment of the present application;

[0038] FIG8 is a schematic structural diagram of a device 20 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to enable people in this technical field to better understand the solutions in this application, the technical solutions in the embodiments of this application will be clearly and completely described below in combination with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0040] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0041] In the description and claims of the embodiments of this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first target object" and "second target object" are used to distinguish different objects, rather than to describe a specific order of objects.

[0042] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0043] In the description of the embodiments of this application, unless otherwise specified, "multiple" means two or more. For example, "multiple processing units" means two or more processing units; "multiple systems" means two or more systems.

[0044] For ease of understanding, the following first explains the relevant nouns or terms used in the embodiments of this application:

[0045] 1. Squashfs

[0046] It is a GPL open source read-only compressed file system for use in the Linux kernel, which can compress files, nodes (inodes) and directories (directories) within the system.

[0047] 2. Image file

[0048] It is to make a specific series of files into a single file according to a certain format.

[0049] 3. Version Upgrade

[0050] Refers to software running on network devices that needs to be upgraded from an older version to the latest version.

[0051] 4. Nilsimsa

[0052] It is a locality-sensitive hashing algorithm for spam. A calculated value of 0 indicates dissimilarity and a value of 128 indicates very similarity.

[0053] The data compression method provided in the embodiments of the present application can be applied in different systems. The embodiments of the present application illustrate the data compression method in a read-only file system, such as Squashfs, and the device executing the compression method is a network device, but the present application is not limited to this. Figure 1 is a flow chart of a data compression method provided in the embodiments of the present application. As shown in Figure 1, the method includes but is not limited to steps S101 to S104.

[0054] S101. A network device divides data contained in a first file into blocks, and obtains at least one first file block based on each first file.

[0055] Assuming that there are multiple files to be compressed in a network device, they can all be defined as first files, or files to be compressed that perform certain operations can be defined as first files, or files to be compressed that do not perform the first operation can be defined as first files, etc. The first file can be defined based on the actual usage requirements of the files to be compressed. In one example, first file A, first file B, and first file C are all files to be compressed. The network device can divide the data contained in first file A, first file B, and first file C into blocks, respectively, to obtain first file blocks for each divided first file. For example, the network device obtains first file block 1 and first file block 2 from first file A, obtains first file block 3 from first file B, and obtains first file block 4, first file block 5, and first file block 6 from first file C.

[0056] Optionally, the network device may divide each first file into first file blocks with the same data size, or may divide each first file block into first file blocks with different data sizes.

[0057] S102: The network device deduplicates the first file blocks with the same data.

[0058] Data comparison is performed on the first file blocks obtained after segmentation, and first file blocks with identical data are deduplicated. Assuming, according to the example of S101, six first file blocks, namely first file block 1 to first file block 6, are obtained after segmentation, and after comparison, it is determined that the data in first file block 1 and first file block 3 are identical, either first file block 1 or first file block 3 is deleted, i.e., one of the first file blocks is retained. If three or more first file blocks have identical data, only one of them is retained.

[0059] Optionally, the data of the first file blocks being completely identical may mean that the sizes of the two first file blocks are the same, and the character strings included in the two first file blocks are completely identical.

[0060] In one example, if the data in the first file block 1 and the first file block 3 are exactly the same, then the first file block 3 is deleted. After deletion, the inode node description of the first file block 1 includes both the first file block 1 and the first file block 3. For example, the inode node description of the first file block 1 shows that both the first file A and the first file B point to the first file block 1, so that when decompression or other operations are required, the inode node description can be used to enable the first file A and the first file B to obtain the content of the first file block 1.

[0061] S103. If the network device obtains multiple first file blocks after deduplication, it performs similarity aggregation on the multiple first file blocks to obtain at least one second file block, wherein similarity aggregation refers to aggregating at least one first file block that meets a preset rule into one second file block based on a hash value.

[0062] Optionally, after deduplication, the network device still obtains multiple first file blocks. If the deduplicated first file blocks are directly compressed, the compression ratio is not high. Therefore, several first file blocks with high similarity can be aggregated together through similarity aggregation to obtain a second file block. Due to the high similarity, the several first file blocks aggregated in one second file block include more identical data. Based on this, reference can be made to methods such as compression dictionaries to take a value for the identical parts and compress them. In this way, more first file blocks can be aggregated and compressed in one second file block, thereby improving the compression ratio.

[0063] For example, after deduplication of the first file block 1, the first file block 2, the first file block 3, the first file block 4, the first file block 5 and the first file block 6, the remaining ones are the first file block 1, the first file block 2, the first file block 4, the first file block 5 and the first file block 6. The network device performs similarity aggregation on the first file block 1, the first file block 2, the first file block 4, the first file block 5 and the first file block 6.

[0064] In one example, the network device may determine similarity and the amount of data that a second file block can contain (such as the size of the second file block, which can be pre-set) based on a preset algorithm to perform similarity aggregation. For example, a hash value-based algorithm (such as the LSH algorithm, the Nils Simsa algorithm, etc.) may be used to calculate all deduplicated first file blocks to obtain a hash value for each first file block. After obtaining the hash value for each first file block, the hash values ​​may be sorted from high to low by similarity, and then, based on the size of the data that a second file block can contain, the first file blocks that can be aggregated into a second file block may be determined. For example, first file blocks, ranked by similarity from highest to lowest, are added to the second file block one by one until the second file block is full. For example, the network device pre-sets the second file block size to contain 1024KB of data, and the divided first file blocks are each 258KB. Then, if the hash values ​​of the first file blocks are obtained and sorted from highest to lowest by similarity, the result is first file block 2, first file block 4, first file block 6, first file block 5, and first file block 1. During similarity aggregation, after first file block 2, first file block 4, first file block 6, and first file block 5 are added to second file block 1 and aggregated, the second file block already contains 1024KB of data, and first file block 1 cannot be added. Therefore, second file block 1 is obtained by aggregating first file block 2, first file block 4, first file block 6, and first file block 5. Other second file blocks can be obtained by similarity aggregation again. The method is similar to this example and will not be further described. The sizes of the first file blocks are not limited to the example shown here and can be the same or different.

[0065] In one example, the network device may pre-set a minimum threshold for similarity. If the similarities of multiple first file blocks sorted from high to low are greater than the pre-set minimum threshold, they may be aggregated into at least one second file block. For example, if the first file block 2, the first file block 4, the first file block 6, the first file block 5, and the first file block 1 are all greater than the pre-set minimum threshold, and when similarity aggregation is performed, the size of the second file block can include three first file blocks, then the first file block 2, the first file block 4, and the first file block 6 are aggregated into the second file block 1, the first file block 5 and the first file block 1 are aggregated into the second file block 2, and so on.

[0066] In one example, a network device can use an algorithm to calculate multiple first file blocks after deduplication. Based on the characteristics of LSH, first file blocks that can be output to the same bucket have a high degree of similarity. Therefore, the LSH algorithm can be used. After the LSH algorithm is used, the similarity between the first file blocks output together is high, and they can be aggregated into the same second file block based on similarity. Similarly, if the Nils Simsa algorithm or other hash value-based algorithm is used to calculate, the first file blocks whose similarity meets the preset rules can be aggregated into the second file block in the same way.

[0067] For example, first file block 1 includes the character strings 9, 8, and 7; first file block 2 includes the character strings 4, 5, and 6; first file block 4 includes the character strings 9, 8, and 5; first file block 5 includes the character strings 9, 5, and 6; and first file block 6 includes the character strings 9, 8, and 1. After the network device performs similarity aggregation on these first file blocks, assuming that two digits of the character strings are identical, it is considered that the preset rule is met. The network device then performs similarity aggregation on the first file blocks with two identical digits to obtain a second file block. The determination of two identical digits of the character strings is based on a hash value algorithm. Second file block 1 includes first file block 1, first file block 4, and first file block 6. That is, the first file block with the first digit of the character string being 9 and the second digit being 8 is aggregated to form second file block 1. Second file block 2 includes first file block 2 and first file block 5. That is, the first file block with the second digit of the character string being 5 and the third digit being 6 is aggregated to form second file block 2. First file blocks that have been aggregated into second file blocks no longer participate in similarity aggregation of other second file blocks.

[0068] In the embodiments of the present application, the character strings are exemplified by numbers only to intuitively represent the similarity of the data of the character strings, and the numbers are not used to limit the content of the character strings.

[0069] S104: The network device compresses each second file block to obtain a first compressed block.

[0070] For example, if a network device compresses second file block 1, which contains the strings 9, 8, and 7 from first file block 1, 9, 8, and 5 from first file block 4, and 9, 8, and 1 from first file block 6, the device can record the repeated data in each first file block in second file block 1 with a short code based on a compression method such as a compression dictionary. For example, if 9 and 8 are identical data, they can be recorded as x. After compression, second file block 1 is compressed into first compressed block 1, which includes x, 7, x, 5, and x, 1. Similarly, if the network device compresses second file block 2, which contains the strings 4, 5, and 6 from first file block 2 and the strings 9, 5, and 6 from first file block 5, the device can record the data 5 and 6 from second file block 2 as y. After compression, second file block 2 is compressed into first compressed block 2, which includes 4, y and 9, y. Furthermore, only one identical short code can be retained, pointing to multiple first file blocks, to facilitate recovery during decompression. This method can be implemented by referring to the inode node description after deduplication in S102, and will not be described in detail.

[0071] Optionally, a short code may generate a correspondence table between repeated data and codes according to repeated data in each second file block to be compressed, and a code length smaller than the number of bits of repeated data is defined as a short code.

[0072] In an embodiment of the present application, a network device first divides and de-duplicates a first file to be compressed, and then aggregates the de-duplicated first file blocks based on similarity. The first file blocks aggregated in a second file block have a high degree of similarity, and compression is performed on each second file block. That is, duplicate data in the first file blocks with high similarity is recorded and compressed using a shorter code. This can significantly reduce the amount of data in the first compressed block, thereby improving the compression ratio.

[0073] In one possible implementation, when executing the first operation, the network device can only identify file blocks that are segmented and compressed according to a preset data length. In this case, the network device needs to first determine whether the data contained in the file to be compressed is used to execute the first operation. The file to be compressed required to execute the first operation can be determined as the second file, and the file to be compressed used to execute operations other than the first operation can be determined as the first file, and compressed in different ways. Figure 2 is a flow diagram of another data compression method provided in an embodiment of the present application. As shown in Figure 2, based on Figure 1, the method also includes S105 before S101, and S106 to S108.

[0074] S105: The network device determines whether the data contained in the file to be compressed is used to perform the first operation.

[0075] If yes, execute S106 to S108 , and if no, execute S101 to S104 .

[0076] Optionally, the first operation includes performing a version upgrade, such as when the network device currently uses a system or a program that needs to be upgraded to the latest version. However, the data used for the version upgrade can only be compressed based on a preset data length before it can be recognized by the current version of the system and upgraded based on recognizable compressed blocks. Therefore, before compressing the file to be compressed, the network device needs to first determine whether the data contained in the file to be compressed is used for version upgrade. If so, it needs to be segmented according to the preset data length. If not, it can be segmented according to other rules.

[0077] The first operation may also be other operations that require identification of the data size of the compressed block. The embodiment of the present application uses the example of the first operation being a version upgrade of Squashfs as an example for explanation, but is not limited to this.

[0078] S106: The network device divides the data in the second file into blocks according to a preset data length, and obtains at least one third file block based on each second file.

[0079] For example, assuming that the network device uses the Squashfs read-only file system, the Squashfs read-only file system can only recognize compressed blocks that are divided into 128k blocks and added to the Squashfs read-only image. Then, the preset data length can be set to 128k, that is, the network device divides the data in each second file into 128k blocks, and obtains at least one 128k third file block for each second file. For example, if the data in a second file A is 256k, after block division, the third file block 1 will include 128k of data and the third file block 2 will include 128k of data. If the data in a second file B is 200k, it will be divided into a third file block 3 including 128k of data and a third file block 4 including 82k of data, etc.

[0080] S107: The network device deduplicates the second file blocks with the same data.

[0081] In one possible implementation, the network device deduplicates the third file block with exactly the same data based on Squashfs, and saves the pointer of the third file block in the file inode table. For example, if the character strings in the third file block 1 and the third file block 3 are exactly the same, namely 0, 1, 2, 3, 4, after deduplication, the character strings 0, 1, 2, 3, 4 are saved, and the pointers of the character strings are pointed to the third file block 1 and the third file block 3. Alternatively, if the third file block 1 is obtained by splitting the second file 1, and the third file block 3 is obtained by splitting the second file 2, then the pointers of the data content can point to the second file 1 and the second file 2 respectively.

[0082] S108: The network device compresses each second file block after deduplication to obtain a corresponding second compressed block.

[0083] Optionally, the network device compresses each second file block to obtain a second compressed block, and the second compressed block can be compressed into a compression format such as xz or lzma. At least one second compressed block is stored in compressed file 1 to compressed file N of the Squashfs image file to facilitate file recovery when decompression is required.

[0084] By compressing the first compressed block and the second compressed block after compression in the read-only image, the compression ratio of the read-only image can be improved, so that network devices with limited storage space, such as miniaturized devices, can store more data, effectively saving the storage cost of the network device. At the same time, the file to be compressed is divided into different blocks based on different operations and then compressed, which is compatible with operations that can only be performed based on file blocks of certain sizes, making the network device compression more compatible. For example, if the first operation is a Squashfs version upgrade, the network device can only recognize the Squashfs image, that is, it can only upgrade and compress files that are divided into 128k blocks and address them according to the addressing method of the existing version. In this case, if the network device obtains file block deduplication and similarity aggregation according to other block division rules, and addresses them according to the addressing method corresponding to the second file block after similarity aggregation, the network device will be unable to recognize the file block, which will cause the old version to be unable to upgrade to the new version, causing a generation gap and affecting the version upgrade of the network device. Therefore, compressing the files used for the Squashfs version upgrade in a way that it can recognize can effectively prevent the occurrence of generation gaps. Therefore, referring to the method provided in FIG2 , the network device determines the files to be compressed as the first file and the second file respectively according to the purpose of the data contained in the files to be compressed, and divides the second file whose data is used for version upgrade into blocks in a manner that can identify the Squashfs image, thereby being compatible with the Squashfs read-only image, so that the old version can be upgraded to the latest read-only file system. The method provided in the embodiment of the present application can improve the compression ratio of the Squashfs read-only image while being compatible with the files to be compressed required for the upgrade of the Squashfs read-only image file system, thereby achieving the miniaturization of network devices and effectively preventing the occurrence of version upgrade discontinuities.

[0085] In one possible implementation, the first compressed block and the second compressed block obtained by executing steps S104 and S108 can both be stored in an image file, such as a Squashfs image file. FIG3 is a schematic diagram of the field composition structure of a Squashfs image file provided by an embodiment of the present application. In one example, the fields included in the Squashfs image file can refer to FIG3 and include but are not limited to the following parts:

[0086] Superblock: used to record the total amount, usage amount, remaining amount, etc. of the second file index node (inode) and the second compressed block.

[0087] Compressed file 1 to compressed file N: each compressed file is used to store a second compressed block, and N is a positive integer greater than 0.

[0088] File index node table (inode table): includes pointers to third file blocks. The pointers to third file blocks refer to S107 and can indicate which second file each third file block comes from.

[0089] Extended superblock (superblockEx): records basic information about data blocks and buckets, such as the size and number of data blocks, the number of buckets, and offsets. The offset is used to represent the position of each first compressed block in the data block.

[0090] Bucket table (Chunk_table): records the correspondence between each first file and a bucket (chuck).

[0091] Bucket (chuck): Indicates in which data block the first compressed block is stored and its offset in the data block.

[0092] Data block 1 to data block M: a data block stores at least one first compressed block, and N and M are positive integers greater than 0.

[0093] In a possible implementation, if a data block still has storage space after storing a first compressed block, a second file block can be placed therein, and so on.

[0094] Optionally, the storage space that may be occupied by the data contained in the second file block after compression can be matched with the size of the remaining storage space of the data block, and a second file block whose storage space may be occupied by the data contained in the second file block after compression is less than or equal to the size of the remaining storage space of the data block can be selected, and the data can be stored in the data block after compression.

[0095] An example of storing multiple first compressed blocks and multiple second compressed blocks in a Squashfs image file is as follows:

[0096] FIG4 is a schematic diagram of a compression process provided by an embodiment of the present application. As shown in FIG4 , referring to the above example, after the first file A, the first file B, and the first file C are divided into blocks, deduplicated, and similarity aggregated, the second file block 1 obtained includes the first file block 1, the first file block 4, and the first file block 6, and the second file block 2 includes the first file block 2 and the first file block 5. After compression, the second file block 1 is compressed into the first compressed block 1, and the second file block 2 is compressed into the first compressed block 2. Assuming that the first compressed block 1 and the first compressed block 2 are both stored in the data block 1, then the bucket indicates that the first compressed block 1 and the first compressed block 2 point to the data block 1, and respectively indicates the offset of the first compressed block 1 in the first data block and the offset of the first compressed block 2 in the data block 1. Based on the fact that the first file A is divided into the first file block 1 and the first file block 2, the first file B is divided into the first file block 3, and the first file C is divided into the first file block 4, the first file block 5, and the first file block 6, and after similarity aggregation, bucket 1 corresponds to the second file block 1, including the first file block 1, the first file block 4, and the first file block 6, and bucket 2 corresponds to the second file block 2, including the first file block 2 and the first file block 5. Therefore, in the bucket table (Chunk_table), the buckets corresponding to the first file A are bucket 1 and bucket 2; the buckets corresponding to the first file C are bucket 1 and bucket 2; the first file block 3 obtained by dividing the first file B is deduplicated (refer to S102), but the data is exactly the same as that of the first file block 1, and corresponds to the same bucket as the first file block 1, that is, the bucket corresponding to the first file B is bucket 2.

[0097] Referring to the above example, for the files to be compressed used to perform the first operation, if the second file A is segmented into blocks, the third file block 1 and the third file block 2 will be obtained, and the second file B is segmented into blocks, the third file block 3 and the third file block 4 will be obtained. If the third file block 1 and the third file block 3 are the same, after deduplication, the third file block 1 is compressed into the second compressed block 1 and saved in the compressed file 1, the third file block 2 is compressed into the second compressed block 2 and saved in the compressed file 2, and the third file block 4 is compressed into the second compressed block 3 and saved in the compressed file 3. Since the deleted third file block 3 is obtained by segmenting the second file 2, the pointers of the first compressed file 1 in the file index node table (inode table) point to the second file 1 and the second file 2 respectively, the pointer of the second compressed file 2 points to the second file 1, and the pointer of the second compressed file 3 points to the second file 2.

[0098] The first compressed block and the second compressed block can also be saved in other files. The embodiment of the present application provides that the first compressed block and the second compressed block are respectively stored in the Squashfs mirror file as shown in the above example. The fields in the Squashfs mirror file include: extended superblock (superblockEx), bucket table (Chunk_table), bucket (chuck) and data block (Data block), superblock (Superblock), compressed file 1 to compressed file N and file index node table (inode table) is an example. In the actual usage scenario, the storage of the first compressed block and the second compressed block can be adjusted according to demand. The system for saving the first compressed block and the second compressed block can refer to the index provided in the embodiment of the present application to indicate each file to be compressed or the block of file blocks, and is not limited to the fields provided in the embodiment of the present application.

[0099] In addition, an embodiment of the present application also provides a device. Figure 5 is a structural schematic diagram of a device provided by an embodiment of the present application. As shown in Figure 5, the device 10 includes: a first blocking module 101, a first deduplication module 102, a similarity aggregation module 103 and a first compression module 104.

[0100] The first block module 101 is configured to divide the data contained in the first file into blocks, and obtain at least one first file block based on each first file.

[0101] The first deduplication module 102 is configured to dedupe all first file blocks having the same data.

[0102] The similarity aggregation module 103 is used to perform similarity aggregation on the multiple first file blocks to obtain at least one second file block if multiple first file blocks are obtained after deduplication by the deduplication module, wherein similarity aggregation refers to aggregating at least one first file block that meets the preset rules into one second file block based on the hash value.

[0103] The first compression module 104 is configured to compress each second file block to obtain a corresponding first compressed block.

[0104] In a possible implementation, the first file is a file to be compressed, and data contained in the first file is used to perform operations other than the first operation.

[0105] In one possible implementation, similarity aggregation includes determining the similarity of each first file block based on a preset algorithm, and performing aggregation based on the amount of data that each second file block can contain; or, similarity aggregation includes determining the similarity of each first file block based on a preset algorithm, and performing aggregation based on a preset minimum threshold of similarity; or, similarity aggregation includes performing aggregation based on an output result of a preset algorithm; wherein the preset algorithm is associated with a hash value.

[0106] It should be understood that the modules shown in Figure 5 are only examples. The first blocking module 101, the first deduplication module 102, the similarity aggregation module 103 and the first compression module 104 can refer to the method part in the embodiment of the present application to perform their operations, or perform variations of their operations. For example, the device 10 can be applied to the scenario provided in Figure 1 and perform its operations as a network device, or perform variations of its operations, which will not be further explained.

[0107] FIG6 is a schematic structural diagram of another device provided in an embodiment of the present application. As shown in FIG6 , the device 10 is based on FIG5 and further includes: a second blocking module 105 , a second deduplication module 106 and a second compression module 107 .

[0108] A second block division module 105 is configured to divide the data contained in the second file into blocks according to a preset data length, and obtain at least one third file block based on each second file, wherein the second file is a file to be compressed and the data contained in the second file is used to perform the first operation;

[0109] A second deduplication module 106, configured to dedupe all second file blocks with identical data;

[0110] The second compression module 107 is configured to compress each second file block after deduplication to obtain a corresponding second compressed block.

[0111] In one possible implementation, the first operation is to upgrade the version of Squashfs.

[0112] It should be understood that the modules shown in Figure 6 are only examples. The first blocking module 101, the first deduplication module 102, the similarity aggregation module 103, the first compression module 104 and the second blocking module 105, the second deduplication module 106 and the second compression module 107 can refer to the method part in the embodiment of the present application to perform their operations, or perform variations of their operations. For example, the device 10 can be applied to the scenario provided in Figure 2 as a network device to perform its operations, or perform variations of its operations, and no further explanation will be given.

[0113] Optionally, FIG7 is a schematic structural diagram of another device provided in an embodiment of the present application. As shown in FIG7 , the device 10 is based on FIG6 and further includes: a storage module 108 .

[0114] The device 10 may further include a storage module 108 for storing at least one first compressed block and at least one second compressed block, such as storing the first compressed block and the second compressed block in a Squashfs image file or other file, such as in the field shown in FIG3 .

[0115] The division of modules in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. The functional modules in the embodiments of the present application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. For example, in the above embodiment, the transceiver module 301 and the processing module 302 can be the same module or different modules; the transceiver module 401 and the processing module 402 can be the same module or different modules. The above-mentioned integrated modules can be implemented in the form of hardware, such as a chip, or in the form of software functional modules.

[0116] In addition, an embodiment of the present application further provides a device 20, as shown in FIG8 , which is a schematic diagram of the structure of a device 20 provided in an embodiment of the present application. As shown in FIG8 , the device 20 may include a processor 201, a memory 202 coupled to the processor 201, and a transceiver 203. The transceiver 203 may be a communication interface, an optical module, etc., and is used to receive messages or data information. The processor 201 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and a NP, and is used to perform the forwarding processing steps in the device exemplified in the above embodiment. The processor may also be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof. The processor 201 may be a single processor or may include multiple processors. The memory 202 may include a volatile memory, such as a random-access memory (RAM); the memory may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid-state drive (SSD); the memory 202 may also include a combination of the above types of memory. The memory 202 may refer to a single memory or may include multiple memories for storing program instructions. In one embodiment, the memory 202 stores computer-readable instructions, which include multiple software modules, such as a sending module, a processing module, and a receiving module. After executing each software module, the processor 201 may perform corresponding operations according to the instructions of each software module. In this embodiment, the operation performed by a software module actually refers to the operation performed by the processor 201 according to the instructions of the software module. Optionally, the processor 201 may also store program codes or instructions for executing the embodiments of the present application. In this case, the processor 201 does not need to read the program codes or instructions from the memory 202.

[0117] The device 20 can be used to execute the methods in the above embodiments. Specifically, the device 20 can be used as a network device to execute the operations of methods S101 to S104, or the device 20 can be used as a network device to execute the operations of methods S101 to S108. For example, the processor 201 is used to divide the data contained in the first file into blocks, and obtain at least one first file block based on each first file; deduplicate the first file blocks with the same data; if multiple first file blocks are obtained after deduplication, similarity aggregation is performed on the multiple first file blocks to obtain at least one second file block, wherein similarity aggregation refers to aggregating at least one first file block that meets the preset rules into a second file block based on the hash value; and compressing each second file block to obtain a corresponding first compressed block.

[0118] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a processor, it implements part or all of the operations in any of the methods in any of the aforementioned embodiments.

[0119] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed on a processor, implements part or all of the operations in any of the methods in any of the aforementioned embodiments.

[0120] The present application also provides a chip including an interface circuit and a processor connected to each other, wherein the processor is configured to cause the chip to execute part or all of the operations in any of the methods in any of the aforementioned embodiments.

[0121] An embodiment of the present application also provides a chip system, including: a processor, the processor is coupled to a memory, the memory is used to store programs or instructions, when the program or instructions are executed by the processor, the chip system implements part or all of the operations of any one of the methods of any one of the embodiments described above.

[0122] Optionally, there may be one or more processors in the chip system. The processor may be implemented in hardware or software. When implemented in hardware, the processor may be a logic circuit, an integrated circuit, etc. When implemented in software, the processor may be a general-purpose processor implemented by reading software code stored in a memory.

[0123] Optionally, the memory in the chip system may be one or more. The memory may be integrated with the processor or may be provided separately from the processor, which is not limited in the embodiments of the present application. For example, the memory may be a non-transient processor, such as a read-only memory (ROM), which may be integrated with the processor on the same chip or provided on different chips. The embodiments of the present application do not specifically limit the type of memory or the configuration of the memory and the processor.

[0124] Exemplarily, the chip system can be an FPGA, an ASIC, a system on chip (SoC), a CPU, an NP, a digital signal processing circuit (DSP), a microcontroller (MCU), a programmable logic device (PLD), or other integrated chips.

[0125] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0126] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For example, the division of units is only a logical business division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0128] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0129] If the integrated unit is implemented in the form of a software business unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the technical solution of the present application can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, ROM, RAM, Random Access Memory, disk or optical disk, etc. Various media that can store program code.

[0130] Those skilled in the art will appreciate that, in one or more of the examples above, the services described herein may be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these services may be stored in a computer-readable medium or transmitted as one or more instructions or codes on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transmission of computer programs from one location to another. Storage media may be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0131] The above specific implementation methods further describe the purpose, technical solutions and beneficial effects of this application in detail. It should be understood that the above are only specific implementation methods of this application.

[0132] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for data compression, characterized in that: include: Dividing the data contained in the first file into blocks, and obtaining at least one first file block based on each of the first files; Deduplication of the first file blocks having the same data; If multiple first file blocks are obtained after deduplication, similarity aggregation is performed on the multiple first file blocks to obtain at least one second file block, wherein the similarity aggregation refers to aggregating at least one first file block that meets a preset rule into one second file block based on a hash value; Compressing each of the second file blocks correspondingly obtains a first compressed block.

2. The method according to claim 1, characterized in that Also includes: Divide the data contained in the second file into blocks according to a preset data length, and obtain at least one third file block based on each of the second files, wherein the data contained in the second file is used to perform the first operation; Deduplication of the second file blocks having the same data; Each of the second file blocks after deduplication is compressed to obtain a corresponding second compressed block.

3. The method according to claim 1 or 2, characterized in that: The similarity aggregation includes determining the similarity of each of the first file blocks based on a preset algorithm, and performing aggregation based on the amount of data that each of the second file blocks can contain; or, The similarity aggregation includes determining the similarity of each of the first file blocks based on the preset algorithm, and performing aggregation based on a preset minimum threshold of the similarity; or, The similarity aggregation includes aggregation based on the output result of the preset algorithm; The preset algorithm is associated with the hash value.

4. The method according to any one of claims 1 to 3, characterized in that: Also includes: Storing the first compressed block in an image file; The image file includes an extended super block, a data block, a bucket, and a bucket table. The extended super block is used to store information about the data block and the bucket, each of the data blocks is used to store at least one of the first compressed blocks, the bucket is used to indicate the data block corresponding to each of the first compressed blocks, and the offset of each of the first compressed blocks in the data block, and the bucket table is used to store the correspondence between each of the first files and the bucket.

5. The method according to any one of claims 2 to 4, characterized in that: The first operation is to upgrade the version of Squashfs.

6. A device, characterized in that: include: A first block division module, used to divide the data contained in the first file into blocks, and obtain at least one first file block based on each of the first files; A first deduplication module, used for deduplicating all the first file blocks with the same data; A similarity aggregation module, configured to perform similarity aggregation on the plurality of first file blocks to obtain at least one second file block if the deduplication module obtains a plurality of the first file blocks after deduplication, wherein the similarity aggregation refers to aggregating at least one first file block that meets a preset rule into one second file block based on a hash value; The first compression module is used to compress each of the second file blocks to obtain a first compressed block.

7. The device according to claim 6, characterized in that Also includes: A second block division module, used to divide the data contained in the second file into blocks according to a preset data length, and obtain at least one third file block based on each of the second files, wherein the second file is a file to be compressed, and the data contained in the second file is used to perform the first operation; A second deduplication module, used for deduplicating all the second file blocks with the same data; The second compression module is used to compress each of the second file blocks after deduplication to obtain a corresponding second compressed block.

8. The device according to claim 6 or 7, characterized in that The similarity aggregation includes determining the similarity of each of the first file blocks based on a preset algorithm, and performing aggregation based on the amount of data that each of the second file blocks can contain; or, The similarity aggregation includes determining the similarity of each of the first file blocks based on the preset algorithm, and performing aggregation based on a preset minimum threshold of the similarity; or, The similarity aggregation includes aggregation based on the output result of the preset algorithm; The preset algorithm is associated with the hash value.

9. The device according to any one of claims 6 to 8, characterized in that Also includes: A storage module, used to store the first compressed block in a mirror file; wherein the mirror file includes an extended super block, a data block, a bucket and a bucket table, the extended super block is used to store information about the data block and the bucket, each of the data blocks is used to store at least one of the first compressed blocks, the bucket is used to indicate the data block corresponding to each first compressed block and the offset of each first compressed block in the data block, and the bucket table is used to store the correspondence between each first file and the bucket.

10. The device according to any one of claims 7 to 9, characterized in that The first operation is to upgrade the version of Squashfs.

Images