Device and method for compacting compressed data blocks and uncompressed data blocks

Abstract

The present invention relates to the field of data compression and data compaction. Specifically, the present invention provides a device for compacting compressed data blocks and uncompressed data blocks into an output buffer in an improved manner. Thus, wasted space is reduced or eliminated. The device is used to: obtain an input data block set, wherein the set includes at least one of a compressed data block and an uncompressed data block; compact the compressed data blocks into the output buffer starting from a first predefined area in the output buffer, so that the compressed data blocks are compacted in sequence; and compact the uncompressed data blocks into the output buffer starting from a second predefined area in the output buffer, so that the uncompressed data blocks are compacted in sequence.

Description

Technical Field

[0001] The present invention generally relates to the field of data compression and data compaction. More specifically, the present invention relates to an apparatus and method for compacting compressed data blocks and uncompressed data blocks into an output buffer. The present invention also relates to an apparatus and method capable of reducing wasted space of a storage medium when storing compressed data blocks, etc. Background Art

[0002] Lossless data compression is one of the commonly used algorithms in all-flash array storage devices. When data is written to the storage device, data compression can reduce the size of the stored data. In addition, when the compressed data blocks are read from the storage device, these compressed data blocks are decompressed to the original content and size. In the primary storage device, data compression and decompression are done transparently "in-line / on-the-fly" to the applications that write and read the data.

[0003] Moreover, lossless data compression can increase the effective storage capacity of storage devices, so more data can be stored in the storage device. In addition, since less data is written to storage devices such as solid state drives (SSDs), data compression can reduce the wear of the SSD, thereby improving the durability of the SSD.

[0004] Traditional devices (e.g., memory vendors) provide a compression ratio (CR) of 4:1, in other words, the algorithms used by traditional memory vendors compress data to 4 / 1 of its original size. In addition, some computer programs and / or applications write data (e.g., input data blocks) to storage devices in fixed-size blocks, and compressing these blocks can produce compressed data blocks of different sizes.

[0005] Furthermore, if the input data block is incompressible, compressing the input data block will produce an output block that is larger than the input block due to metadata compression overhead, etc. In this case, the compressed data block will be discarded by the storage pipeline, and the original input data block can be written to the storage device.

[0006] Some computer programs and / or applications can read data blocks from storage devices in any manner, such as non-continuously. In addition, in order to speed up reading compressed data blocks from storage devices at any address, other data structures of indirect addressing (such as address translation) can be used due to the variable size of compressed data blocks.

[0007] Traditionally, to perform online data compression, a granularity unit of a fixed size of 1 kilobyte (KB) is used to write a compressed data block. For example, if an 8KB input data block is compressed to a 3.5KB compressed data block, four 1KB granularity units are required to write the compressed data block. In this case, the compressed data block can be written to the storage medium using the granularity unit. In addition, the size and offset of the written compressed data block can be represented using the granularity unit.

[0008] Then, some other algorithms may be used to compact (hereinafter also referred to as "pack") the granular units with compressed data into chunks (hereinafter also referred to as "output buffer"), such as chunks of a fixed size of 1 megabyte (MB).

[0009] Fig.14 Schematic diagram of the traditional compression method for 8KB input data blocks and the further packing of compressed data blocks into chunks.

[0010] First, the 8KB input data blocks are compressed; then, the compressed data blocks are packed into chunks. Fig.14 Two chunks (i.e., chunk 1 and chunk 2) of a fixed size of 1 MB are schematically shown. Both chunks include compressed data chunks and uncompressed data chunks, which are packed by aligning the 1KB granularity unit.

[0011] In addition, the block address (granular address) includes a chunk identifier (chunk ID), a data offset, and a data size (hereinafter also referred to as length). The block address can be used for address indirection (ie, address translation) in a random read scenario.

[0012] Fig.15 It is a schematic diagram of a method of unpacking and decompressing a data block packed by 1KB granularity alignment from a chunk in the prior art.

[0013] Fig.15 The schematic diagram shows two chunks including chunk 1 and chunk 2. Fig.14 The two chunks are based on the two chunks in the . These two chunks have a fixed size of 1MB and include compressed data chunks and uncompressed data chunks that are aligned to 1KB granularity units during packaging.

[0014] As mentioned above, the chunk address includes a chunk ID, an offset, and a length. First, the chunk address is used and the data (i.e., the packed compressed data) is unpacked. Then, the compressed data chunk is decompressed into an 8KB output data chunk.

[0015] As shown in the figure, the chunk address includes a chunk ID, an offset, and a length, which are used to read, unpack, and decompress data from a chunk.

[0016] also, Fig.14 and Fig.15 The method described uses 1 bit per granularity unit in order to mark data that is in use (eg, active) and / or deleted. A garbage collector algorithm that performs storage defragmentation requires the use of the 1 bit per granularity unit used above.

[0017] However, a disadvantage of the above conventional method is that the last granularity unit of the compressed data block usually includes a compressed data block smaller than 1 KB, thus wasting the unused space of the granularity unit.

[0018] For example, if an 8KB input data block is compressed to 2KB+1B, three 1KB granularity units are required for packing. Since only 1 byte of the third granularity unit is used, 1023 bytes of the third granularity unit are wasted.

[0019] Therefore, the above conventional method has the disadvantage of generating wasted space, such as wasted space generated when the compressed data blocks are packed into chunks.

[0020] Fig.16 A schematic diagram of wasted space in a granularity unit when compressing data blocks and aligning them in 1KB granularity units for packing.

[0021] exist Fig.16 In the example, 4 compressed data blocks are packed by aligning them in 1KB granularity units. The first compressed data block occupies 2 granularity units, each of which has a fixed size of 1KB. Since the size of the first block is less than 2KB, part of the second granularity unit is wasted. Similarly, the second compressed data block occupies the 3rd, 4th, and 5th granularity units, wasting part of the 5th granularity unit. In addition, the third compressed data block occupies the 6th and 7th granularity units, wasting part of the 7th granularity unit, and so on.

[0022] Fig.17 A schematic diagram of wasted space in a granularity unit when compressed data blocks and uncompressed data blocks are packed by aligning them in a 1KB granularity unit.

[0023] exist Fig.17In the example, several blocks including compressed data blocks B0, B1, B3, B4, B5 and Bn and uncompressed data blocks B2 and B6 are aligned and packed into chunks by 1KB granularity units.

[0024] First, two uncompressed data blocks B0 and B1 are packed into a chunk. It can be seen that two wasted spaces are formed, the first wasted space is located between B0 and B1, and the second wasted space is located in the last granularity unit occupied by B1 (the fifth granularity unit of the chunk). Then, the uncompressed data block B2 is packed into a chunk, followed by the packed compressed data blocks B3, B4, and B5, etc. In this way, several wasted spaces are formed between the compressed data blocks, namely the wasted space between the packed data blocks B3 and B4, the wasted space between the packed data blocks B4 and B5, and so on.

[0025] The disadvantage of the above method is that it generates wasted space when the last granularity unit of the compressed data block is usually less than 1KB, and wastes the used space in the granularity unit. For example, in the case of an average compression ratio of 2:1, a chunk of 1MB may include 256 compressed data chunks. In addition, a 1MB chunk may also include an average wasted space size of 128KB (512×256=128KB), which results in 12.5% ​​wasted space in the chunk, where the average wasted space size in the last 1KB granularity unit of the compressed data chunk is 512 bytes.

[0026] For example, when an 8KB input data block is compressed to 3KB+1B on average, a larger amount of wasted space, ie, 256KB or 25% of the chunk, is accumulated.

[0027] In addition to the waste of space in the storage medium, there is also waste in the central processing unit, which is called CPU waste. For example, when a compressed data block is discarded because it does not fit into 6 granularity units, etc., CPU waste occurs. The limitation on the size of the compressed data block in the above method is due to the low bit utilization in the block address. The above conventional method uses 3 bits as the compressed data block size, expressed in 1KB granularity units. For example, when the uncompressed data block uses the bit pattern "111", the bit pattern "000" is invalid, and the compressed data block with a size ranging from 1 to 6 granularity units uses the other 6 bit patterns.

[0028] In conventional devices and methods, any compressed data blocks that require more than 6KB will be discarded, thus wasting corresponding CPU cycles. Therefore, in addition to wasting space in the storage medium, the above method also has other disadvantages, namely causing CPU waste.

[0029] Fig.18 A disclosed method of compacting compressed data blocks and uncompressed data blocks into a solid state drive (SSD) memory in the prior art is shown.

[0030] For example, the input data corresponding to the input file is first compressed. The sizes of the compressed data blocks are 0.5KB, 1.0KB, 2.0KB, 1.3KB, 1.0KB, 0.5KB, 0.5KB, 1.0KB, and 2.0KB, respectively, and these compressed data blocks are ready to be written to the SSD. Without data compression, a 4KB SSD block is allocated to each file (i.e., each compressed data block), using a total of 36KB of the storage device.

[0031] If data is compacted by compressing the data blocks, multiple files can be written to each 4KB block, thus using only 12KB of the storage device.

[0032] However, in the above method, the sum of the sizes of multiple blocks compressed into one 4KB block may be less than 4KB (eg, 3KB, 3.5KB, etc.), and therefore, wasted space is formed in each 4KB block.

[0033] Therefore, the disadvantage of the above method is that, for example, the compressed data block will form wasted space when compacted into the SSD. In addition, another disadvantage of the above method is that read amplification may occur when reading the "wasted bytes".

[0034] Fig.19 Another method of compressing and compacting input data files into a memory device is shown as described in US 2012 / 0290798 A1.

[0035] Fig.19 The HOST column in FIG. 1 shows a plurality of data areas grouped according to logical addressable units (LAUs), with each LAU having 4 data areas. There are a total of 5 LAUs, labeled LAA0 to LAA4.

[0036] The data area shown in the host column is the uncompressed data area. After compression, the data area occupies less space. For example, the data area of ​​LAA0 stored in DDR is compressed by 50%, so LAA0 occupies 50% of the memory block in DDR. The LAU written to DDR does not adopt this compression scheme, resulting in part of the space in the DDR memory block being empty. The data area in LAA1 is also compressed by 50%, and the data area in LAA2 is compressed by 25%. The data area in LAA3 cannot be compressed evenly, but the overall compression ratio of LAA3 is 50%. The compression ratio of LAA4 is 100%.

[0037] After LAUs are compacted, they are grouped together into hardware addressable units (HAUs). When LAUs stored in double data rate (DDR) memory are compacted into a single HAU, the firmware causes these LAUs to be written to the storage device (SSD).

[0038] However, the above method generates wasted space, for example, when rounding the compressed data area, wasted space may be formed between compressed data blocks. In addition, wasted space may also be formed during the process of compacting into the SSD.

[0039] Although prior art exists for compressing and compacting input data files into storage devices, such as by compacting each input data file into a single block of 4KB in size, it is generally desirable to improve devices and methods for compacting compressed and uncompressed data blocks into output buffers and / or storage media. Summary of the invention

[0040] In view of the above problems and shortcomings, the present invention aims to improve the conventional apparatus and method for data compression and compaction. Therefore, the object of the present invention is to provide an apparatus and method for compacting compressed data blocks and uncompressed data blocks into an output buffer. Specifically, when compressing and compacting data into a buffer, it is necessary to reduce wasted space.

[0041] The objects of the present invention are achieved by the technical solutions provided in the attached independent claims. Advantageous implementations of the present invention are further defined in the dependent claims.

[0042] A first aspect of the present invention provides a device for compacting compressed data blocks and uncompressed data blocks into an output buffer. The device is used to: obtain an input data block set, wherein the set includes at least one of a compressed data block and an uncompressed data block; compact the compressed data blocks into the output buffer starting from a first predefined area in the output buffer, so that the compressed data blocks are compacted in sequence; and compact the uncompressed data blocks into the output buffer starting from a second predefined area in the output buffer, so that the uncompressed data blocks are compacted in sequence.

[0043] According to the first aspect, the waste space formed between compressed data blocks is greatly reduced or even eliminated. For example, using the lossless data compression method of the LZ4 algorithm, through the device provided by the first aspect, the waste elimination can be close to the "ideal situation". In other words, the compressed data blocks can be compacted (for example, in sequence) so that no wasted space is formed between two adjacent compressed data blocks. In addition, a better data reduction ratio can be obtained. For example, the device provided by the first aspect can obtain a data reduction ratio of 13% to 27%. Therefore, compared with the typical online compression packaging scheme of traditional devices, the speed (for example, write and read speed) can also be increased by 6% to 42%.

[0044] Another advantage of the device provided by the first aspect is that the actual compressed data size is taken into account in the packaging scheme, and compared with the typical online compression packaging scheme, the waste of SSD memory is minimized. For example, the wasted space can be reduced to 0.1% of the memory capacity.

[0045] In an implementation of the first aspect, the device is further configured to determine a byte upper limit of the compressed data block according to a size of the output buffer, a size of the compressed data block to be compressed into the output buffer, and a header size of the compressed data block.

[0046] By determining the byte upper limit of the compressed data block, the device can allocate a predetermined space in the output buffer for the compressed data block to be compacted into the output buffer (eg according to the determined upper limit).

[0047] In an implementation manner of the first aspect, the device is further used to: obtain an input data block, compress the obtained input data block according to a predefined compression ratio, and determine the size of the compressed data block.

[0048] The advantage of this implementation is that, by compressing the acquired input data block, the device reduces the data size when writing the data to the storage device.

[0049] In an implementation of the first aspect, when it is determined that the size of the compressed data block is smaller than the granularity unit of the output cache, the device copies the compressed data and adds it to a separate output cache, wherein the separate output cache is associated with the compressed data block smaller than the granularity unit, and the granularity unit represents the granularity of the memory.

[0050] By allocating a separate output buffer for the compressed data blocks smaller than the granularity unit, it is possible to avoid wasting space in the granularity unit including the small data blocks. Therefore, not only can the wasted space in the granularity unit be avoided, but also several small data blocks are copied to the separate output buffer, thereby efficiently utilizing the memory.

[0051] In an implementation manner of the first aspect, the first predefined area is the beginning of the output buffer, and the compressed data blocks are compressed in sequence from the beginning of the output buffer toward the end of the output buffer.

[0052] The advantage of this implementation is that the compressed data blocks are compacted closely to each other, and waste space between the compressed data blocks can be reduced and / or eliminated. In addition, since the compressed data blocks are compacted according to their actual sizes, waste in the storage medium can be minimized.

[0053] In an implementation of the first aspect, the compressed data blocks are compacted into the output buffer without gaps and / or independently of a granularity unit of the output buffer.

[0054] By using a gapless method and / or compacting the compressed data blocks independently of the granularity unit, the device can reduce and / or eliminate wasted space that would otherwise be formed after the compressed data blocks stored in the granularity unit. For example, the device can use actual compressed data blocks with variable sizes. Therefore, the device solves the problem existing in conventional devices, that is, wasted space is retained in each compressed data block and correspondingly retained in the storage medium, that is, in conventional devices, the rounded size of the compressed data block is used instead of the actual size.

[0055] In an implementation manner of the first aspect, the second predefined area is the end of the output buffer, and the uncompressed data blocks are compressed in sequence from the end of the output buffer toward the beginning of the output buffer.

[0056] The advantage of this implementation is that the uncompressed data blocks are compressed closely to each other, and the wasted space between the uncompressed data blocks can be reduced and / or eliminated.

[0057] In an implementation manner of the first aspect, the uncompressed data block is compressed into the output buffer according to a granularity unit of the output buffer.

[0058] The advantage of this implementation is that the size of the uncompressed data block is based on the granularity unit of the output cache. Therefore, no wasted space is formed between two and / or more uncompressed data blocks because these uncompressed data blocks are arranged according to the granularity unit of the output cache.

[0059] In an implementation of the first aspect, the device is further configured to calculate a block address of the compressed data block by determining an offset of the compressed data block in the output buffer and estimating a length of the compressed data block in the output buffer.

[0060] The advantage of this implementation is that the block address, offset and length are calculated, so the device can allocate temporary buffers etc. according to the calculated block address, determined offset and estimated length. Therefore, the compressed data block can be read, written, stored etc. more efficiently.

[0061] In an implementation manner of the first aspect, a block address of a compressed data block smaller than the granularity unit corresponds to an index of the granularity unit.

[0062] Using the index of the granularity unit, the block address of the compressed data block smaller than the granularity unit can be calculated. Therefore, the block address of the small compressed data block can be calculated.

[0063] In an implementation of the first aspect, the device is further configured to calculate a block address of the uncompressed data block by determining an offset of the uncompressed data block in the output buffer and estimating a length of the uncompressed data block in the output buffer.

[0064] The advantage of this implementation is that calculating the block address of the uncompressed data block can allocate space at the end of the chunk (for example) for the uncompressed data block. Therefore, the uncompressed data block can be read, written, stored, etc. more efficiently.

[0065] In an implementation manner of the first aspect, the device is further configured to generate a block preamble for each compressed data block, wherein the block preamble indicates an offset of the compressed data block relative to the start of the granularity unit.

[0066] Generating the block preamble may determine the position of the compressed data block relative to the start of the granularity unit. In addition, a byte offset of the compressed data block may be determined, and the offset may be stored at the start of the first granularity unit of the compressed data block in a chunk.

[0067] In an implementation of the first aspect, the device is further configured to generate a block tail header for each compressed data block, wherein the block tail header indicates an offset of a last byte of the compressed data block relative to a start of a last granularity unit of the compressed data block.

[0068] Generating the block tail header can determine the offset of the last byte of the compressed data block relative to the start of the last granularity unit of the compressed data block. In addition, the tail header can be placed at the start of the last granularity unit of the compressed data block.

[0069] For example, in some embodiments, the compressed data block may occupy 3KB+1 bytes. In addition, assuming that 3KB includes the leading header, 1 byte is retained in the last granularity unit of the compressed data block. In addition, the device may insert the tail header at the beginning of the last granularity unit of the compressed data block (for example, assuming that the tail header is 2 bytes), so the last granularity unit of the compressed data block may be 3 bytes, 2 bytes of which are used for the tail header and 1 byte is used for the compressed data block. In this case, the tail header may include an offset of 3 bytes.

[0070] In an implementation manner of the first aspect, the device further includes a writing module, configured to write the compressed data block and the uncompressed data block into the memory.

[0071] The advantage of this implementation is that the device comprises a writing module and can write data (ie, the compressed data block and the uncompressed data block) into the memory.

[0072] In an implementation of the first aspect, the compressed data blocks and the uncompressed data blocks compressed into the output buffer (104) are written into the memory.

[0073] The advantage of this implementation is that all the advantages provided to the output buffer for the compressed data blocks and the uncompressed data blocks can be transferred to the memory accordingly. In other words, the device can reduce the size of the stored data when the data is written to the storage device. In addition, when the compressed data blocks are read from the storage device, the compressed data blocks are decompressed to the original content and size. In addition, the device increases the effective storage capacity of the memory. The wear of the memory can be reduced, and the durability can also be improved, etc.

[0074] In an implementation manner of the first aspect, the device further includes a reading module, configured to read the compressed data block and the uncompressed data block from the memory.

[0075] The advantage of this implementation is that the compressed data block and the uncompressed data block can be read from the memory.

[0076] In an implementation of the first aspect, the reading module is further configured to read the compressed data block and the uncompressed data block according to an identification number of the output buffer, a size of a corresponding block, and an offset of the corresponding block relative to a start of the output buffer.

[0077] According to a chunk ID (hereinafter also referred to as a chunk identifier, an identification number of the output buffer), the size of the compressed data chunk and the offset of the compressed data chunk relative to the start of the chunk, data can be read faster and more efficiently.

[0078] In an implementation manner of the first aspect, the memory includes a volatile memory or a non-volatile memory.

[0079] The present invention can be applied to memories based on volatile and / or non-volatile systems, and the present invention is not limited to a specific type of memory.

[0080] A second aspect of the present invention provides a method for compacting compressed data blocks and uncompressed data blocks into an output buffer. The method comprises: obtaining an input data block set, wherein the set comprises at least one of a compressed data block and an uncompressed data block; compacting the compressed data blocks into the output buffer starting from a first predefined area in the output buffer, so that the compressed data blocks are compacted in sequence; and compacting the uncompressed data blocks into the output buffer starting from a second predefined area in the output buffer, so that the uncompressed data blocks are compacted in sequence.

[0081] In an implementation manner of the second aspect, the method further includes: determining a byte upper limit of the compressed data block according to a size of the output buffer, a size of the compressed data block to be compressed into the output buffer, and a header size of the compressed data block.

[0082] In an implementation manner of the second aspect, the method further includes: acquiring an input data block, compressing the acquired input data block according to a predefined compression ratio, and determining a size of the compressed data block.

[0083] In an implementation of the second aspect, when it is determined that the size of the compressed data block is smaller than the granularity unit of the output cache, the method further includes copying and adding the compressed data to a separate output cache, wherein the separate output cache is associated with the compressed data block smaller than the granularity unit, and the granularity unit represents the granularity of the memory.

[0084] In an implementation manner of the second aspect, the first predefined area is the beginning of the output buffer, and the compressed data blocks are compressed in sequence from the beginning of the output buffer toward the end of the output buffer.

[0085] In an implementation of the second aspect, the compressed data blocks are compacted into the output buffer without gaps and / or independently of a granularity unit of the output buffer.

[0086] In an implementation manner of the second aspect, the second predefined area is the end of the output buffer, and the uncompressed data blocks are compressed in sequence from the end of the output buffer toward the beginning of the output buffer.

[0087] In an implementation manner of the second aspect, the uncompressed data block is compressed into the output buffer according to a granularity unit of the output buffer.

[0088] In an implementation manner of the second aspect, the method further includes: calculating a block address of the compressed data block by determining an offset of the compressed data block in the output buffer and estimating a length of the compressed data block in the output buffer.

[0089] In an implementation manner of the second aspect, a block address of a compressed data block smaller than the granularity unit corresponds to an index of the granularity unit.

[0090] In an implementation manner of the second aspect, the method further includes: calculating a block address of the uncompressed data block by determining an offset of the uncompressed data block in the output buffer and estimating a length of the uncompressed data block in the output buffer.

[0091] In an implementation manner of the second aspect, the method further includes: generating a block preamble for each compressed data block, wherein the block preamble indicates an offset of the compressed data block relative to the start of the granularity unit.

[0092] In an implementation of the second aspect, the method further includes: generating a block tail header for each compressed data block, wherein the block tail header indicates an offset of the last byte of the compressed data block relative to the start of the last granularity unit of the compressed data block.

[0093] In an implementation manner of the second aspect, the method further includes: writing the compressed data block and the uncompressed data block into the memory.

[0094] In an implementation manner of the second aspect, the compressed data blocks and the uncompressed data blocks compressed into the output buffer are written into the memory.

[0095] In an implementation manner of the second aspect, the method further includes: reading the compressed data block and the uncompressed data block from the memory.

[0096] In an implementation manner of the second aspect, the method further includes: reading the compressed data block and the uncompressed data block according to an identification number of the output buffer, a size of a corresponding block, and an offset of the corresponding block relative to a start of the output buffer.

[0097] In an implementation manner of the second aspect, the memory includes a volatile memory or a non-volatile memory.

[0098] It should be noted that all devices, elements, units and components described in this application can be implemented in software or hardware elements or any type of combination thereof. All steps performed by various entities described in this application and the described functions to be performed by various entities are intended to indicate that each entity is suitable for or used to perform respective steps and functions. Although in the description of the following specific embodiments, the specific functions or steps performed by external entities are not reflected in the description of the specific detailed elements of the entity performing the specific steps or functions, it should be clear to the technician that these methods and functions can be implemented in corresponding hardware or software elements or any combination thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] The following description of specific embodiments in conjunction with the accompanying drawings will illustrate the various aspects of the present invention and their implementation methods.

[0100] Figure 1 A schematic diagram of a device for compacting compressed data blocks and uncompressed data blocks provided by one embodiment of the present invention.

[0101] Figure 2 A detailed schematic diagram of an apparatus for compacting compressed data and uncompressed data provided for one embodiment of the present invention.

[0102] Figure 3 A schematic diagram of compacting compressed data blocks and uncompressed data blocks into 1MB chunks is provided for various embodiments.

[0103] Figure 4 Schematic diagram of calculating block addresses and block headers provided for various embodiments.

[0104] Figure 5 A schematic diagram of an example of a block header provided for various embodiments.

[0105] Figure 6 Schematic diagram of calculating block length provided for various embodiments.

[0106] Figure 7 Schematic diagram of wasted space between two compressed data blocks provided for various embodiments.

[0107] Figure 8 Schematic diagram of two wasted spaces formed between three compressed data blocks provided for various embodiments.

[0108] Fig. 9 Schematic diagram of wasted space between two compressed data blocks provided for various embodiments.

[0109] Fig.10 Schematic diagram of placing a header of several small compressed data blocks for various embodiments.

[0110] Fig.11 A schematic diagram of a method for compacting compressed data blocks and uncompressed data blocks into an output buffer provided by an embodiment of the present invention.

[0111] Fig.12 Schematic diagram of a method of writing an input data block into a memory provided for various embodiments.

[0112] Fig.13 Schematic diagram of a method for reading compressed data blocks and uncompressed data blocks from a memory provided for various embodiments.

[0113] Fig.14 Schematic diagram of traditional compression and packing method of aligning data blocks into chunks with 1KB granularity units.

[0114] Fig.15 A schematic diagram of a conventional method of unpacking and decompressing a data block packed by 1KB granularity unit alignment from a chunk.

[0115] Fig.16 A schematic diagram of wasted space in a granularity unit when compressed data blocks are packed by aligning them in a 1KB granularity unit.

[0116] Fig.17 A schematic diagram of wasted space in a granularity unit when compressed data blocks and uncompressed data blocks are packed by aligning them in a 1KB granularity unit.

[0117] Fig.18 A schematic diagram of a conventional method for compacting compressed data blocks and uncompressed data blocks into a solid state drive memory.

[0118] Fig.19 A schematic diagram of a method for compressing and compacting an input data file into a memory device provided in the prior art. DETAILED DESCRIPTION

[0119] Figure 1 A schematic diagram of a device 100 for compacting compressed data blocks and uncompressed data blocks provided in one embodiment of the present invention. The device 100 is specifically used to obtain a set of input data blocks 101. The set of input data blocks 101 may include at least one of a compressed data block 102 and an uncompressed data block 103. Figure 1In an embodiment of the present invention, the device 100 obtains a set of input data blocks 101, which exemplarily includes 4 compressed data blocks 102 and 4 uncompressed data blocks 103. In some embodiments, the device 100 can also be used to compress uncompressed data blocks, the obtained set of input data blocks, etc.

[0120] The device 100 is configured to compact the compressed data blocks 102 into the output buffer 104 starting from a first predefined area in the output buffer 104 , so that the compressed data blocks 102 are compacted in sequence.

[0121] The device 100 is further configured to compact the uncompressed data blocks 103 into the output buffer 104 starting from a second predefined area in the output buffer 104 , so that the uncompressed data blocks 103 are compacted in sequence.

[0122] For example, the device 100 may obtain a set of input data blocks 101. In addition, the compression methods of the compressed data blocks 102 and the uncompressed data blocks 103 may be different. The starting area for compacting the compressed data blocks 102 into the output buffer 104 (also referred to as a chunk) may be different from the starting area for compacting the uncompressed data blocks 103 into the output buffer 104.

[0123] The compressed data block 102 can be compacted using a granularity unit such as a granularity unit of fixed size 1KB and a chunk (chunk) of fixed size 1MB. The present invention does not limit the specific size of the granularity unit and / or chunk (chunk). In addition, a chunk address including a chunk (chunk) ID, a granularity offset and a length of the compressed data block can be used.

[0124] For example, the compressed data block 102 may be compacted starting from the first predefined area (which may be the beginning of the output buffer 104), and may continue to be compacted toward the end of the output buffer 104, and may be compacted in sequence. In addition, the compressed data block 102 may not be compacted by aligning the granularity unit of the output buffer. Compressing the compressed data block 102 in this way may eliminate wasted space. However, in some embodiments, for example, when 8KB is compressed to less than 1KB, some wasted space may be formed.

[0125] In addition, the uncompressed data block 103 may be compressed into the output buffer 104 starting from a second predefined area in the output buffer 104. The second predefined area may be the end of the output buffer 104. The uncompressed data block 103 may also be compressed from the end of the output buffer toward the beginning of the output buffer 104. The present invention is not limited to a specific area and / or direction. Moreover, the uncompressed data block 103 may be compressed in sequence.

[0126] Furthermore, compacting the compressed data blocks 102 into the output buffer 104 can effectively eliminate wasted space between adjacent compressed data blocks 102. Similarly, compacting the uncompressed data blocks 103 can also eliminate wasted space between adjacent uncompressed data blocks 103. However, for example, when the compressed data blocks 102 and the uncompressed data blocks 103 are compressed next to each other (e.g., are adjacent data blocks), some wasted space may be formed.

[0127] Figure 2 A detailed schematic diagram of a device 100 for compacting compressed data blocks and uncompressed data blocks is provided for one embodiment of the present invention. Figure 2 The device 100 in Figure 1 The device 100 in FIG. 1 is based on the device 100 in FIG. 1 and thus includes all functions and features of the device 100. For this reason, the same features are marked with the same reference numerals. Figure 2 The other features described are optional features of device 100 .

[0128] The device 100 further includes an interface 201 in the form of an adapter for communicatively coupling with other devices such as a host computer, for obtaining a set of input data blocks 101, wherein the set includes at least one of a compressed data block 102 and an uncompressed data block 103. The interface 201 may include a computer bus interface, such as a Serial AT Attachment (SATA), a Parallel Advanced Technology Attachment (PATA), and the like.

[0129] The device 100 further includes a data compressor 202, which is used to compress the input data block 101 obtained by the interface 201 according to a predefined compression ratio and determine the size of the compressed data block 102. For example, the data compressor 202 can use various compression schemes to remove redundant data, metadata, etc. and / or reduce the size of the obtained input data block 101.

[0130] In addition, the device 100 , such as the data compressor 202 therein, can also be used to determine the byte upper limit of the compressed data block 102 according to the size of the output buffer, the size of the compressed data block 102 to be compressed into the output buffer 104 , and the header size of the compressed data block 102 .

[0131] In addition, the device 100 may also optionally include a data compressor 203. The data compressor 203 may obtain information about the size, header size, output buffer size, etc. of the compressed data block, and may also compress the compressed data block 102 and the uncompressed data block 103 into the output buffer 104.

[0132] The device 100 may also optionally include a block address calculator 204. The block address calculator 204 may calculate the block address of the compressed data block 102 by determining the offset of the compressed data block 102 in the output buffer 104 and estimating the length of the compressed data block 102 in the output buffer 104.

[0133] As described above, data compressor 202 may determine the size of compressed data block 102. Furthermore, when it is determined that the size of compressed data block 102 is smaller than the granularity unit of output buffer 104, block address calculator 204 may assign the index of the granularity unit as the block address of the compressed data block smaller than the granularity unit.

[0134] The block address calculator 204 may also calculate the block address of the uncompressed data block 103 by determining the offset of the uncompressed data block 103 in the output buffer 104 and estimating the length of the uncompressed data block 103 in the output buffer 104 .

[0135] As described above, the length bits in the block address can be estimated. For example, the device 100 (e.g., the block address calculator 204 therein) can assign the bit pattern "111" to uncompressed data occupying 8 granularity units. In addition, the length of the compressed data block can use other bit patterns "000" to "110" (encoded numbers from 0 to 6). The length of the compressed data block can be calculated by adding 2 to the value represented by the bit pattern. For example, the bit pattern "000" represents a value of 0. Then, by adding 2 to the value 0 (i.e., 0+2=2), the present invention encodes the length of 2 granularity units for the compressed data block. Therefore, the length of the compressed data block can be represented by granularity units, ranging from 2 to 8 granularity units.

[0136] In addition, the compressed data length of 8 granularity units encoded by the bit pattern "110" represents the actual length of a compressed data block less than 8KB (i.e. <8KB), rounded to 8 granularity units. Therefore, it is possible that there are compressed data blocks of up to 8KB-256 bytes. This value is used as the upper limit of the compressed data size because, for example, if all blocks are compressed to 8KB-256 bytes and are not further packed into chunks by alignment, then a chunk may include 129 compressed data blocks, which is 1 block more than 128 uncompressed data blocks. In some embodiments, the present invention may require 1 bit per block for garbage collection, rather than 1 bit per granularity unit in conventional devices.

[0137] The device 100 may also optionally include a block header generator 205. The block header generator 205 may also be used to generate a block header for each compressed data block 102, wherein the block header indicates the offset of the compressed data block relative to the start of the granularity unit. In addition, the block header generator 205 may also generate a block tail header for each compressed data block 102, wherein the tail header indicates the offset of the last byte of the compressed data block relative to the start of the last granularity unit of the compressed data block.

[0138] As described above, the data compressor 203 may compact the compressed data block 102 and the uncompressed data block 103. The data compressor may compact the compressed data block 102 into the output buffer 104 starting from the first predefined area in the output buffer 104. The first predefined area may be the beginning of the output buffer 104, and the compressed data block 102 may be compacted in sequence from the beginning of the output buffer 104 toward the end of the output buffer 104. In addition, the compressed data block 102 may be compacted into the output buffer 104 without gaps and / or independently of the granularity unit of the output buffer 104. In other words, the compaction may be performed without aligning the granularity unit of the output buffer 104. Such packing may eliminate wasted space, etc.

[0139] The data compressor may also start from the second predefined area and compact the uncompressed data blocks 103 into the output buffer 104. The second predefined area may be the end of the output buffer 104, and the uncompressed data blocks 103 may be compacted in sequence from the end of the output buffer 104 toward the beginning of the output buffer 104. In addition, the uncompressed data blocks 103 may be compacted into the output buffer 104 according to the granularity unit of the output buffer 104. For example, the uncompressed data blocks 103 may be packed into the output buffer so that the uncompressed data blocks 103 are naturally aligned to 1KB (i.e., the total size is 8KB).

[0140] The device 100 may also optionally include a separate output buffer 206. The separate output buffer 206 may be used for compaction of small compressed data blocks, such as compressed data blocks that are smaller than the granularity unit of the output buffer 104. As described above, the data compressor 202 may determine the size of the compressed data block. In addition, when it is determined that the size of the compressed data block is smaller than the granularity unit of the output buffer 104, the device (e.g., the data compressor 203) may copy and add the compressed data to the separate output buffer 206. The separate output buffer 206 is associated with compressed data blocks that are smaller than the granularity unit, and the granularity unit represents the granularity of the memory 208 and / or the output buffer 104.

[0141] The device 100 may also optionally include a memory 208. The memory 208 may include volatile memory, non-volatile memory, or the like.

[0142] The device 100 may also optionally include a writing module 207. The writing module may be used to write the compressed data blocks 102 and the uncompressed data blocks 103 into the memory 208. In addition, the compressed data blocks 102 and the uncompressed data blocks 103 compacted into the output buffer 104 (and / or the separate output buffer 206) are written into the memory 208.

[0143] The device 100 may also optionally include a reading module 209. The reading module 209 may be used to read the compressed data block 102 and the uncompressed data block 103 from the memory 208. In addition, the reading module 209 may be used to read the compressed data block 102 and the uncompressed data block 103 according to the identification number of the output buffer, the size of the corresponding block, and the offset of the corresponding block relative to the start of the output buffer.

[0144] Figure 3 Schematic diagram of compacting compressed data blocks 102 and uncompressed data blocks 103 into 1MB chunks 104 (output buffer) for various embodiments. Without limiting the present invention, 1MB chunks 104 are used below to compact compressed data blocks 102 and uncompressed data blocks 103 into chunks 104.

[0145] The device 100 obtains a set of input data blocks, wherein the set includes compressed data blocks 102 such as B0, B1, B3, B4, B5 and Bn to be compacted into a chunk 104 and uncompressed data blocks 103 such as B2 and B6.

[0146] The compressed data blocks 102 (i.e., B0, B1, B3, B4, B5, and Bn) are grouped together at the beginning of a chunk 104. The compressed data blocks 102 are compacted from the left (i.e., the beginning of the chunk) toward the end of the chunk 104. The compressed data blocks 102 are packed without any alignment, so there is no wasted space between any two adjacent compressed data blocks 102.

[0147] Furthermore, uncompressed data chunks 103 of sizes such as 8KB, 16KB (and / or 32KB), etc. are grouped together at the end of a chunk 104. The uncompressed data chunks 103 are packed from the right (i.e., the end of the chunk) toward the beginning of the chunk 104. The uncompressed data chunks 103 are packed by natural alignment of chunk sizes such as 8KB.

[0148] In the case where the chunk 104 is full, the compressed data chunk 102 is adjacent to the uncompressed data chunk 103 , etc., there may be (a small amount of) wasted space 301 between the compressed data chunk 102 and the uncompressed data chunk 103 .

[0149] exist Figure 3 In the embodiment of the present invention, a wasted space 301 is formed between Bn and B6. Wasted space occurs when another compressed or uncompressed data block cannot be added. In this case, the size of the wasted space can be in the range of 1 byte to 8KB-1 byte.

[0150] Figure 4 A schematic diagram of calculating a block address and a block header 401 is provided for various embodiments.

[0151] As described above, one or more of several types of metadata may be used, including the block address of the compressed data block, the leading header 401, and the trailing header 402 (shown as white rectangles).

[0152] For example, a chunk address may include a chunk ID, a chunk offset (eg, an offset relative to the start of a chunk), and a chunk length. In addition, the chunk offset and the chunk length may be represented using a granularity unit.

[0153] The block address may occupy 64 bits. In addition, the block address may be stored in a read access memory (RAM) for faster access to blocks written to a storage medium (eg, a solid state device (SSD)).

[0154] The leading header 401 of the compressed data block may include a block byte offset of the compressed data block relative to the start of the granularity unit, etc. The leading header 401 of the compressed data block may be stored at the start of the first granularity unit of the compressed data block 102 in the chunk 104. The first granularity unit of the compressed data block may be an offset in the corresponding block address. The relationship between the block address and the block header is as follows: Figure 4 As shown, the position of the block header is marked with a white rectangle. In addition, 403 represents the actual starting position indicated by the header.

[0155] When the compressed data block 102 is the last compressed data block, the tail header 402 of the compressed data block may include a (e.g., appropriate) tail, etc. However, if there is another compressed data block compressed immediately after the compressed data block 102 is compressed, such as the next compressed data block, the tail header 402 of the previous compressed data block may be and / or may represent the leading header of the next compressed data block.

[0156] Each compressed data block 102 may include a leading header 401 and a trailing header 402. The leading header 401 may determine the offset of the first byte of the compressed data block 102, and the trailing header 402 may determine the byte offset following the last byte of the compressed data block.

[0157] exist Figure 4In the block B0, the leading header 401 is located in the first granularity unit, and the trailing header 402 is located in the second granularity unit.

[0158] The uncompressed data block may only include the block address.In addition, the uncompressed data block may not include a header, such as a leading header and / or a trailing header.

[0159] Figure 5 A schematic diagram of an example of a block header provided for various embodiments. Only compressed data uses a block header.

[0160] exist Figure 5 In FIG. 4 , H0 and H1 are the leading header 401 and the trailing header 402 of block B0, respectively. H1 and H2 are the leading header 401 and the trailing header 402 of block B1, respectively.

[0161] The H0 offset 501 field includes 4 bytes, indicating the actual starting position of the B0 block in the granularity unit. The H0 size 502 field includes 1532 bytes, which is the compressed size of the B0 block reported by a data compressor or the like.

[0162] The H1 offset field includes a 516-byte offset, indicating the starting position of the B1 block in the granularity unit. The H1 size field includes 2296 bytes, ie, the compressed size of the B1 block.

[0163] Figure 6 Schematic diagram of calculating block length provided for various embodiments.

[0164] The size (also referred to as length) of the compressed block data may be measured using bytes and / or granularity units.

[0165] In the present invention, the compressed data block 102 does not need to be aligned, so the compressed data block 102 can start at any position in the granularity unit 601, span one or more granularity units 601, and can also end at any position in the granularity unit 601.

[0166] Figure 6 Three compressed blocks 102 are shown, including blocks B0, B1, and B2, with corresponding headers 401 and 402 (also referred to as H0, H1, and H2). Block B0 spans granularity units 0 and 1. Block B1 spans granularity units 1, 2, 3, and 4 (the last granularity unit includes the tail header of B1). Block B2 spans granularity units 4 to 9 (the last granularity unit (not shown) includes the tail header of B2).

[0167] The present invention limits the size of the compressed data block to a range of 2 to 8 granularity units. In addition, the present invention can implement a compressed data block 102 with a size up to 8KB-256 bytes, which can also be represented using 8 granularity units.

[0168] Figure 7 A schematic diagram of wasted space 301 formed between two compressed data blocks 102 provided for various embodiments.

[0169] For example, when the size of (eg, at least one) compressed data block 102 is smaller than a granularity unit (eg, smaller than 1 KB), wasted space 301 may be formed between two compressed data blocks 102. Figure 7 In the embodiment of FIG. 1 , a wasted space 301 is formed between block B1 and block B2 .

[0170] In addition, for example, wasted space 301 may be generated in the following 3 situations:

[0171] 1. The last part of the compressed data block B0 is less than 1KB;

[0172] 2. CR is very high, causing the next data block B1 to be compressed to less than 1KB;

[0173] 3. The last part of B0 is put into the same granularity unit together with the whole block B1, so wasted space may remain in this granularity unit (block B2 with the corresponding header will start from the next granularity unit).

[0174] Figure 8 A schematic diagram of two wasted spaces 301 formed between three compressed data blocks 102 provided for various embodiments.

[0175] Figure 8 It is shown that when the compression ratio (CR) is very high (e.g., CR=9:1), wasted space 301 is formed. In some embodiments, two consecutive blocks can be compressed to less than 1KB, thus wasted space 301 is formed. Figure 8 In the embodiment, the block B1 with the corresponding header starts from the next granularity unit, and therefore, a wasted space 301 is formed between two consecutive granularity units.

[0176] Fig. 9 A schematic diagram of wasted space 301 formed between two compressed data blocks 102 provided for various embodiments.

[0177] Fig. 9 It is shown that wasted space 301 is formed when there are not enough bytes for the tail header in the last granularity unit of the compressed data block.

[0178] For example, assume that the block header occupies 4 bytes. The size of the last part of the compressed data block B0 is greater than 1020 bytes and less than 1024 bytes. Therefore, a wasted space 301 in the range of 1 to 3 bytes is formed here. In addition, the last granularity unit header is placed in the next granularity unit, so the size of the compressed data block 102 increases by one granularity unit. In addition, the next compressed data block of B1 may not exist, so the last granularity unit only includes the tail header. In this case, the wasted space formed is 1020 bytes plus 1 to 3 bytes of the previous granularity unit. This situation may exist for each chunk 104.

[0179] Fig.10 Schematic diagram of placing headers 401, 402 of several small compressed data blocks 102 for various embodiments.

[0180] In some embodiments, several (e.g., two or more) headers may be grouped together and may also be placed before corresponding blocks. In addition, this may further eliminate and / or reduce wasted space.

[0181] exist Fig.10 In the example, headers 401, 402 of several small compressed data blocks 102 are placed together at the beginning of the output buffer 104. In addition, 1001 represents the starting unit indicated by the chunk ID and offset, and the chunk header is obtained by indexing, and 1002 represents the actual starting position of the chunk indicated by the header in the unit, as described above.

[0182] Fig.11 A schematic diagram of a method 1100 for compacting compressed data blocks 102 and uncompressed data blocks 103 into an output buffer 104 is provided in accordance with an embodiment of the present invention.

[0183] The method 1100 comprises step 1: obtaining 1101 a set of input data blocks 101 , wherein the set comprises at least one of a compressed data block 102 and an uncompressed data block 103 .

[0184] The method 1100 further comprises step 2: starting from a first predefined area in the output buffer 104 , compacting 1102 the compressed data blocks 102 into the output buffer 104 , such that the compressed data blocks 102 are compacted in sequence.

[0185] The method 1100 further comprises step 3: starting from a second predefined area in the output buffer 104 , compacting 1103 the uncompressed data blocks 103 into the output buffer 104 , so that the uncompressed data blocks 103 are compacted in sequence.

[0186] Fig.12 A schematic diagram of a method 1200 of writing an input data block to a memory is provided for various embodiments.

[0187] At 1201 , the device 100 acquires an input data block and starts writing the acquired input data block into a memory.

[0188] At 1202, the device 100 compresses the obtained input data block according to a predefined compression ratio.

[0189] At 1203, the device 100 determines whether the input data block is compressible. When it is determined that the input data block is compressible, the device 100 further determines the size of the compressed data block and then executes step 1204. However, when it is determined that the input data block is not compressible (i.e., compression fails), the device 100 executes step 1208.

[0190] At 1204, the device 100 calculates a chunk length based on a compressed size and an offset in a current unit of a chunk.

[0191] In addition, the device determines the upper limit of the compressed block size according to the result of [output buffer size / (1+output buffer size / input block size)-2×header size]. For example, when the output buffer is 1MB, the input block is 8KB, and the header size is 4, the device determines the upper limit to be 8120 bytes. In the above formula, since each compressed block includes a leading header and a trailing header, the header size is multiplied by 2.

[0192] In addition, the device 100 adjusts the upper limit by the number of bytes in the last granular unit occupied by the previous compressed data block, so that the length of the subsequent compressed data block in the granular unit should be less than or equal to the length bit (i.e., 1≤length bit). For the case where the length bit is 3, the length of the compressed data block represented by the granular unit is 8 (i.e., less than or equal to 1<3).

[0193] For example, assume that the last granularity unit of the previous compressed data block (together with its tail header) occupies 15 bytes. The device 100 adjusts the upper limit by subtracting 15 so that the total limit on the size of the compressed data block expressed in granularity units is equal to or less than the granularity unit (1≤length bits).

[0194] In addition, the device 100 compares the adjusted upper limit with the remaining available space in the output buffer. When it is determined that there is enough available space, the device 100 continues to adjust the upper limit. Otherwise, the device 100 replaces the upper limit value with the size of the remaining available space in the output buffer and then executes step 1205.

[0195] At 1205, the device 100 places the first portion of the block into a chunk from the offset of the current unit to the last unit of the block.

[0196] The device 100 stores the compressed data block with the leading header and the trailing header in the output buffer from the starting position. For example, the device 100 increases the current pointer of the compressed data block by the size of the compressed data block, and increases the leading header and the trailing header to write the next uncompressed data block.

[0197] Furthermore, when the size of the compressed data block is smaller than one granularity unit, the device copies and adds the compressed data block to a separate buffer corresponding to the small compressed data block, as described above.

[0198] In addition, when the separate cache is full, for example, the separate cache includes two or more small compressed data blocks, the device 100 flushes its content (ie, the content of the separate cache) to the output cache and executes step 1209 to calculate the block address of the compressed data block.

[0199] At 1206, the device 100 generates headers in the first unit and the last unit of the block.

[0200] In addition, for example, when the first granularity unit for storing the compressed data block is empty, the device 100 writes a leading header of the compressed data block. The leading header includes the actual starting position of the compressed data block, as described above. For example, for a header size of 4 bytes, the actual starting position of the compressed data block is 4.

[0201] As described above, the device 100 writes the compressed data blocks in all granularity units except the last granularity unit into the output buffer. In the last granularity unit, the device 100 first writes the tail header. The tail header includes the actual starting position of the available space after the compressed data block. The tail header can be the leading header of the next compressed data block.

[0202] In addition, the last granularity unit of the compressed data block may include zero or more bytes of the tail header and the compressed data block. In addition, the size of the compressed data block to be written to the last granularity unit may not allow the tail header to be written. For example, the size of the granularity unit is 1024 bytes, the header size is 4 bytes, and the size of the compressed data block to be written to the last granularity unit is 1021 or 1022 or 1023 bytes. In this case, the compressed data block can add 0 bytes to reach the size of the granularity unit. In addition, the tail header can be written at the beginning of the next granularity unit.

[0203] At 1207, the device 100 places the second portion of the block into the chunk, and the device 100 places the remaining bytes starting from the last unit offset of the chunk.

[0204] In addition, the device 100 may also write zero or more bytes of the compressed data block into the last granularity unit.

[0205] At 1208, the device 100 places the uncompressed data block at the end of the chunk before the previously packed uncompressed data block.

[0206] For example, the device 100 determines that compression fails, for example, the size of the block is greater than or equal to the adjusted upper limit. In addition, the device 100 stores the original block (i.e., the uncompressed data block) in the output buffer, storing from the end to the beginning. In addition, the device 100 subtracts the original block size from the current pointer of the uncompressed data block to prepare to write the uncompressed data block.

[0207] In addition, when there is not enough available space in the output buffer to store the uncompressed data block, the device 100 flushes the output buffer to initialize all data, and further stores the uncompressed data block in the newly initialized output buffer.

[0208] Alternatively, if a flush occurs in the output buffer, for example, because the upper limit of available space is reached, as described above, the device 100 flushes the output buffer, initializes all data, and further recompresses the input data block.

[0209] At 1209, the device 100 generates a block address.

[0210] At 1210, the device 100 writes the block and finishes writing the block to the memory.

[0211] As described above, the device 100 calculates the block addresses of the compressed data blocks and / or the uncompressed data blocks.

[0212] The block address includes bit fields such as offset and length, which are calculated as follows.

[0213] The device 100 calculates the offset of the compressed data block or the uncompressed data block in the output buffer. For example, if the output buffer size is 1MB and the size of the uncompressed data block is 8KB, the device 100 stores 128 uncompressed data blocks in the output buffer. In addition, the device 100 allocates at least two granularity units for each compressed data block because each compressed data block requires 2 headers. Therefore, the device 100 can store 512 compressed data blocks in the output buffer. The device 100 also uses 9 bits to encode the offset of any one of the 512 compressed data blocks. For the 1KB granularity unit, the device 100 uses 10 bits as the minimum addressable object for offset addressing, and 9 bits are 1 bit less than 10 bits.

[0214] In addition, the device 100 calculates the length of the compressed data block and / or the uncompressed data block in the output buffer. For example, when the output buffer is 1MB, the data block is 8KB, and the granularity unit is 1KB, the device 100 needs 3 bits of length. As described above, the bit value 111 represents an uncompressed data block with a length equal to 8 granularity units or 8KB. In addition, the bit value in the range of 000 to 011 represents the size of the compressed data block with a length in the range of 2 to 8 granularity units or 2KB to 8KB.

[0215] In the case of a separate output buffer, the device 100 uses the index of the small compressed data block in one granularity unit. For example, the size of the output buffer is 1MB, the size of the data block is 8KB, the size of the granularity unit is 1KB, and each granularity unit has 3 small blocks. The device 100 needs a 2-bit index. An index value of 0 (bit field value 00) indicates that there is no small compressed data block.

[0216] In addition, the block address of the uncompressed data block includes the actual offset of the original block in the chunk, and the length is equal to [(1<<length bit)–1]. For example, for the case where the length bit is 3, the block address length of the uncompressed data block is 7 (i.e., (1<3)–1=7). However, the original length of the granularity unit is 8 (i.e., (1<3)=8). In addition, the block address may include an index of a small compressed data block in a granularity unit, as described above. Index values ​​in the range 1 to 3 are used for 1 to 3 small compressed data blocks, respectively.

[0217] In addition, the compressed data block offset of the block address of the compressed data block is the offset of the first granularity unit occupied by the compressed data block. In addition, when the compressed block data occupies one or less granularity units, the device 100 requires a length of at least 2 because a leading header and a trailing header are required.

[0218] Moreover, the length of the compressed data block in the block address is the total number of granularity units occupied by the compressed data block minus 2, taking into account the adjusted upper limit of the compressed data block size. When the device 100 determines that the difference between the current pointer of the uncompressed data block and the compressed data block is at least two granularity units, the device 100 considers using the next input data block. Otherwise, the device 10 flushes the output buffer and initializes all data.

[0219] Fig.13 Schematic diagram of a method for reading compressed data blocks and uncompressed data blocks from a memory provided for various embodiments.

[0220] At 1301, the device 100 begins to execute step 1: reading a data block from a memory. The device 100 initiates a block read using a block address.

[0221] At 1302, the device 100 parses the chunk address and derives the chunk ID, offset, and length, as described above.

[0222] The device 100 uses a chunk identifier (chunk ID) to locate the start position of a chunk on the storage medium. Then, the device 100 uses the chunk address offset to find the position of the data chunk in the chunk.

[0223] At 1303, the device 100 reads data from the chunk according to the chunk ID, offset, and length.

[0224] The device 100 adjusts the length of a data block (which may be a compressed data block or an uncompressed data block), and may read the data block as a plurality of continuous granularity units.

[0225] For example, when the length of the data block encoded by the corresponding bit field of the block address is equal to [(1<<length bit)–1], the data block is an uncompressed data block. Then, the device 100 reads [(1<<length bit)] granularity units. However, when the length of the data block encoded by the corresponding bit field of the block address is in the range of 0 to [(1<<length bit)–2], the data block is a compressed data block, and then, the device 100 reads [(length+2)] granularity units.

[0226] At 1304 , the device 100 determines whether the data block is an uncompressed block. Furthermore, when it is determined that the data block is an uncompressed data block, the device 100 executes step 1311 . However, when it is determined that the data block is not an uncompressed data block, the device 100 executes step 1305 .

[0227] At 1305, the device 100 parses the block start header, and the device 100 derives the size and byte offset of the compressed data block in the first granularity unit.

[0228] The device 100 parses the leading header and the trailing header of the compressed data block to determine the starting position of the compressed data block in the first read granularity unit and the ending position of the compressed data block in the last granularity unit.

[0229] At 1306, the device 100 allocates a temporary buffer. The temporary buffer corresponds to the derived block size.

[0230] At 1307, the device 100 copies the first portion of the block to the output buffer from the offset to the last unit.

[0231] At 1308, the device 100 skips the trailing header in the last unit of the block.

[0232] At 1309 , the device 100 copies the second portion of the block to the output buffer starting from the last granularity unit.

[0233] In addition, when the last granularity unit includes a compressed data block, the device 100 moves the compressed data block to the "proper position" (ie, the beginning of the granularity unit) to cover the tail header and further make the compressed data blocks continuous.

[0234] At 1310, the device 100 decompresses the unpacked blocks.

[0235] The device 100 decompresses the consecutive compressed data and restores the compressed data to the original block contents.

[0236] At 1311, the device 100 reads the decompressed data block and ends the reading process.

[0237] Table 1 shows the benchmark results of compressing and packaging two data sets based on the method disclosed in the prior art, and Table 2 shows the benchmark results of compressing and packaging two data sets based on the device disclosed in the present invention (or the method run by the device).

[0238] The benchmark results of the prior art solution (a prototype implementation of the solution proposed in the prior art) and the standalone application prototype of the present invention were obtained according to the following conditions.

[0239] 1. Implemented chunk packaging and chunk address calculation

[0240] 2. Data read from memory - IO (disk) overhead is not considered

[0241] 3. Repeat (packing, compression), (unpacking, decompression) 100 times, average speed

[0242] 4. Block compression / decompression is done in a single thread

[0243] 5. LZ-like algorithms are used as the default compressor and scheme packaging scheme

[0244] In addition, the environment for obtaining the benchmark results is Intel(R) Xeon(R) CPU E5-2670 0@2.60GHz, where the benchmark was compiled using GCC 6.3-O3 and Linux kernel 3.19.

[0245] Table 1: Benchmark results of compressing and packing two datasets based on the methods disclosed in the prior art

[0246]

[0247]

[0248] Table 2: Benchmark results of compressing and packing two data sets based on the device (or method executed by the device) disclosed in the present invention

[0249]

[0250] Compared with the typical online compression and packaging scheme in the method disclosed in the prior art, the benchmark verification result of the present invention improves the data reduction ratio by 13% to 27%, and the rate by 6% to 42%. In addition, the present invention enables the selected LZ4-type algorithm to achieve near-ideal waste elimination.

[0251] The present invention considers the actual compressed data size in the packing scheme and minimizes the waste of SSD memory compared with the typical online compression packing scheme in the prior art. For example, for the Oracle DB data set, the memory waste (% input) is reduced from 9.4% to 0.1%, and for the standard data set, the memory waste is reduced from 6.1% to 0.1%.

[0252] The invention has been described in conjunction with various embodiments and implementations as examples. However, other variations will be understood and implemented by those skilled in the art in practicing the claimed invention, based on a study of the drawings, the present invention and the appended claims. In the claims and in the specification, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may fulfil the functions of several entities or items recited in the claims. The enumeration of certain measures in mutually different dependent claims does not indicate that a combination of these measures cannot be used in an advantageous implementation.

Claims

1. A device (100) for compacting compressed data blocks (102) and uncompressed data blocks (103) into an output buffer (104), characterized in that: The device (100) is used for: Acquire a set of input data blocks (101), wherein the set includes at least one of a compressed data block (102) and an uncompressed data block (103); Starting from a first predefined area in the output buffer (104), the compressed data blocks (102) are compressed into the output buffer (104) without gaps, so that the compressed data blocks (102) are compressed in sequence; Starting from a second predefined area in the output buffer (104), the uncompressed data blocks (103) are compacted into the output buffer (104), so that the uncompressed data blocks (103) are compacted in sequence; The step of compacting the compressed data block (102) into the output buffer (104) without gaps comprises: adjusting the upper limit of the length of the compressed data block (102) by the number of bytes in the last granularity unit (601) of the output buffer (104) occupied by the previous compressed data block (102), so that the length of the subsequent compressed data block (102) in the granularity unit (601) is less than or equal to the upper limit of the length.

2. The device (100) according to claim 1, characterized in that The device is also used to determine the byte upper limit of the compressed data block according to the size of the output buffer (104), the size of the compressed data block (102) to be compressed into the output buffer (104), and the header size of the compressed data block (102).

3. The device (100) according to claim 1 or 2, characterized in that The device is also used to: obtain an input data block (101), compress the obtained input data block (101) according to a predefined compression ratio, and determine the size of the compressed data block (102).

4. The device (100) according to claim 3, characterized in that When it is determined that the size of the compressed data block (102) is smaller than the granularity unit (601) of the output buffer (104), the compressed data is copied and added to a separate output buffer (206), wherein the separate output buffer (206) is associated with a compressed data block smaller than the granularity unit (601), and the granularity unit (601) represents the granularity of the memory (208).

5. The device (100) according to any one of claims 1 to 4, characterized in that The first predefined area is the beginning of the output buffer (104), and the compressed data blocks (102) are compressed in sequence from the beginning of the output buffer (104) toward the end of the output buffer (104).

6. The device (100) according to any one of claims 1 to 5, characterized in that The second predefined area is the end of the output buffer (104), and the uncompressed data blocks (103) are compressed in sequence from the end of the output buffer (104) toward the beginning of the output buffer (104).

7. The device (100) according to claim 6, characterized in that The uncompressed data block (103) is compacted into the output buffer (104) according to the granularity unit (601) of the output buffer (104).

8. The device (100) according to any one of claims 1 to 7, characterized in that The device (100) is further configured to calculate a block address of the compressed data block (102) by determining an offset (501) of the compressed data block (102) in the output buffer (104) and estimating a length (502) of the compressed data block (102) in the output buffer (104).

9. The device (100) according to claim 8, characterized in that The block address of the compressed data block (102) smaller than the granularity unit (601) corresponds to the index of the granularity unit (601).

10. The device (100) according to any one of claims 1 to 9, characterized in that The device (100) is further configured to calculate a block address of the uncompressed data block (103) by determining an offset of the uncompressed data block (103) in the output buffer (104) and estimating a length of the uncompressed data block (103) in the output buffer (104).

11. The device (100) according to any one of claims 1 to 10, characterized in that The device (100) is further configured to generate a block header (401) for each compressed data block, wherein the block header indicates an offset of the compressed data block relative to the start of the granularity unit (601).

12. The device (100) according to any one of claims 1 to 11, characterized in that The device (100) is further configured to generate a block tail header (402) for each compressed data block, wherein the block tail header indicates an offset of a last byte of the compressed data block relative to a start of a last granularity unit of the compressed data block.

13. The device (100) according to any one of claims 1 to 11, characterized in that The device (100) further comprises a writing module (207) for writing the compressed data block (102) and the uncompressed data block (103) into a memory (208).

14. The device (100) according to claim 13, characterized in that The compressed data block (102) and the uncompressed data block (103) compressed into the output buffer (104) are written into the memory (208).

15. The device (100) according to any one of claims 1 to 14, characterized in that The device (100) further comprises a reading module (209) for reading the compressed data block (102) and the uncompressed data block (103) from the memory (208).

16. The device (100) according to claim 15, characterized in that The reading module (209) is further configured to read the compressed data block (102) and the uncompressed data block (103) according to the identification number of the output buffer (104), the size of the corresponding block and the offset of the corresponding block relative to the start of the output buffer (104).

17. The device (100) according to any one of claims 1 to 16, characterized in that The memory (208) includes volatile memory or non-volatile memory.

18. A method (1100) for compacting compressed data blocks (102) and uncompressed data blocks (103) into an output buffer (104), characterized in that: The method (1100) comprises the following steps: Acquiring (1101) a set of input data blocks (101), wherein the set includes at least one of a compressed data block (102) and an uncompressed data block (103); Starting from a first predefined area in the output buffer (104), the compressed data blocks (102) are compacted (1102) into the output buffer (104) without gaps, so that the compressed data blocks (102) are compacted in sequence; Starting from a second predefined area in the output buffer (104), compacting (1103) the uncompressed data blocks (103) into the output buffer (104) so ​​that the uncompressed data blocks (103) are compacted in sequence; The step of compacting the compressed data block (102) into the output buffer (104) without gaps comprises: adjusting the upper limit of the length of the compressed data block (102) by the number of bytes in the last granularity unit (601) of the output buffer (104) occupied by the previous compressed data block (102), so that the length of the subsequent compressed data block (102) in the granularity unit (601) is less than or equal to the upper limit of the length.

Images