Hardware-friendly data compression method, system and device

Abstract

Systems, devices, and methods for compressing data are provided. An exemplary method may include interleaving one or more literal length fields with one or more literal fields to an output. The literal field may contain a first data segment that is copied verbatim to the output, and each of the one or more literal length fields may contain a value representing the length of a subsequent literal field. The method may also include determining a second data segment and a matching position that matches a previously copied verbatim data sequence, and writing the one or more matching length fields of the output and a matching position field containing the matching position to the output. The literal length field may contain the total length of the first data segment, and the matching length field may contain the total length of the second data segment.

Description

[0001] This application is a divisional application, parent application number: 201910972547.0, parent application date: October 10, 2019, parent application name: Hardware-friendly data compression method, system and device. Technical Field

[0002] The disclosure herein relates to data compression, and in particular to lossless compression. Background Art

[0003] Data compression is widely used in digital signal systems, such as communications and computing. For example, to save storage space or transmission time, computer data is often compressed into TAR or ZIP file formats. Over the years, a series of compression schemes, known as LZ77 and LZ78, have been developed based on the compression techniques originally published by Abraham Lempel and Jacob Ziv in 1977 and 1978. Some schemes in the LZ family favor compression ratio over speed, while others sacrifice compression ratio for higher speed. However, these various LZ compression techniques were developed for general-purpose computing systems, which typically use a general-purpose central processing unit (CPU) to perform data compression or decompression operations. Therefore, a more hardware-friendly compression technology, such as one suitable for field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), is needed. Summary of the Invention

[0004] The present disclosure provides systems and methods for lossless compression. Compressed data may include two types of data: literals and matches. A literal may refer to data that can be copied verbatim from input to output. A match may refer to data that matches a previously copied literal and may therefore be replaced with a reference to the previous copy. In one embodiment, a compressed data block may include: one or more literal length fields interleaved with one or more literal fields, one or more match length fields with a match position field inserted after an initial match length field, and zero or more repeated length fields.

[0005] An exemplary compression process can generate one or more literal length fields interleaved with one or more literal fields, such that once a literal length field and a literal field can be filled, the data to be copied verbatim can be output. Once a match for a data segment of a minimum match size is found, one or more match length fields can be generated, and the match position can be located and inserted into the match position field after the initial match length field. The compression process can further determine whether to repeat the matching data segment and record the number of times the matching data segment is repeated. Therefore, the repeat length fields can be combined to contain a total value indicating how many times the matching data segment can be repeated.

[0006] In one exemplary embodiment, a method for generating compressed data is provided. The method may include: interleaving one or more literal length fields with one or more literal fields to an output, each of the one or more literal length fields containing a portion of a first data segment copied verbatim from an input data stream to the output, each of the one or more literal length fields containing a value representing a length of a subsequent literal field, the combination of the one or more literal length fields containing a first total value representing a total length of the first data segment; determining a second data segment of the input data stream that matches a previously copied verbatim data sequence and a match position; and writing one or more match length fields and a match position field containing a value representing the match position to the output, the combination of the one or more match length fields containing a second total value representing a total length of the second data segment.

[0007] In another exemplary embodiment, an apparatus including an encoder is provided. The encoder may include: a data packer for generating one or more data segments from an input data stream, each of the one or more data segments having a length that is a minimum match length; a hash value calculator for generating a hash value for each of the one or more data segments; a hash table for tracking the one or more data segments using respective hash values; a match position searcher for searching, based on entries in the hash table, whether a data segment has a match in processed data; a match position merger for merging two or more data segments that have found a match in the processed data; and an output assembler. The output assembler can be configured to: interleave one or more literal length fields with one or more literal fields to the output, each of the one or more literal fields containing a portion of a first data segment copied verbatim from the input data stream to the output, each of the one or more literal length fields containing a value representing a length of a subsequent literal field, the combination of the one or more literal length fields containing a first total value representing a total length of the first data segment; determine a second data segment of the input data stream that matches a previously verbatim copied data sequence and a match position; and write one or more match length fields and a match position field containing a value representing a match position to the output, the combination of the one or more match length fields containing a second total value representing the total length of the second data segment.

[0008] In another exemplary embodiment, what is disclosed herein may also include a non-transitory machine-readable medium having executable instructions, wherein the executable instructions, when executed by a computer system, cause the computer system to: interleave one or more literal length fields with one or more literal fields to an output, each of the one or more literal length fields containing a portion of a first data segment copied verbatim from an input data stream to an output, each of the one or more literal length fields containing a value representing a length of a subsequent literal field, the combination of the one or more literal length fields containing a first total value representing a total length of the first data segment; determine a second data segment of the input data stream that matches a previously verbatim copied data sequence and a match position; and write one or more match length fields and a match position field containing a value representing the match position to the output, the combination of the one or more match length fields containing a second total value representing a total length of the second data segment. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A compressed data block according to an embodiment of the present disclosure is schematically shown.

[0010] Figure 2 An encoder according to an embodiment of the present disclosure is schematically illustrated.

[0011] Figure 3 A decoder according to an embodiment of the present disclosure is schematically illustrated.

[0012] Figure 4 A computing system according to an embodiment of the present disclosure is schematically illustrated.

[0013] Figure 5 is a flow chart of a compressed data flow according to an embodiment of the present disclosure.

[0014] Figure 6 is a flowchart of decompressing a compressed data stream according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0015] Now, the specific embodiments according to the present application will be described in detail with reference to the accompanying drawings. For consistency, the same elements in various drawings are represented by the same reference numerals.

[0016] The present disclosure provides a system and method for lossless compression. Compressed data may include two types of data: literals and matches. A literal may refer to data that can be copied verbatim from an input. A match may refer to data that matches a previously copied literal and may therefore be replaced with a reference to the previous copy. In one embodiment, a compressed data block may include: one or more literal length fields interleaved with one or more literal fields, one or more match length fields with a match position field inserted after an initial match length field, and zero or more repeated length fields.

[0017] An exemplary compression process can generate one or more literal length fields interleaved with one or more literal fields, such that once a literal length field reaches its maximum value and the corresponding literal field is filled, the data to be copied verbatim can be output. Once a match for a data segment of a minimum match size is found, one or more match length fields can be generated, and the match position can be located and inserted into the match position field after the initial match length field. The compression process can further determine whether to repeat the matching data segment and record the number of times the matching data segment is repeated. Therefore, the repeated length fields can be combined to include a total value representing the number of times the matching data segment is repeated.

[0018] Figure 1A compression format according to an embodiment of the present disclosure is schematically illustrated. This compression format can be represented in a compressed data block 100, which can be a compressed data unit generated by the compression process. Compressed data block 100 can include a first segment 120 and a second segment 126. The first segment 120 is for data segments copied verbatim from the input data stream, and the second segment 126 is for matched data segments. The first segment 120 can be referred to as the literal segment of compressed data block 100, and the second segment 126 can be referred to as the matching segment of compressed data block 100. In some embodiments, the data to be compressed can be processed as a byte stream, with each byte being 8 bits. Therefore, the literal segment 120 can contain the byte string copied from the input data stream and the length of the string to be copied verbatim. The second segment 126 can include information about byte strings that can be matched from previously processed data, eliminating the need to copy the byte string verbatim again. It should be noted that using bytes as the data unit for compression is merely an example, and various embodiments can implement the techniques disclosed herein for data units of other suitable sizes (e.g., 4 bits, 16 bits, 24 bits, 32 bits, or 64 bits).

[0019] In some embodiments, the data segments to be searched and matched may have a minimum match length. That is, for byte sequences that are less than the minimum match length, no matching operation needs to be performed. The minimum match length may be denoted as a minimum match. A typical minimum match may be four (4), but other numbers may be used in some embodiments. For illustrative purposes, the examples in this disclosure may use four as an example of a minimum match.

[0020] The first segment 120 can be referred to as a literal section and can include an initial literal section 122 and an extended literal section 124. The initial literal section 122 can include an initial literal length field 102 and an initial literal field 104. The initial literal length field 102 can store a value referred to as the initial literal length. The initial literal length field 102 can have a fixed bit width, and this fixed width can determine the maximum value of the initial literal length and the size of the initial literal field 104. The fixed width can be a plurality of bits, and the corresponding numerical value is a positive integer, which can be 2 but is typically greater than 2, for example, 3, 4, 5, etc. For illustrative purposes, in this disclosure, the initial literal length field 102 can use 3 bits as an example, so the initial literal length can range from 0 (all three bits are 0) to 7 (all three bits are 1). If the initial literal length is 0, no data is copied literally from the input data stream, and the initial literal field 104 can be empty. In one embodiment, when the initial literal length is 0 (“000” in binary), the initial literal field 104 may not exist in the compressed data block 100 .

[0021] If the initial literal length is a number "x" that can be greater than 0 (for example, x can be 1, 2, 3, 4, 5, 6, or 7), the initial literal field 104 can have a width of "x" bytes and contain "x" bytes of data copied verbatim from the input data stream. When the initial literal length is less than 7, the extended literal section 124 may not be required. Therefore, in one embodiment, when the value in the initial literal length field 102 is less than 7, the compressed data block 100 may not have an extended literal section 124. For example, if the byte string to be copied verbatim from the data stream can be five bytes represented as "ABCDE", the literal section 120 can be composed of an initial literal length field 102 with a value of "101" (for example, the binary number of the number 5) and an initial literal field 104 of five bytes "ABCDE". Because there is no more literal data to copy from the input stream, the literal segment 120 in this example does not have any extended literal length field 106 or extended literal field 108. Note that the byte strings in the examples of this disclosure can be any data, not necessarily characters. For example, the character symbols of "ABCDE" and any other example character strings can be used to represent different byte values ​​in the data stream.

[0022] The extended literal section 124 may include extended literal length fields 106.1 through 106.k and extended literal fields 108.1 through 108.k. Each of the extended literal length fields 106.1 through 106.k may also have a fixed bit width. In some embodiments, each of the extended literal length fields 106.1 through 106.k may have an equal fixed width. In some other embodiments, at least one of the extended literal length fields 106.1 through 106.k may have a fixed width that may be different from the other fields. Furthermore, in one embodiment, the fixed width of the extended literal length fields 106.1 through 106.k may be equal to the fixed width of the initial literal length field 102. However, in some other embodiments, the fixed width of the extended literal length fields 106.1 through 106.k may be different from the fixed width of the initial literal length field 102. Exemplary fixed widths of the extended literal length fields 106.1 to 106.k may be 8 bits, 16 bits, etc. For purposes of illustration, the present disclosure may use 4 bits as an example of the extended literal length fields 106.1 to 106.k. Thus, the value in any extended literal length field 106.1 to 106.k may range from 0 (all four bits are 0) to 15 (all four bits are 1).

[0023] When all bits of the initial literal length field 102 are 1, the initial literal length can be 7 (3-bit binary "111"), and at least the first extended literal length field 106.1 of the extended literal section 124 may be required. The functions of the extended literal length field 106.1 and the extended literal field 108.1 can be similar to the initial literal length field 102 and the initial literal field 104, in that the value in the extended literal length field 106.1 can indicate how many bytes can be contained in the extended literal field 108.1. When all bits of the extended literal length field 106.1 reach 1, the next extended literal length field 106.2 may be required (and the next extended literal field 108.2 may be required). When the extended literal length field 106.1 is 0, the extended literal field 108.1 can be empty. Thus, in one embodiment, when the value of the extended literal length field 106.1 is zero, the extended field 108.1 may not exist. For example, a byte string copied verbatim from a data stream may be 7 bytes represented as "ABCDEFG", and the literal segment 120 may include an initial literal length field 102 with 3 bits all set ("111"), an initial literal field 104 with seven bytes of "ABCDEFG", and an extended literal length field 106.1 with 4 bits all set ("0000") because there is no more literal data to copy from the input stream. The literal segment 120 in this example does not have any extended literal fields 108 or any further extended literal length fields 106 (other than the extended literal length field 106.1).

[0024] When the extended literal length field 106.1 is greater than zero, an extended literal field 108.1 may be required. For example, if the byte string copied verbatim from the data stream is 8 bytes represented as "ABCDEFGH", the literal segment 120 may include an initial literal length field 102 with 3 bits all set to 1 ("111"), an initial literal field 104 of ten bytes "ABCDEFG", an extended literal length field 106.1 with 3 bits set to 0 and 1 bit set to 1 ("0001"), and an extended literal field 108.1 of one byte "H". The literal segment 120 in this example does not have any more extended literal length fields 106 (other than the extended literal length field 106.1) or any more extended literal fields 108 (other than the extended literal field 108.1).

[0025] When all bits of the extended literal length field 106.1 reach 1, for example, for all 4 bits of binary "1111" and the value 15, an extended literal length field 106.2 may be needed, and the extended literal length field 108.2 may contain the next portion of the byte string copied from the input data stream. Therefore, the extended literal length fields 106.1 to 106.k and their accompanying extended literal fields 108.1 to 108.k can be added one by one after the previous literal field is filled and all bits of the previous literal length field reach 1. That is, the extended literal length field 106.k only exists when all bits of the previous extended literal length field 106.k-1 are 1. The number k can represent the total number of extended literal fields added after the initial literal field 104. In some embodiments, the number of bits in the literal length field and / or the extended literal length field may be determined by hardware limitations, such as, but not limited to, the size of the memory allocated for compression operations, the size of the microprocessor's cache, and the like.

[0026] The second segment 126 may include an initial match section 128, an extended match length section 130, and a repeated match section 132. The initial match section 128 and the extended match length section 130 may be collectively referred to as a match section, and the repeated match section 132 may be simply referred to as a repeated section. The initial match section 128 may include an initial match length field 110 and a match position field 112. The initial match length field 110 may have a fixed bit width to indicate the length of the byte sequence that can be matched in previously processed data. An exemplary fixed width of the initial match length field 110 may be 2 bits, 3 bits, 4 bits, 5 bits, or any suitable integer number of bits. For illustrative purposes, the examples in the present invention may use 3 bits as an example of the initial match length field 110. The match position field 112 may contain a position value indicating where the matched byte sequence may be located. In one embodiment, the position value may be an absolute index from the beginning of the input data stream. Large files may be divided into multiple blocks, for example, 4KB blocks, and each block may be referred to as an input stream during processing. In another embodiment, the position value can be an offset from the current position in the input data stream to the data that has been processed. For example, if the position value is the number "y" as an offset, the matched byte string can be counted back y bytes from the current position. An exemplary width of the match position field 112 can be 8 bits, 10 bits, 12 bits, or any suitable bit value. For illustrative purposes, the examples in this disclosure use 12 bits as an example of the match position field 112.

[0027] The extended match length segment 130 may include one or more extended match length fields 114.1 to 114.m. Each of the extended match length fields 114.1 to 114.m may have a fixed bit width. The fixed widths of the extended match length fields 114.1 to 114.m may be equal in some embodiments and unequal in other embodiments. Exemplary fixed widths of the extended match length fields 114.1 to 114.m may be 2 bits, 3 bits, 4 bits, 5 bits, or any suitable bit value. For illustrative purposes, the examples in this disclosure may use 8 bits as an example of extended match length fields 114.1 to 114.m. When the length of the matched byte sequence reaches the maximum value of the initial match length field 110 (e.g., all bits are 1), the first extended match length field 114.1 may be used. And when the length of the matched byte sequence reaches the sum of the maximum values ​​contained in the initial match length field 110 (e.g., all bits are 1) and the first extended match length field 114.1 (e.g., all bits are 1), the next extended match length field 114.2 can be used. Therefore, the extended match length field 114.m can be used only when all bits in the previous match length fields (e.g., the initial match length field 110 and the extended match length field 114.1 to the extended match length field 114.m-1) are 1.

[0028] In one embodiment, because the match is at least for a sequence of minimum match length, the actual match length of the byte sequence may be the total value contained in the initial match length field 110 and all extended match length fields 114 plus the minimum match and minus 1 (e.g., actual match length = total value + minimum match - 1). For example: when the actual match length of the byte sequence is 4, the value in the initial match length field 110 may be 1 (1); when the actual match length of the byte sequence is 6, the value in the initial match length field 110 may be 3 (3); when the actual match length of the byte sequence is 15, the value in the initial match length field 110 may be 7 (7) (e.g., binary "111"), and the value in the extended match length field 114.1 may be 5 (5) (e.g., binary "00000101"); and so on.

[0029] For example, the input data stream may be "ABCDEFGHIJKABCDE". In one embodiment, the first 11 bytes "ABCDEFGHIJK" may be divided into two parts, "ABCDEFG" and "HIJK", where the first part "ABCDEFG" is copied to the initial literal field 104, and the second part "HIJK" is copied to the extended literal field 108.1. The initial literal length field 102 may include 3 bits that are all 1 (e.g., "111"). The extended literal length field 106.1 may contain a value of 4 (e.g., "0100"). The next 5 bytes of "ABCDE" may be a matching sequence of the first five 5 bytes. Considering the minimum match, the initial match length field 110 may contain a value of 2 (e.g., binary "010"). In an embodiment using an absolute index as the match position, the match position field 112 may contain a value of 0 (for the case of indexing from zero) or a value of 1 (for the case of indexing from one). In an embodiment that uses an offset as the match position, the match position field 112 may contain a value of eleven (11) because the matched sequence begins by counting eleven bytes from the current position.

[0030] Repeat match section 132 may include an initial repeat length field 116 and extended repeat length fields 118.1 through 118.n. The matched data may be repeated to form a repeating pattern. Initial repeat length field 116 may have a fixed width equal to or greater than 1, for example, 1 bit, 2 bits, 3 bits, etc. Initial repeat length field 116 and extended repeat length fields 118.1 through 118.n may contain a value indicating the number of repetitions. Each of repeat length fields 118.1 through 118.n may have a fixed bit width. The fixed widths of repeat length fields 118.1 through 118.n may be equal in some embodiments and unequal in other embodiments. Exemplary widths of repeat length fields 118.1 through 118.n may be 2 bits, 3 bits, 4 bits, or any suitable number of bits. For illustrative purposes, the initial repeat length 116 and extended repeat length fields 118.1 through 118.n in this disclosure may use 2 bits as an example. Similar to the extended literal length field and the extended match length field, the extended repeat length fields 118.2 through 118.n may be used only if all bits in the preceding repeat length field are ones.

[0031] For example, the data to be compressed may be 100 bytes of repeated zeros. With a minimum match of 4, the first four zero bytes may be copied to the initial literal field 104 as a string of zeros. The next four zero bytes may be a matching sequence of four zero bytes. The second four zero bytes that match (the third of the 100 zero bytes) and all subsequent matching four zero bytes may form a repeating pattern with 23 repetitions. In this example, the initial repeat length field 116 may have a value of 3 (e.g., binary "11"), the extended repeat length fields 118.1 to 118.6 may contain a value of 3 (e.g., binary "11"), and the extended repeat length field 118.8 may contain a value of 2 (e.g., binary "10").

[0032] The "repeating pattern" matching described above can be used to alleviate the problem of overlapping matches in traditional LZ series compression algorithms. In one embodiment, it can be assumed that long repeating patterns rarely exist, and therefore, the repeating match segment 132 can only exist when the value of the initial match length field 110 is less than its maximum value (e.g., less than a value of 1). Considering the minimum match, for the example where the width of the initial match length field 110 is 3 bits and the maximum value is 7, this can effectively set the maximum length of the sequence of repeating patterns to 9 bytes (e.g., 6 + minimum batch number - 1). In other words, in this embodiment, the initial repeat length field 116 and the first extended match length field 114.1 can be mutually exclusive.

[0033] The compressed data block 100 may be generated by various implementations in different embodiments. Figure 2An encoder 200 according to one embodiment of the present disclosure is schematically illustrated. Encoder 200 can be configured to receive an input data stream and generate compressed data blocks for the input data, and thus can be referred to as a compressor or compression engine. Encoder 200 can include a data packer 202, a hash value calculator 204, a hash table updater 206, a hash table 212, a matching position searcher 208, and a matching position merger 210. Data packer 202 can receive an input data stream and process the received data by generating a short sequence of multiple data segments. Each data segment can include a minimum matching byte. The first data segment can begin at the starting byte of the data stream, and all other data segments can begin at the second byte of the previous data segment. For example, the data stream "ABCDEFG" can be packed into a sequence of 4-byte data segments of "ABCD," "BCDE," "CDEF," and "DEFG." In some embodiments, the data to be compressed can optionally be divided into multiple blocks, for example, 4K byte (4KB) blocks. The position information can also be a position within a block. For example, an absolute index can be a position from the beginning of the block. In an embodiment in which data is processed in units of blocks, except that the first data fragment may start at the start byte of the current block, each data fragment may start at the second byte of the previous data fragment.

[0034] The data packager 202 may have a data processing bandwidth that determines how many data segments can be packaged in a batch. For example, if the data packager 202 has a data processing bandwidth of 4, the data packager 202 may package four 4-byte data segments in a batch. Thus, the data sequence "ABCDEFGHABCDE" may be packaged into "ABCD," "BCDE," "CDEF," and "DEFG" in the first four 4-byte sequences and output from the data packager 202. The next four 4-byte sequences "EFGH," "FGHA," "GHAB," and "HABC" may be packaged in the data packager 202 in a second batch and output from the data packager 202. In another embodiment, the data packager 202 may have a data processing bandwidth of 8, and eight 4-byte sequences may be generated and output from the data packager 202 in batches.

[0035] The hash value calculator 204 may be configured to generate a hash value for the short sequence generated by the data packager 202. For example, if the data packager 202 generates four short sequences and sends them in batches to the hash value calculator 202, the hash value calculator 204 may generate four hash values ​​in batches, each hash value being generated for a data segment of a short sequence. The four-byte sequence used to generate the hash value may also be referred to as a four-byte sequence corresponding to the hash value.

[0036] The hash value generated by the hash value calculator 204 can be sent to a hash table updater 206, where previously processed hash values ​​and their corresponding 4-byte data fragment sequences (and positions) can be tracked in a hash table 212. The hash table 212 can be implemented with a suitable data structure, such as, but not limited to, a dictionary, in which the hash value can be used as an index to an entry in the dictionary. Each entry pointed to by the index can have a field for storing the corresponding 4-byte sequence and another field for storing the position. In one embodiment, the position can be an absolute index of the corresponding 4-byte sequence in the 4K byte block currently being processed, for example, starting with the first byte in the 4K byte block being 0 (positioned with a 0 index) or 1 (positioned with a 1 index).

[0037] In some embodiments, when multiple hash values ​​and their corresponding 4-byte sequences are received from the hash value calculator 204, the hash table updater 206 may process the hash values ​​in the order of the corresponding 4-byte sequences' positions in the input stream. For example, the first four 4-byte sequences of the input data stream "ABCDEFGHABCDE" may be "ABCD," "BCDE," "CDEF," and "DEFG," and the 4-byte sequence "ABCD" may be processed first, followed by "BCDE," "CDEF," and finally "DEFG."

[0038] If the hash table 212 does not have an entry pointed to by the newly received hash value by the hash table updater 206, the hash table updater 206 may attempt to update the hash table 212 using the newly received hash value as the index of the new entry, and store the corresponding 4-byte sequence and the position of the 4-byte sequence in the data stream. In some embodiments, if there is no room for the new entry, the existing entry may be cleared and the new entry may occupy the space of the cleared entry. In one embodiment, the existing entry may be selected using a cache replacement algorithm, such as, but not limited to, a least recently used (LRU) algorithm.

[0039] If hash table 212 already has an entry pointed to by the newly received hash value, it can be a match or a hash value collision. A match may mean that the 4-byte sequence corresponding to the newly received hash value is in the already processed data and has been found. That is, the 4-byte sequence has been copied to the output of the literal string and its location is in the hash table. A hash value collision may mean that the 4-byte sequence corresponding to the newly received hash value is not in the already processed data, but the already processed data has another 4-byte sequence with a hash value equal to the newly received hash value.

[0040] If the newly received hash value is an existing index of an entry in the hash table, the matching position searcher 208 may compare the 4-byte sequence corresponding to the newly received hash value with the 4-byte sequence stored in the entry of the hash table pointed to by the index. If the two 4-byte sequences are equal, there is a match, and the 4-byte sequence and the position stored in the entry may be passed to the matching position merger 210.

[0041] If the two 4-byte sequences are different, then there is a hash collision and a collision resolution technique can be used. The collision resolution technique can be one of the techniques known in the art or any suitable technique developed in the future. For example, one way to handle hash collisions is to add another entry to the hash table so that one index can point to two or more entries in the hash table, and each time a newly received hash value is an existing index, each 4-byte sequence in the entry pointed to by the index can be checked to determine whether there is a match or a hash collision.

[0042] In one embodiment, the matching location searcher 208 may receive a 4-byte sequence for generating a hash value and a fixed width (eg, 12 bits) representing the length of a window (eg, the current 4KB data block) for searching for matching locations in the window.

[0043] The match position merger 210 can be configured to generate match length and match position information. In some embodiments, the match position merger 210 can be configured to determine whether there are multiple matching 4-byte sequences, whose positions are obtained from the hash table in a continuous order, and determine whether the length of the match can be the length of a single matching data sequence. For example, if two consecutive 4-byte sequences "ABCD" and "BCDE" are matched, the match position merger 210 can be configured to merge these two sequences into a matching sequence "ABCDE" whose matching length is 5 bytes, and the matching position is the position of the matching sequence "ABCD". In one embodiment, the position of the matching sequence can be obtained directly from the hash table entry, which can be the absolute index of the matching sequence in the input data stream, and the absolute index of the matching sequence can be placed in the match position field as the matching position. In another embodiment, the value to be placed in the match position field as the matching position can be the offset obtained by subtracting the absolute index from the value of the current position.

[0044] In some embodiments, hash table 212 may have a limited size, and not all previously processed hash values ​​and their corresponding 4-byte sequences may be stored in hash table 212. Therefore, there may be missed matches in the hash value and the corresponding 4-byte sequence. In one embodiment, match position merger 210 may be configured to recover the missed match pattern if it is located in the input stream between two match sequences separated by a maximum index distance of a minimum match. When the index difference is equal to the difference between the two match positions, and the difference is a minimum match or less, all sequences between two consecutive matches may be merged into one match sequence.

[0045] For example, if five consecutive sequences "ABCD," "BCDE," "CDEF," "DEFG," and "EFGH" and their hash values ​​are being processed, and the string "ABCDEFGH" is in the processed data, then ideally, all five sequences should be tracked in hash table 212 and matched. However, due to size limitations, hash table 212 may have entries for "ABCD" and "EFGH," but no entries for "BCDE," "CDEF," or "DEFG." Assuming that the matched sequence "ABCD" has an absolute index of "N" stored in the hash table, the matched sequence "EFGH" may have an absolute index of "N+4" stored in the hash table. The index distance between "EFGH" and "ABCD" is 4 and is equal to the difference between the matched sequences, and the index difference is a minimum match, thus satisfying the minimum match or less condition. In this case, match position merger 210 may determine that the matched pattern is "ABCDEFGH," the match length is 8 ("8"), and the match position is the position of the "ABCD" sequence (e.g., "N" or the offset of the current index minus N).

[0046] In another example, assume that the "ABCDEDEFG" sequence is in the processed data, and the position values ​​of "ABCD", "BCDE", and "DEFG" in the hash table are L, L+1, and L+5, and L can be an index value greater than or equal to zero. Now assume that another "ABCDEFG" sequence is packed into "ABCD" with a position value of M, "BCDE" with a position value of M+1, "CDEF" with a position value of M+2, and "DEFG" with a position value of M+3. The 4-byte sequences "ABCD", "BCDE", and "DEFG" can be matched with the corresponding hash table entries. "BCDE" and "DEFG" are consecutive matches, but the difference in matching position values ​​is 4 and the difference in index is 2. Therefore, the sequence between "BCDE" and "DEFG" in the "ABCDEFG" sequence is not a missed match, and the two sequences cannot be combined into a matching sequence.

[0047] Therefore, in one embodiment, the match position merger 210 may check the index distance and the match position difference between two consecutive matches before determining the match length and the match position.

[0048] In one embodiment, the matching position merger 210 may receive a 4-byte sequence corresponding to a hash value and a fixed width (eg, 12 bits) representing a window length for merging matching sequences within the window.

[0049] Output assembler 214 can be configured to generate compressed data blocks for an input data stream by first interleaving one or more literal length fields with one or more literal fields. Each of the one or more literal fields can contain a portion of a first data segment that is copied verbatim from the input stream to the output, and each of the one or more literal length fields can contain a value representing the length of a subsequent literal field. The combination of the one or more literal length fields can contain a total value representing the total length of the first data segment.

[0050] The output assembler 214 can also be configured to handle any repeating patterns. If there are any consecutive matching sequences with the same matching position and matching length, a repeating pattern can be detected, and the output assembler 214 can generate the initial repeat length field 116 and any extended repeat length fields 118, wherein the total value in these fields represents the number of repetitions.

[0051] In one embodiment, all 4-byte sequences of a data segment may be passed to the output assembler 214, regardless of whether any of the sequences have been matched with previously processed data or determined to be recoverable missed matches. When a sequence does not match in previously processed data and cannot be recovered as a missed match, it may be determined that the first byte in the 4-byte sequence needs to be copied verbatim to the compressed data block. For example, if "ABCD" does not match and cannot be recovered as a missed match, then "A" may be copied verbatim to the compressed data block because "BCD" may be the first three bytes of the next 4-byte sequence "BCDE" that may match the previously processed data segment. If "BCDE" does not match and cannot be recovered as a missed match, then "B" may be copied verbatim to the compressed data block. When the number of bytes to be copied verbatim reaches the maximum value that can be represented by the initial literal length (e.g., 7), the initial literal length field 102 can be complete and the initial literal field 104 can be filled with the maximum number of bytes represented by the initial literal length, and any subsequent bytes can be filled into one or more extended literal fields up to the maximum value indicated by each corresponding extended literal length field.

[0052] When one or more 4-byte data fragment sequences can be matched, the interleaving of one or more literal length fields and one or more literal fields can be completed. Match position merger 210 can pass matching position (e.g., absolute index or offset) and matching length to output assembler 214. Output assembler 214 can write one or more matching length fields and the matching position field comprising the matching position to output. The combination of one or more matching length fields can include the total value representing the total length of the matching data segment.

[0053] In one embodiment, once the data arrives at the output assembler 214, the literal length field, literal field, match length field, position field, repeat field, and repeat length field are generated and output immediately. Therefore, there is no need to wait for the entire compressed data block to be assembled before outputting the compressed data from the encoder 200.

[0054] Figure 3 A decoder 300 according to one embodiment of the present disclosure is schematically shown. The decoder 300 may include an input buffer 302, a segment extractor 304, a memory command generator 306, a decompression buffer 308, and an output buffer 310. The decoder 300 may process an input stream of compressed data to generate decompressed data. The decoder 300 may receive compressed data blocks at the input buffer 302. In one embodiment, the input buffer 302 may have a fixed width to facilitate subsequent processing. For example, if the minimum match is 4, the input buffer may have an 8-byte width so that 8 bytes can be temporarily stored therein. Two 8-byte blocks can be combined to form 16 bytes for input to the segment extractor 304.

[0055] At the segment extractor 304, literal segments, matching segments, and repeating segments can be extracted from the input compressed data. For example, based on the format of the compressed data block 100. The segment extractor 304 can be configured to read the content of the input data, distinguish different segments, and separate literal segments, matching segments, and repeating segments. The separated literal segments, matching segments, and repeating segments can be passed to the memory command generator 306.

[0056] In the memory command generator 306, read and write commands can be generated. For example, from the extracted literal segment, a literal string can be generated by combining the literal string in the initial literal field and all extended literal fields (if any). The memory command generator 306 can generate one or more write commands to write the literal string to the decompression buffer 308. The decompression buffer 308 can send the literal string to the output buffer 310 and also save a copy of the literal string. From the extracted matching segment, the matching position can be obtained, and the matching length can be determined by combining the values ​​in the initial matching length field and all extended matching length fields (if any), and a read command can be issued by the memory command generator 306 to read a matching byte sequence of the matching length from the decompressed data at the matching position and write the matching byte sequence to the decompression buffer 308. Furthermore, if there is a repeated segment, the number of repetitions can be determined by combining the values ​​of all repeated length fields, and the repeated data sequence can be reconstructed in the decompression buffer 308. In one embodiment, the decompressed data reconstructed from the matching segments and the repeated segments may be sent to the output buffer 310 .

[0057] In one embodiment, the decompression buffer 308 may include a decompression buffer and a decompression memory. The decompression memory can be used as a memory for storing processed data so that a matching pattern can be obtained from the processed data. The decompression buffer can be used as a temporary storage for literal strings before the literal strings are sent to the output buffer 310, and can also be used as a working buffer to combine the matching and repeating patterns obtained from the decompression memory before any matching patterns and repeating patterns can be restored to a decompressed (or pre-compressed) format. In one embodiment, the input of the output buffer 310 can have a width that matches the width of the data bus coupled to the decoder 300.

[0058] In some embodiments, the throughput of the decoder may be limited by the bandwidth of the decompression memory. For example, the decompression memory may be implemented as static random access memory (SRAM), and read or write operations to the decompression memory may be constrained by the read or write throughput limitations of the SRAM. A cache can be used to alleviate bottlenecks. For example, a cache implemented as a D flip-flop (DFF) can improve throughput.

[0059] In one embodiment, either or both of the encoder 200 and the decoder 300 may be implemented in a field programmable gate array (FPGA) or an application specific IC (ASIC). In another embodiment, the encoder 200 and the decoder 300 may be implemented by a computer system 400, such as Figure 4The computer system 400 may include a computer processor 402 and a memory 404. The memory 404 may include executable instructions so that the processor 402 can execute and perform all functions of the encoder 200 and the decoder 300.

[0060] Figure 5 A process 500 for compressing a data stream according to an embodiment of the present disclosure is shown. The process 500 can generate one or more compressed data blocks 100 from an input data stream. In block 502, one or more literal length fields are interleaved with one or more literal fields to an output. The one or more literal length fields can contain a first data segment copied verbatim from the input stream to the output. Each of the one or more literal fields can contain a portion of the first data segment copied verbatim from the input data stream to the output, and each of the one or more literal length fields can contain a value representing a subsequent literal length field. The combination of the one or more literal length fields can contain a first total value representing the total length of the first data segment;

[0061] In block 504, a second data segment may be determined to match the previously copied data sequence, and a matching position may be determined. The second data segment may be a matching data segment having a minimum matching length, or may include data merged from multiple matching data segments. In block 506, one or more match length fields and a match position field may be written to the output. The match position field may contain a value indicating the matching position, and the combination of the one or more match length fields may contain a second total value indicating the total length of the second data segment.

[0062] Figure 6 A process 600 for decompressing a compressed data stream according to an embodiment of the present disclosure is shown. The compressed data stream may include one or more compressed data blocks 100. In box 602, one or more compressed data blocks may be received. For example, an input buffer may receive one or more compressed data blocks to be decompressed. In box 604, different segments may be extracted from the compressed data blocks. For example, literal segments, matching segments, and repeating segments may be extracted from the compressed data blocks 100. In box 606, memory commands may be generated. For example, a write command may be generated to write a literal string in a literal segment to a decompression buffer, and a read command may be generated to read a matching byte sequence with a matching length from a matching position and send the matching sequence to the decompression buffer. In box 608, the decompressed data may be written to an output buffer. In box 610, the decompressed data may be output from the decompressor. For example, the decompressed data may be output to a PCIe bus.

[0063] In some embodiments, interleaving the literal length field and the literal field can make the encoding process more hardware-friendly because there is no need to store the potentially long literal data strings in a buffer. Therefore, an exemplary encoder can output immediately after processing the literal data fragments, and the size of the data fragments can be determined by design choices (e.g., throughput requirements or hardware resources). For example, if long string data is copied word for word from the input stream to the output stream, it can be output immediately after processing each data fragment of the long string. In addition, in some traditional LZ series compression techniques such as LZ4, the first field (e.g., token) in the compressed data block includes an initial character length and an initial match length. Such a token requires that the data copied word for word be cached and output processing is started only after the first match is found. In contrast, an embodiment of the encoder according to the present disclosure can start outputting literal data immediately (once the literal length field reaches its maximum value) without waiting for the first match to be found.

[0064] Furthermore, most conventional LZ-series compression techniques, such as LZ4, rely on overlapping matches during decoding, where portions of the matching data segments are contained within the undecompressed data. Overlapping matches make the data time-dependent because the decoder cannot use the undecompressed data when a match is found, and the decoder must wait for data to become available before outputting the decompressed data. Embodiments of the present disclosure can use a repeat pattern determination, a repeat field, and one or more repeat length fields to avoid the need for overlapping matches during decoding, because the repeated data segments have already been decoded and are time-independent.

[0065] Additionally, in some embodiments, the matching position may be an absolute index in the search window (eg, 4KB), and the matching position does not need to be calculated based on an offset.

[0066] In one exemplary embodiment, a method for generating compressed data is provided. The method may include: interleaving one or more literal length fields with one or more literal fields to an output, each of the one or more literal length fields containing a portion of a first data segment copied verbatim from an input data stream to the output, each of the one or more literal length fields containing a value representing a length of a subsequent literal field, the one or more literal length fields in combination containing a first total value representing a total length of the first data segment; determining a second data segment of the input data stream that matches a previously copied verbatim data sequence and a match position; and writing one or more match length fields and a match position field containing a value representing the match position to the output, the one or more match length fields in combination containing a second total value representing a total length of the second data segment.

[0067] In one embodiment, the method may further include: packaging the input data stream into a plurality of data segments, calculating a hash value for each of the plurality of data segments, and determining whether each of the plurality of data segments has a match in previously processed data based on the hash value. The length of each data segment may be a minimum matching length, a first data segment of the plurality of data segments starts from a starting data unit of the input data stream, and each data segment after the first data segment starts from a second data unit of the previous data segment;

[0068] In one embodiment, the second data segment may be obtained by merging two or more matching data segments, and the matching position thereof is the position of the first matching data segment among the two or more matching data segments.

[0069] In one embodiment, the second data segment may be obtained by merging at least two matching data segments separated by an index difference value, wherein the index difference value is equal to the matching position difference value, and the index difference value is the minimum matching length or less than the minimum matching length.

[0070] In one embodiment, the method may also keep track of previously processed data in a hash table.

[0071] In one embodiment, the hash table is a dictionary having hash values ​​as indexes, and each entry pointed to by the index may store a data segment corresponding to the hash value and a location of the data segment.

[0072] In one embodiment, the method may further include: determining whether a repetitive pattern exists; and generating one or more repetition length fields containing a third total value representing the number of repetitions of the repetitive pattern.

[0073] In one embodiment, the matching position may be an absolute index from the beginning of the input data stream.

[0074] In one embodiment, the matching position may be an offset from the current position of the data segment being processed.

[0075] In another exemplary embodiment, an encoder apparatus is provided. The encoder includes: a data packer for generating one or more data segments from an input data stream, each of the one or more data segments having a minimum matching length; a hash value calculator for generating a hash value for each of the one or more data segments; a hash table for tracking the one or more data segments using respective hash values; a matching position searcher for searching whether a data segment has a match in processed data based on entries in the hash table; a matching position merger for merging two or more data segments that have found a match in the processed data; and an output assembler configured to: combine one or more literal length fields with one or more literal literal length fields; The method comprises: interleaving segments of the input data stream to an output, each of the one or more literal length fields containing a portion of a first data segment copied verbatim from the input data stream to the output, each of the one or more literal length fields containing a value representing a length of a subsequent literal field, the combination of the one or more literal length fields containing a first total value representing a total length of the first data segment; determining a second data segment of the input data stream that matches a previously copied verbatim data sequence and a match position; and writing one or more match length fields and a match position field containing a value representing a match position to the output, the combination of the one or more match length fields containing a second total value representing a total length of the second data segment.

[0076] In one embodiment, a first data fragment of the one or more data fragments starts at a starting data unit of the input data stream, and each data fragment after the first data fragment starts at a second data unit of the previous data fragment.

[0077] In one embodiment, the second data segment may be obtained by merging two or more matching data segments, and the matching position is the position of a first matching data segment of the two or more matching data segments.

[0078] In one embodiment, the second data segment may be obtained by merging at least two matching data segments separated by an index difference value equal to a matching position difference value, and the index difference value is the minimum matching length or less than the minimum matching length.

[0079] In one embodiment, the hash table is a dictionary having hash values ​​as indexes, and each entry pointed to by the index stores a data segment corresponding to the hash value and a location of the data segment.

[0080] In one embodiment, the output assembler may be further configured to: determine whether a repeating pattern exists; and generate one or more repeat length fields containing a third total value representing the number of repetitions of the repeating pattern.

[0081] In one embodiment, the matching position is an absolute index from the beginning of the input data stream.

[0082] In one embodiment, the matching position is an offset from the current position of the data segment being processed.

[0083] In one embodiment, the device may also include: a decoder, which includes: a segment extractor, configured to extract the one or more literal length fields containing one or more literal fields as a literal segment, and extract the one or more matching length fields and matching position fields as a matching segment; a decompression buffer for storing decompressed data; and a memory command generator, configured to write the literal string obtained from the literal segment into the decompression buffer and read the matching data sequence from the decompression buffer.

[0084] In another exemplary embodiment, what is disclosed herein may also include a non-transitory machine-readable medium having executable instructions, wherein the executable instructions, when executed by a computer system, cause the computer system to: interleave one or more literal length fields with one or more literal fields to an output, each of the one or more literal length fields containing a portion of a first data segment copied verbatim from an input data stream to an output, each of the one or more literal length fields containing a value representing a length of a subsequent literal field, the combination of the one or more literal length fields containing a first total value representing a total length of the first data segment; determine a second data segment of the input data stream that matches a previously verbatim copied data sequence and a match position; and write one or more match length fields and a match position field containing a value representing the match position to the output, the combination of the one or more match length fields containing a second total value representing a total length of the second data segment.

[0085] In one embodiment, the executable instructions, when executed by the processor, further cause the computer system to: determine whether a repeating pattern exists; and generate one or more repeat length fields containing a third total value representing a number of repetitions of the repeating pattern.

[0086] Any disclosed methods and operations can be implemented as computer-executable instructions (e.g., software code for the operations described herein) stored on one or more computer-readable storage media (e.g., non-transitory computer-readable media, such as one or more optical media disks, volatile memory components (such as dynamic random access memory (DRAM) or static random access memory (SRAM)), or non-volatile memory components (e.g., hard drives)) and executed on a device controller (e.g., firmware executed by an ASIC). Any computer-executable instructions for implementing the disclosed techniques and any data created and used during implementation of the disclosed embodiments can be stored on one or more computer-readable media (e.g., non-transitory computer-readable media).

[0087] Although various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and not limitation, with the true scope and spirit being indicated by the following claims.

Claims

1. A data compression method, characterized in that: include: Packing an input data stream into a plurality of data segments, each data segment having a length of a minimum matching length, wherein a first data segment of the plurality of data segments starts from a starting data unit of the input data stream, and each data segment after the first data segment starts from a second data unit of the previous data segment; calculating a hash value for each of the plurality of data segments; determining, based on respective hash values, that two data segments of the plurality of data segments are two consecutive matches with two previously processed data segments; merging the two data segments into a matching sequence; Output a compressed data stream containing a matching segment, wherein the matching segment includes one or more matching length fields and a matching position field, wherein the combination of the one or more matching length fields contains the total length of the matching sequence, and the matching position field contains a value representing the matching position of the matching sequence.

2. The method according to claim 1, characterized in that The two data segments include two consecutive data sequences.

3. The method according to claim 1, characterized in that The two data segments are separated by an index difference value that is equal to a position difference value of matching positions of the two previously processed data segments, and the index difference value is less than or equal to the minimum matching length.

4. The method according to claim 1, wherein The matching sequence is obtained by merging three or more matching data segments, any pair of consecutive matching data segments contains a continuous data sequence, or is separated by an index difference equal to the position difference of the matching positions of a pair of previously processed data segments, and the index difference is less than or equal to the minimum matching length.

5. The method according to claim 1, characterized in that The matching position is the position of the first matching data segment among all matching data segments merged into the matching sequence.

6. The method according to claim 5, characterized in that The matching position is an absolute index from the beginning of the input data stream.

7. The method according to claim 5, characterized in that The matching position is an offset relative to the current position of the data segment being processed.

8. The method according to claim 1, characterized in that This also involves keeping track of the hash values ​​of the processed data in a hash table.

9. The method according to claim 8, characterized in that The hash table is a dictionary that uses the hash value of the processed data as an index, and each entry pointed to by the index stores a data segment corresponding to the hash value and a position of the data segment in the input data stream.

10. The method according to claim 1, characterized in that Also includes Identify the presence of recurring patterns; and One or more repetition length fields are generated in the compressed data stream.

11. The method according to claim 1, wherein The compressed data stream also includes a literal segment, wherein the literal segment includes one or more literal length fields, each literal length field is followed by a corresponding subsequent literal field, each of the one or more literal length fields contains a value representing the length of the corresponding subsequent literal field, wherein the combination of the one or more literal fields includes data copied from one of the multiple data fragments, the one data fragment has no match in the processed data, or data copied from a continuous series of data fragments of the multiple data fragments, the series of data fragments has no match in the processed data.

12. A data compression device, characterized in that: include: Encoder, including: a data packetizer configured to generate a plurality of data segments from an input data stream, each data segment having a length of a minimum matching length, wherein a first data segment of the plurality of data segments starts from a starting data unit of the input data stream, and each data segment after the first data segment starts from a second data unit of the previous data segment; a hash calculator configured to calculate a hash value for each of the plurality of data segments; a matching location searcher configured to search for a match between the data segment and the processed data by comparing the hash value of the data segment with the hash value of the processed data; a matching position merge configured to merge two data segments of the plurality of data segments into a matching sequence when the two data segments are two consecutive matches with two previously processed data segments based on the respective hash values; and An output assembler is configured to output a compressed data stream containing a matching segment, wherein the matching segment includes one or more matching length fields and a matching position field, wherein a combination of the one or more matching length fields contains a total length of the matching sequence, and the matching position field contains a value indicating a matching position of the matching sequence.

13. The device according to claim 12, characterized in that The two data segments include two consecutive data sequences.

14. The device according to claim 12, characterized in that The two data segments are separated by an index difference value that is equal to a position difference value of matching positions of the two previously processed data segments, and the index difference value is less than or equal to the minimum matching length.

15. The device according to claim 12, characterized in that The matching sequence is obtained by merging three or more matching data segments, any pair of consecutive matching data segments contains a continuous data sequence, or is separated by an index difference equal to the position difference of the matching positions of a pair of previously processed data segments, and the index difference is less than or equal to the minimum matching length.

16. The device according to claim 15, characterized in that The matching position is an absolute index from the beginning of the input data stream.

17. The device according to claim 15, characterized in that The matching position is an offset relative to the current position of the data segment being processed.

18. The device according to claim 12, characterized in that Also included is a hash table that keeps track of the hash values ​​of processed data.

19. The device according to claim 18, characterized in that The hash table is a dictionary with hash values ​​as indexes, and each entry pointed to by the index stores a data segment corresponding to the hash value and a position of the data segment.

20. A non-transitory machine-readable medium having executable instructions, characterized in that When the executable instructions are executed by a processor of a computer system, the computer system is caused to perform the following operations: Packing an input data stream into a plurality of data segments, each data segment having a length of a minimum matching length, wherein a first data segment of the plurality of data segments starts from a starting data unit of the input data stream, and each data segment after the first data segment starts from a second data unit of the previous data segment; calculating a hash value for each of the plurality of data segments; determining, based on respective hash values, that two data segments of the plurality of data segments are two consecutive matches with two previously processed data segments; merging the two data segments into a matching sequence; Output a compressed data stream containing a matching segment, wherein the matching segment includes one or more matching length fields and a matching position field, wherein the combination of the one or more matching length fields contains the total length of the matching sequence, and the matching position field contains a value representing the matching position of the matching sequence.

Images