Using subliterals for DEFLATE compression with reduced-complexity Huffman coding

Abstract

Receive a text element with multiple bits. Divide the multiple bits in the text element into a first sub-text including a first group of bits and a second sub-text including a second group of bits. Encode the first sub-text using a first Huffman code tree to obtain a first sub-text codeword; encode the second sub-text using a second Huffman code tree to obtain a second sub-text codeword. Output encoded data including information associated with the first Huffman code tree, information associated with the second Huffman code tree, the first sub-text codeword, and the second sub-text codeword.

Description

Background Art

[0001] The DEFLATE file format is commonly used in a variety of protocols and file formats, such as ZIP, gzip, Hypertext Transfer Protocol (HTTP), etc., and uses Huffman coding and other coding techniques, such as LZ77, to compress data. Huffman coding is a variable-length coding technique in which a code table is constructed or otherwise structured so that more commonly occurring symbols are encoded as shorter codewords, and less commonly occurring codewords are encoded as longer codewords. It would be desirable if new Huffman coding techniques were developed that were less complex, required less hardware, and / or reduced latency (i.e., processing time). BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments of the invention are disclosed in the following detailed description and accompanying drawings.

[0003] Figure 1 is a flow chart illustrating an embodiment of an encoding process for compressing data, including compressing data by dividing a word element into sub-words.

[0004] Figure 2 is a flow chart illustrating an embodiment of a decompression and / or decoding process for combining sub-words into a word.

[0005] Figure 3 is a diagram illustrating an example of a typical DEFLATE block.

[0006] Figure 4 is a diagram illustrating an example of a typical dynamic weight table associated with words, lengths, and ends of blocks.

[0007] Figure 5 is a diagram illustrating an example of a typical codeword tree associated with literals, lengths, and block ends.

[0008] Figure 6 is a diagram illustrating an embodiment of a DEFLATE block having sub-literal elements.

[0009] Figure 7 is a diagram illustrating an embodiment of a dynamically generated weight table associated with a first sub-word, a length, and a block end.

[0010] Figure 8 is a diagram illustrating an embodiment of a weight table associated with a second sub-word.

[0011] Fig. 9 is a diagram illustrating an embodiment of a dynamic weight table associated with distance.

[0012] Fig.10is a diagram illustrating an embodiment of a codeword tree associated with a first subword, a length, and a block end.

[0013] Fig.11 is a diagram illustrating an embodiment of a DEFLATE encoder that divides a word into sub-words.

[0014] Fig.12 is a diagram illustrating an embodiment of a DEFLATE decoder in which sub-words are combined into words.

[0015] Fig.13 is a diagram illustrating an embodiment of a DEFLATE encoder and decoder using sub-literals in a communications application.

[0016] Fig.14 is a diagram illustrating an embodiment of a DEFLATE encoder and decoder using sub-literals in a storage application. DETAILED DESCRIPTION

[0017] The present invention can be implemented in many ways, including as a process; an apparatus; a system; a material composition; a computer program product contained on a computer-readable storage medium; and / or a processor, such as a processor configured to execute instructions stored on and / or provided by a memory coupled to the processor. In this specification, these implementations or any other form that the present invention can take can be referred to as techniques. In general, within the scope of the present invention, the order of the steps of the disclosed process can be changed. Unless otherwise stated, components such as processors or memories described as being configured to perform tasks can be implemented as general components that are temporarily configured to perform the task at a given time, or as specific components that are manufactured to perform the task. As used herein, the term "processor" refers to one or more devices, circuits, and / or processing cores that are configured to process data (such as computer program instructions).

[0018] A detailed description of one or more embodiments of the present invention is provided below together with the accompanying drawings that illustrate the principles of the present invention. The present invention is described in conjunction with such embodiments, but the present invention is not limited to any embodiment. The scope of the present invention is limited only by the claims, and the present invention encompasses numerous replacements, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the present invention. These details are provided for illustrative purposes, and the present invention can be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical materials known in the technical field related to the present invention are not described in detail so that the present invention will not be unnecessarily obscured.

[0019] Various embodiments of a new Huffman encoding technique that can be used in DEFLATE or a similar file format are described herein. First, an encoding embodiment of the technique is described in which the bits in a word element (e.g., an ASCII character) are divided into a first sub-word (e.g., including a first set of bits from the word element) and a second sub-word (e.g., including the remaining bits from the word element); then, the two sub-words are encoded using corresponding Huffman code trees. Then, a corresponding exemplary decoding process is described. As will be described in more detail below, this can be faster (e.g., at both the encoder and the decoder) and / or less complex (e.g., at the encoder, when constructing the Huffman code tree used to encode the sub-words).

[0020] Figure 1 is a flow chart illustrating an embodiment of an encoding process for compressing data, including compressing data by dividing a literal element into sub-literals. In some embodiments, the exemplary process is performed by an encoder implemented in hardware, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In some embodiments, the exemplary process is performed by an encoder implemented in software, such as a computer program product embodied in a non-transitory computer-readable medium and including computer instructions.

[0021] At 100, a literal element having a plurality of bits is received. For example, assume that an encoder receives input data to be encoded. In the case of the DEFLATE file format, one encoding paradigm is to LZ77 encode a given data sequence by referencing an earlier occurring copy of that data sequence (e.g., within the input data to be encoded). However, this technique will not work for the first segment or block of data to be processed because nothing exists before that first segment or block of data. Thus, the first segment or block of data (e.g., within the input data) is encoded as a literal element (e.g., as an ASCII character or symbol) that does not reference an earlier occurring copy of a repeating sequence. Literal elements may also be used if an earlier occurring copy is not located or otherwise identified.

[0022] At 102, a plurality of bits in the word element are divided into a first sub-word including a first group of bits and a second sub-word including a second group of bits. In some examples described below, the word element has 8 bits, and the 3 most significant bits form the first sub-word, and the 5 least significant bits form the second sub-word. It is noted that this division is merely exemplary, and as will be described in more detail below, some bit divisions at step 102 may provide better performance and / or advantages than other bit divisions.

[0023] At 104, the first subword is encoded using the first Huffman code tree to obtain a first subword codeword. At 106, the second subword is encoded using the second Huffman code tree to obtain a second subword codeword. As will be described in more detail below, in some embodiments, the first Huffman code tree is also used to encode a length (e.g., associated with an LZ77 length and backward distance pair) and / or other characters or symbols (e.g., end of symbol).

[0024] At 108, encoded data including information associated with the first Huffman code tree, information associated with the second Huffman code tree, the first sub-word codeword, and the second sub-word codeword is output. For example, a header or beginning portion of a DEFLATE block or message includes information associated with the first Huffman code tree and the second Huffman code tree, which allows a decoder to know how to decode compressed data included in the body or later portions of the DEFLATE block. The compressed data included in the body of the DEFLATE block may in turn include the first sub-word codeword and the second sub-word codeword. Some examples are described in more detail below.

[0025] In some applications, encoded (i.e., compressed) data is stored in a storage system or transmitted over a communication channel. For example, by compressing data before storage, the amount of available storage can be increased. Similarly, if data is compressed before it is exchanged over a communication channel, the bandwidth consumed and / or transmission time can be reduced.

[0026] Figure 2 is a flow chart illustrating an embodiment of a decompression and / or decoding process for combining sub-words into a word. For example, Figure 2 The decoding process can be used to use Figure 1 The data encoded by the process is decoded. Similar to the above, the exemplary decoding process can be implemented in a variety of ways (such as hardware or software).

[0027] At 200, encoded data including information associated with a first Huffman code tree, information associated with a second Huffman code tree, a first sub-word codeword, and a second sub-word codeword is received. For example, each DEFLATE packet may use a different Huffman code tree (e.g., to improve overall compression), and the information associated with the first Huffman code tree and the second Huffman code tree lets the decoder know how to decode the compressed data in the body of the DEFLATE packet.

[0028] At 202, a first subword codeword is decoded using a first Huffman code tree to obtain a first subword including a first set of bits. At 204, a second subword codeword is decoded using a second Huffman code tree to obtain a second subword including a second set of bits.

[0029] At 206, the first group of bits in the first subword and the second group of bits in the second subword are combined to obtain a word element. For example, if the first group of bits includes the most (least) significant bits of the word element and the second group of bits includes the least (most) significant bits of the word element, the two groups of bits may be concatenated together.

[0030] The text element is output at 208. For example, there may be some higher level application or user that is waiting for the data to be decoded and / or decompressed.

[0031] In order to better and / or more clearly illustrate the Huffman coding technique(s) described herein and their associated benefits, it may be helpful to first illustrate a typical Huffman code tree in a typical DEFLATE block, which is more similar to (for example) Figure 1 and 2 The Huffman encoding techniques described in are more complex and / or slower. The following figures describe examples of more complex and / or slower Huffman encoding techniques in a typical DEFLATE block.

[0032] Figure 3 is a diagram illustrating an example of a typical DEFLATE block. In the example shown, a DEFLATE block (300) has two general parts: (1) a Huffman code tree (302 and 304) that describes how compressed data (306) was encoded at the encoder and is used by the decoder to decode the compressed data; and (2) compressed data (306). In some applications, there may be a stream or sequence of DEFLATE blocks, and if desired, each DEFLATE block is encoded using a different set of Huffman code trees (e.g., if different DEFLATE blocks contain different symbols and / or have different frequencies of occurrence, the compression rate may be improved by using different Huffman code trees optimized for each DEFLATE block).

[0033] The compressed data (306) includes a sequence of elements, one of which is a pointer element, including a length element (308) and a distance element (310). Conceptually, a pointer element is a reference to a repeating pattern or sequence and / or an earlier occurring copy thereof, and is represented by a length and (backward) distance pair (e.g., a length element (308) and a distance element (310)). For context, DEFLATE (also) uses LZ77 encoding, and this reference to an earlier copy using a length and (backward) distance pair is from LZ77. For convenience and brevity, the term "distance" is understood to mean the backward distance as used herein.

[0034] In contrast to pointer elements, literal elements (312) are memoryless and do not require storage of previous symbols, bytes, etc. In the examples described herein, literal elements include ASCII symbols or values.

[0035] The third element in the compressed data (306) in this example is the end-of-block element (314). The end-of-block is a special symbol that indicates the end of the compressed data (306), and therefore also the end of the DEFLATE block (300).

[0036] Each of the Huffman code trees (302 and 304) is associated with and / or used to encode a different element or portion of an element. Huffman code tree A (302) is associated with literals, lengths, and block ends, and thus the length (308), literal elements (312), and block end elements (314) are encoded using that code tree. Huffman code tree B (304) is associated with distances, and thus the distance element (310) is encoded using that Huffman code tree.

[0037] Conceptually, each of the Huffman code trees (302 and 304) can be considered to consist of two separate parts: a dynamic weight table and its corresponding codeword tree, which is different for each DEFLATE block. The following figures describe examples of dynamically generated weight tables and their corresponding codeword trees associated with Huffman code tree A (302).

[0038] Figure 4 is a diagram illustrating an example of a typical dynamic weight table associated with words, lengths, and block ends. For example, Figure 3 The Huffman code tree A (302) in can use the exemplary dynamic weight table shown here. In some embodiments, in addition to and / or as an alternative to the entries shown here, the dynamic weight table includes other entries.

[0039] In this example, Huffman Code Tree A (HCTA) symbols HCTA(0) through HCTA(255) are associated with and / or used to represent the literals ASCII 0 through ASCII 255, respectively, as shown in the first group (400) of rows.

[0040] According to line 402, the Huffman code tree A symbol HCTA (256) is used to indicate the end of block symbol. For example, this symbol is used Figure 3 The end element of the block (314).

[0041] According to the last group (404) of rows in the table, Huffman code tree A symbols HCTA (257) through HCTA (285) are used to represent (e.g., configurable) lengths L0 through L28, each of which is a length between 3 and 258. For example, the values ​​of L0 through L28 are set to the values ​​that appear in and / or are used by a particular length-distance pair that appears in a given DEFLATE block. Due to the dynamic nature of the lengths, the table is referred to as a dynamic weight table (and other mappings and / or elements in the table may also change dynamically from DEFLATE block to DEFLATE block, if desired). For example, HCTA (257) corresponds to a length (L0) of 3 (see row 410), HCTA (258) corresponds to a length (L1) of 4 (see row 412), HCTA (284) corresponds to a length (L27) between 227 and 257 (including 227 and 257) (see row 414), and HCTA (285) corresponds to a length (L28) of 258 (see row 416).

[0042] The Huffman code tree A symbols shown in this table (ie, HCTA(0)-HCTA(285)) are (Huffman) encoded using a codeword tree of maximum length 15. The figure below shows an example of this.

[0043] Figure 5 is a diagram illustrating an example of a typical codeword tree associated with a literal, length, and block end. In this example, Figure 4 A symbol of the Huffman code tree (i.e., HCTA(0)-HCTA(285)) is Huffman encoded (i.e., mapped to a codeword) depending on their frequency of occurrence.

[0044] The most frequently used Huffman code tree A symbol (502) (eg, having the highest frequency of occurrence in the associated DEFLATE block) is mapped to the shortest codeword (500a), in this example, a length 1 codeword with a value of 0.

[0045] The second most common Huffman code tree A symbol (504) is mapped to the second shortest codeword (500b), a codeword of length 2 with a value of 10 in this example.

[0046] This continues until the second least frequent Huffman code tree A symbol (506) and the least frequent Huffman code tree A symbol (508), which are mapped to the longest codewords (500c and 500d) that are 15 bits long and have values ​​of 1111 1111 1111 110 and 1111 1111 1111 111, respectively. In other words, the Huffman tree shown here has a maximum length of 15, and / or must pass through a maximum of 15 multiplexers to obtain a Huffman code tree A symbol from a codeword.

[0047] For simplicity, the distance corresponding to the match (e.g., Figure 3 The dynamic weight table of the Huffman code tree B (304) in and its codeword tree are not shown, but can be constructed similarly. In some typical DEFLATE software blocks, the matching distance is limited to values ​​between 1 and 32,768. In some such typical DEFLATE, Huffman coding is applied only to the most significant two bits of the distance, and the remaining bits (if any) are appended (it should be noted that the given most significant two bits contain information about the number of remaining bits). This effectively reduces the Huffman tree size to 30, instead of 32,768.

[0048] To decode a codeword, the decoder can check the first (e.g., most significant) bit in the codeword. If the first bit is 0, then it is the shortest codeword (500a) corresponding to the most common Huffman code tree A symbol (502). If the first bit is 1, the decoder will check the next bit in the sequence, and so on. Thus, in the worst case, the decoder can go through 15 layers of multiplexers (e.g., to reach the longest codeword (500c and 500d) and the least common Huffman code tree A symbol (506 and 508)).

[0049] Return briefly to Figure 3 , the compressed data (306) includes Figure 5 The Huffman code tree A (302) at the beginning of the DEFLATE block (300) contains Figure 4 The dynamic weight table shown in Figure 5 The decoder uses the dynamic weight table and the codeword tree (or equivalent information) shown in FIG. 3 to decode the codewords contained in the compressed data ( 306 ).

[0050] It should be noted that once the Huffman code tree A symbols are determined (for example, according to Figure 5 ), then determine the appropriate text (see e.g. Figure 4 400 in ) or other symbols (see e.g. Figure 4 The 402 and 404 lines in the . Figure 4 The dynamic weight table shown in is itself Huffman encoded. Figure 4 The size of the dynamic weight table shown in (i.e., having 286 entries), the decoding path passes through 9 layers of multiplexers to identify the appropriate literal (or length or block end) for a given Huffman code tree A symbol.

[0051] Consider a data storage application that uses a hardware encoder and decoder to perform the typical DEFLATE described above. In such applications, the data size (e.g., the amount of data that can be read back from the storage device during a single read operation) is limited to ~4K (e.g., 4,096) or ~8K (e.g., 8,192) bytes. Assume that a single DEFLATE block corresponds to 4K of data, so that each read from the storage device corresponds to one or two DEFLATE blocks (e.g., depending on whether the data size is ~4K or ~8K bytes). One drawback associated with the techniques described above is that the construction of a Huffman code tree for a literal (e.g., at the encoder) (which includes constructing a dynamic weight table with 286 literals (see, e.g., Figure 4 ) and a codeword tree with maximum length 15 (see e.g. Figure 5 )) requires long waiting times (i.e., long processing times) and a large degree of complexity.

[0052] On the decoder side, latency is often an important consideration (e.g., because the user and / or higher-level applications are waiting for data in the DEFLATE block). However, for literal elements (e.g., Figure 3 The critical path delay for decoding a word (312 in ) depends linearly on the maximum tree length and the number of words. For the exemplary storage application described above, the critical path (for decoding a word) passes through 15 layers of multiplexers to determine the Huffman code tree A symbol from the codeword (e.g., according to Figure 5 ); then, to determine which word corresponds to a given Huffman code tree A symbol (for example, according to Figure 4 )’s dynamic weight table takes 9 layers of multiplexers.

[0053] In contrast, Figure 1 The encoding techniques described in (and Figure 2 The corresponding decoding technique described in ) has less complexity (e.g., at the encoder, when constructing the Huffman code tree) and / or faster processing time (e.g., at the encoder and / or decoder). The following embodiments describe different aspects of this technology in more detail.

[0054] Figure 6is a diagram illustrating an embodiment of a DEFLATE block having sub-word elements. In the example shown, the DEFLATE block (600) has three Huffman code trees. The first Huffman code tree (602) is associated with and / or used to encode the first sub-word, the length, and the block end. Thus, the length element (610), the first sub-word element (614), and the block end element (618) in the compressed data (608) are encoded using the first Huffman code tree (602).

[0055] The second Huffman code tree (604) is associated with and / or used to encode the second sub-word. Thus, the second sub-word element (616) in the compressed data (608) is encoded using the second Huffman code tree (604).

[0056] The third Huffman code tree (606) is associated with and / or used to encode distances, such as the distance element (612) in the compressed data (608).

[0057] Figure 7 is a diagram illustrating an embodiment of a dynamically generated weight table associated with a first subword, length, and block end. For example, Figure 6 The first Huffman code tree (602) in may use the exemplary dynamic weight table shown here. In this example, the literal is constrained, limited, or assumed to have 8 bits so that 2 8 = 256 literals (e.g., 256 ASCII characters). This may be a safe assumption to make, since many DEFLATE blocks may contain (only) 256 or fewer unique literals. In this example, the 3 most significant bits in a literal are associated with the first sub-literal included in this table. Thus, according to the first eight rows (700), HCT1 symbols HCT1(0) through HCT1(7) are used for the first sub-literal values ​​000 through 111, respectively. Note that any combination of bits may be selected for or included in the first sub-literal, and it is not limited to the three most significant bits.

[0058] In this example, the (supported) lengths in LZ77 matching (i.e., length and distance pairs) are restricted to between 3 and 24. In the next group (702) of rows, HCT1 symbols HCT1(8) to HCT1(29) are used for and / or associated with lengths of 3 to 24, respectively.

[0059] Note that the maximum supported length is increased from 258 (see Figure 4 ) is reduced to 24 (e.g. Figure 7 One benefit of ) is that smaller memory can be used in the encoder and decoder to store previous symbols.

[0060] The last row (704) shows that the HCT1 symbol HCT1 (30) is used for the end-of-block element. In some embodiments, unused symbols (eg, HCT1 (31)) not shown are reserved.

[0061] Use a smaller number of allowed and / or supported lengths (e.g. Figure 7 The 22 supported length pairs in Figure 4 The size of the table is reduced linearly with the 29 supported lengths in the table. In addition, the table is simplified because Figure 7 The length (702) in is a fixed value, which is Figure 4 Due to the smaller table size, the dynamic length value (404) in Figure 4 When the table in has a maximum Huffman length of 9 (for example, a 9-layer multiplexer), Figure 7 The tables in have implemented a maximum Huffman length of 8 or 9 (i.e., sometimes and / or potentially shorter).

[0062] Figure 8 is a diagram illustrating an embodiment of a weight table associated with a second sub-word. For example, Figure 6 The second Huffman code tree (604) in can use the exemplary dynamic weight table shown here. In this example, the 5 least significant bits from the 8-bit word comprise the second sub-word. As shown in the first row (800), the second Huffman code tree symbol HCT2 (0) is used to represent the second sub-word 00000. The second Huffman code tree symbol HCT2 (1) is used to represent the second sub-word 00001 as shown in the second row (802), and so on, to the second Huffman code tree symbols HCT2 (30) and HCT2 (31), which correspond to the second sub-words 11110 and 11111 respectively as shown in the last two rows (804 and 806).

[0063] Fig. 9 is a diagram illustrating an embodiment of a dynamic weight table associated with distance. For example, Figure 6 The third Huffman code tree (606) in may use the exemplary dynamic weight table shown here. In this example, there are 22 support distances, ranging from distance 3 (see, for example, the first row (900)) to distance 24 (see, for example, the last row (902)).

[0064] Fig.10 is a diagram illustrating an embodiment of a codeword tree associated with a first subword, a length, and a block end. In this example, Figure 7The first Huffman code tree symbols (i.e., HCT1(0)-HCT1(30)) are Huffman encoded as shown here. The two most common first Huffman code tree symbols (1000a and 1000b) are assigned to the two shortest code words (1002a and 1002b). The two least common first Huffman code tree symbols (1004a and 1004b) are assigned to the two longest code words (1006a and 1006b) of length 9 bits. This exemplary codeword tree with 31 first Huffman code tree symbols or code words has a maximum Huffman length of 9.

[0065] For the sake of brevity, no details are given for the second Huffman code tree (see Figure 8 ) shows a codeword tree (e.g., similar to Fig.10 ), nor for the third Huffman code tree (see Fig. 9 ) show the codeword trees as they are similar to the examples described above.

[0066] By dividing the exemplary 8-bit word into two sub-words, and then combining one of the sub-words with a smaller support length, a much smaller Huffman code tree can be constructed for the sub-word, as described above. Using two smaller Huffman code trees for the sub-words (e.g., as opposed to using a single larger Huffman code tree for all words) reduces the complexity associated with building the two smaller Huffman code trees and the latency associated with traversing and / or decoding using the two smaller Huffman code trees. For example, the overhead of the two smaller trees is 63×4=252 bits, due to the 63 codes each having 4 bits, while the overhead of the original tree is 286×4=1,144 bits, due to the 286 codes each expressed with 4 bits. Although the compression rate may be slightly degraded, in some applications, this is an acceptable tradeoff for reduced complexity and / or reduced latency (e.g., real-time applications where users are waiting for decoded data, or storage-rich applications where a slightly degraded compression rate is acceptable).

[0067] As shown in the above example, in some encoding (decoding) embodiments, the encoded data passes through (e.g., is stored in or read back from) a storage system having a data size that is a multiple of 4,096 bytes (e.g., 4,096 bytes, 8,192 bytes, etc.), and the first Huffman code tree is further associated with a set of support lengths having 22 support lengths (see, e.g. Figure 7 ).

[0068] As shown in the above example, in some encoding (decoding) embodiments, the encoded data passes through a storage system having a data size that is a multiple of 4,096 bytes, and the first Huffman code tree is further associated with a set of support lengths having a maximum support length of 24 (see, e.g. Figure 7 ).

[0069] As shown in the examples above, in some encoding (decoding) embodiments, the encoded data is passed through a storage system having a data size that is a multiple of 4,096 bytes, with the first group of bits in the first subword having three bits (see e.g. Figure 7 ), and the second group of bits in the second sub-word has five bits (see e.g. Figure 8 ).

[0070] Fig.11 is a diagram illustrating an embodiment of a DEFLATE encoder that divides a word into sub-words. In the example shown, the input (uncompressed) data is first analyzed by a Huffman code tree generator (1100) that generates three Huffman code trees. Huffman code tree information is output by the Huffman code tree generator (1100) and is passed to a first Huffman code tree (HCT1) encoder (1102), a second Huffman code tree (HCT2) encoder (1104), and a third Huffman code tree (HCT3) encoder (1106), and appropriate code information is used to configure those encoders. In order to more clearly distinguish between configuration information (e.g., Huffman code tree information) and payload, the Huffman code tree information is shown with dashed lines.

[0071] Once the encoders are configured or otherwise set up, the input data can be encoded. In this example, LZ77 encoding is first attempted. The input data is passed to a buffer (1108). A repeat locator (1110) searches the buffer (1108) for repeating sequences, such as when a given sequence is being processed and an earlier occurring copy of that sequence is located in the buffer. If the repeating sequence is located within the buffer (1108), the length is passed to the HCT1 encoder (1102), and the distance is passed from the repeat locator (1110) to the HCT3 encoder (1106).

[0072] It is noted that the size of the buffer (1108) is smaller than that of a buffer implementing a typical DEFLATE scheme. For example, in a typical DEFLATE scheme, the supported lengths are drawn from the range of 3 to 258, and the supported distances are drawn from the range of 1 to 32,768. In contrast, in the case of the reduced complexity DEFLATE embodiment described above, the supported lengths are drawn from the range of 3 to 24, and the supported distances are drawn from the range of 1 to 32,768.

[0073] If the repeat sequence locator (1110) is unable to locate a repeat sequence within the buffer (1108), the repeat sequence locator (1110) communicates with the word encoder interface (1112) so that the appropriate symbol(s) and / or byte(s) of the input data can be encoded as sub-words. The word encoder interface (1112) outputs a first sub-word (i.e., sub-word 1) to the HCT1 encoder (1102) and outputs a second sub-word (i.e., sub-word 2) to the HCT2 encoder (1104).

[0074] The outputs of the encoders (1102, 1104, and 1106) and the Huffman code tree information from the Huffman code tree generator (1100) are passed to a multiplexer (1114) which selects the appropriate input at the appropriate time and outputs the output (compressed) data. For example, the output of the multiplexer (1114) may correspond to Figure 6 The DEFLATE block (600) shown in FIG. 110 is a block that is used to control the DEFLATE block. The multiplexer (1114) can select the H input (e.g., Figure 6 Huffman code tree information (602, 604 and 606) in the HCT1 encoder (1102). When the HCT1 encoder (1102) has generated a codeword (e.g., corresponding to Figure 6 The A input of the multiplexer (1114) may be selected when the length element (610), the first subword element (614), or the end-of-block element (618) in the HCT2 encoder (1104) has generated a codeword (e.g., corresponding to Figure 6 , the B input of the multiplexer (1114) may be selected when the second sub-word element (616) in the HCT3 encoder (1106) has generated a codeword (e.g., corresponding to Figure 6 When the distance element (612) in the input is used, the C input of the multiplexer (1114) can be selected.

[0075] In some embodiments, the exemplary encoder shown here performs Figure 1 For example, the text encoder interface (1112) may perform steps 100 and 102, the HCT1 encoder (1102) may perform step 104, the HCT2 encoder (1104) may perform step 106, and the multiplexer (1114) may perform step 108.

[0076] Fig.12is a diagram illustrating an embodiment of a DEFLATE decoder in which sub-words are combined into words. In this example, the input (compressed) data is first analyzed by a parser (1200), which acts as an input interface and also extracts Huffman code tree information (e.g., in Figure 6 Then, the Huffman code tree information is passed to the first Huffman code tree (HCT1) decoder (1202), the second Huffman code tree (HCT2) decoder (1204), and the third Huffman code tree (HCT3) decoder (1206), and the appropriate code information is used to configure those decoders.

[0077] In this example, the DEFLATE block includes identification information (e.g., field or element identifiers) that identifies the element and / or codeword type. These identifiers allow the parser (1200) to separate the length codeword and the first sub-word codeword and pass them to the HCT1 decoder (1202), separate the second sub-word codeword and pass it to the HCT2 decoder (1204), and separate the distance codeword and pass it to the HCT3 decoder (1206).

[0078] For LZ77 information, the HCT1 decoder (1202) decodes the length codeword to obtain the length, and the HCT3 decoder (1206) decodes the distance codeword to obtain the distance. The length and distance are passed to the repeat sequence extractor (1208), which accesses the buffer (1210) with the specified length and distance to obtain the repeat sequence. The repeat sequence is then passed from the repeat sequence extractor (1208) to the A input of the multiplexer (1212), which also serves as the output interface. The buffer (1210) samples the output of the multiplexer (1212) so that all previously occurring symbols or bytes on the window of interest are available for replication.

[0079] For a sub-word, the HCT1 decoder (1202) generates a first sub-word from the corresponding first sub-word codeword, and the HCT2 decoder (1204) generates a second sub-word from the corresponding second sub-word codeword. The two sub-words are passed to a (e.g., bit) combiner (1214), which combines the two sub-words to obtain a word (e.g., by concatenating the two sub-words in the example described above). The word is passed from the combiner (1214) to the B input of the multiplexer (1212). The multiplexer selects the appropriate input at the appropriate time to generate output (uncompressed) data.

[0080] In some embodiments, the exemplary decoder shown herein performs Figure 2For example, the parser (1200) may perform step 200, the HCT1 decoder (1202) may perform step 202, the HCT2 decoder (1204) may perform step 204, the combiner (1214) may perform step 206, and the multiplexer (1212) may perform step 208.

[0081] In some embodiments, how the bit splitting is performed (e.g., how the literal is split into sub-literals) is simplified. In the simple example described above, the 3 most significant bits of the literal element form the first sub-literal, and the 5 least significant bits form the second sub-literal. Other bit splittings may be used, and if 5 of the 8-bit indices are selected, there is There are three possible ways to divide an 8-bit word into 5-bit sub-words and 3-bit sub-words.

[0082] If an optimized compression ratio is desired, the Huffman code tree generator in the encoder (e.g. Fig.11 1100 in) can collect frequencies for each of the 56 possibilities in order to determine the best way to partition the bits (e.g., which optimizes the compression ratio). In some embodiments, instead of considering all 56 possibilities (as an example), only four possible ways to partition the word bits into sub-words are considered (e.g., using samples and / or representative data sets). Alternatively, a single predetermined bit partitioning is used that uses samples and / or representative data sets to achieve the best average compression. In some applications, this tradeoff between simplifying system complexity and / or reducing latency at the expense of some compression performance is acceptable.

[0083] Another subtask or operation where it may be desirable to simplify the complexity of the system is the association of symbols with codewords on a codeword tree (e.g., a Huffman code tree). Fig.10 In some embodiments, the Huffman code tree generator (e.g., Fig.11 1100) performs this task by determining or otherwise estimating the compressed size of a given input data stream using the collected set of frequencies, without actually encoding the input data of interest.

[0084] For example, let f 0 , f 1 , f 1 … f (n-1) is the sequence of collected frequencies. Let l 0 , l 1 , l 1… l (n-1) is the associated Huffman code length. Use Fig.10 As an example, l 0 = 1 bit of codeword 1002a, l 1 = 2 bits of codeword 1002b, and so on; and f 0 is the frequency of the most common first Huffman code tree symbol 1000a, f 1 is the second most common frequency of occurrence of the first Huffman code tree symbol 1000b, and so on.

[0085] The overall Huffman coded data length is given by For example, this value can correspond to Figure 6 In some embodiments, instead of considering all possible mappings or associations between Huffman code tree symbols and code words, the Huffman code tree generator only considers a limited number of possibilities. In one example, only four predefined mappings or associations are evaluated. ; select the predefined mapping or association with the lowest value.

[0086] However, using equation alone It may still be necessary to compute a Huffman code for each candidate (e.g., according to Fig.10 To further reduce complexity, in some embodiments, by assigning a Huffman code tree symbol to each candidate for a codeword, f i Replace integer bit length l i The following formula is used to estimate the Huffman compressed data length:

[0087]

[0088] This avoids building a Huffman tree.

[0089] DEFLATE encoders and decoders that implement the techniques described herein may be used in a variety of applications and / or systems. The following figures illustrate exemplary communication applications and storage applications, respectively.

[0090] Fig.131 is a diagram illustrating an embodiment of a DEFLATE encoder and decoder using sub-literals in a communication application. In some embodiments, the DEFLATE encoder (1300) and the DEFLATE decoder (1308) are implemented in hardware, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In this example, a DEFLATE encoder using sub-literals (200a) inputs (e.g., uncompressed) input data to be encoded and / or compressed prior to transmission over a communication channel (1304). The DEFLATE block to be transmitted (e.g., which contains compressed input data) is passed from the DEFLATE encoder (1300) and passed to a transmitter (1302), which modulates and / or formats the DEFLATE block as appropriate for a particular communication channel and sends the transmitted data over a communication channel (1304), such as a wired or wireless channel and / or network.

[0091] A receiver (1306) coupled to a communication channel (1304) inputs the received data and demodulates and / or extracts the DEFLATE block from the received data. The DEFLATE block is then decoded using a sub-text DEFLATE decoder (1308) and outputs (e.g., uncompressed) output data. For example, the output data may be passed to some higher-level application and / or rendered or otherwise displayed to a user.

[0092] In one example, an exemplary DEFLATE encoder (1300) and DEFLATE decoder (1308) are used to exchange web-related files and / or information. A web server may, for example, use a DEFLATE encoder (e.g., 1300) to compress web pages and / or CSS files before they are transmitted over a communication channel (1304), and a requesting device may include a DEFLATE decoder (e.g., 1308). Compressing information before it is exchanged over a communication channel allows the information to be provided to a requester and / or receiver more quickly, and also uses the bandwidth of the communication channel more efficiently.

[0093] Fig.14is a diagram illustrating an embodiment of a DEFLATE encoder and decoder using sub-literals in a storage application. In some embodiments, the DEFLATE encoder (1400) and the DEFLATE decoder (1406) are implemented in hardware, such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). In this example, input data to be encoded is passed to the DEFLATE encoder (1400) using sub-literals. The DEFLATE encoder (1400) passes the (e.g., compressed) DEFLATE blocks to be stored and / or written to a storage interface (1402), which in turn writes the data to a storage device (1404).

[0094] To obtain the original data, the storage interface (1402) reads back the data stored on the storage device (1404) and passes the DEFLATE block to the DEFLATE decoder (1406) using sub-literals. The DEFLATE decoder (1406) decodes the DEFLATE block and outputs (e.g., uncompressed) data. In some embodiments, the LDPC decoder (1406) performs an early decoding termination process (e.g., Figure 1 ), the process allows LDPC decoding to end earlier than some other LDPC decoders, thereby reducing latency. For example, the output data can be passed to some higher-level application and / or rendered or otherwise displayed to a user. By compressing the information before it is stored, the capacity of the storage device (1404) is more efficiently utilized and more usable and / or payload information can be stored thereon.

[0095] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways to implement the invention. The disclosed embodiments are illustrative and not restrictive.

Claims

1. A system comprising: Text encoder interface, where: The text encoder interface receives a text element having a plurality of bits; and The word encoder interface divides the plurality of bits in the word element into a first subword including a first group of bits and a second subword including a second group of bits; a first Huffman code tree encoder, wherein the first Huffman code tree encoder encodes the first subword using the first Huffman code tree to obtain a first subword codeword; a second Huffman code tree encoder, wherein the second Huffman code tree encoder encodes the second subword using a second Huffman code tree to obtain a second subword codeword; and A multiplexer, wherein the multiplexer outputs encoded data including information associated with a first Huffman code tree, information associated with a second Huffman code tree, a first subword codeword, and a second subword codeword.

2. The system of claim 1, wherein the system is implemented using one or more of: an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

3. The system of claim 1 , further comprising a storage device having a data size that is a multiple of 4,096 bytes, wherein: The encoded data is output to the storage device and stored on the storage device; as well as The first Huffman code tree is further associated with a set of support lengths having 22 support lengths.

4. The system of claim 1 , further comprising a storage device having a data size that is a multiple of 4,096 bytes, wherein: The encoded data is output to the storage device and stored on the storage device; as well as The first Huffman code tree is further associated with a set of support lengths having a maximum support length of 24.

5. The system of claim 1 , further comprising a storage device having a data size that is a multiple of 4,096 bytes, wherein: The encoded data is output to the storage device and stored on the storage device; The first group of bits in the first subword has three bits; as well as The second group of bits in the second subword has five bits.

6. A method comprising: Receives a literal element with multiple bits; dividing the plurality of bits in the word element into a first subword including a first group of bits and a second subword including a second group of bits; encoding the first subword using a first Huffman code tree to obtain a first subword codeword; encoding the second subword using a second Huffman code tree to obtain a second subword codeword; and The output includes information associated with the first Huffman code tree, information associated with the second Huffman code tree, encoded data for the first sub-word codeword, and the second sub-word codeword.

7. The method of claim 6, wherein the method is performed using one or more of: an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

8. The method according to claim 6, wherein: The encoded data is output to a storage device and stored on the storage device, wherein the storage device has a data size that is a multiple of 4,096 bytes; as well as The first Huffman code tree is further associated with a set of support lengths having 22 support lengths.

9. The method according to claim 6, wherein: The encoded data is output to a storage device and stored on the storage device, wherein the storage device has a data size that is a multiple of 4,096 bytes; as well as The first Huffman code tree is further associated with a set of support lengths having a maximum support length of 24.

10. The method according to claim 6, wherein: The encoded data is output to a storage device and stored on the storage device, wherein the storage device has a data size that is a multiple of 4,096 bytes; The first group of bits in the first subword has three bits; and The second group of bits in the second subword has five bits.

11. A system comprising: an input interface, wherein the interface receives encoded data, the encoded data comprising information associated with a first Huffman code tree, information associated with a second Huffman code tree, a first sub-word codeword, and a second sub-word codeword; a first Huffman code tree decoder, wherein the first Huffman code tree decoder decodes the first subword codeword using the first Huffman code tree to obtain a first subword including a first set of bits; a second Huffman code tree decoder, wherein the second Huffman code tree decoder decodes the second subword codeword using the second Huffman code tree to obtain a second subword including a second set of bits; a combiner, wherein the combiner combines a first group of bits in a first subword and a second group of bits in a second subword to obtain a word element; and An output interface, wherein the output interface outputs text elements.

12. The system of claim 11, wherein the system is implemented using one or more of: an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

13. The system of claim 11, further comprising a storage device having a data size that is a multiple of 4,096 bytes, wherein: The encoded data is stored on and received from the storage device; and The first Huffman code tree is further associated with a set of support lengths having 22 support lengths.

14. The system of claim 11, further comprising a storage device having a data size that is a multiple of 4,096 bytes, wherein: The encoded data is stored on and received from the storage device; and The first Huffman code tree is further associated with a set of support lengths having a maximum support length of 24.

15. The system of claim 11, further comprising a storage device having a data size that is a multiple of 4,096 bytes, wherein: The encoded data is stored on and received from the storage device; The first group of bits in the first subword has three bits; as well as The second group of bits in the second subword has five bits.

16. A method comprising: receiving encoded data including information associated with a first Huffman code tree, information associated with a second Huffman code tree, a first sub-word codeword, and a second sub-word codeword; decoding the first subword codeword using the first Huffman code tree to obtain a first subword including a first group of bits; decoding the second subword codeword using a second Huffman code tree to obtain a second subword including a second set of bits; combining a first group of bits in the first subword and a second group of bits in the second subword to obtain a word element; as well as Output the text element.

17. The method of claim 16, wherein the method is performed using one or more of: an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

18. The method of claim 16, wherein: The encoded data is stored on and received from a storage device, wherein the storage device has a data size that is a multiple of 4,096 bytes; as well as The first Huffman code tree is further associated with a set of support lengths having 22 support lengths.

19. The method of claim 16, wherein: The encoded data is stored on and received from a storage device, wherein the storage device has a data size that is a multiple of 4,096 bytes; as well as The first Huffman code tree is further associated with a set of support lengths having a maximum support length of 24.

20. The method of claim 16, wherein: The encoded data is stored on and received from a storage device, wherein the storage device has a data size that is a multiple of 4,096 bytes; The first group of bits in the first subword has three bits; and The second group of bits in the second subword has five bits.

Images