Normalized Huffman encoding and decoding method and neural network computing chip

Abstract

The present invention proposes a normalized Huffman coding method and a neural network computing chip, comprising: performing run-length all-zero coding on the neural network data to obtain run-length compressed data, wherein the run-length all-zero coding includes run-length coding only for zero characters in the neural network data; performing Huffman coding on the run-length compressed data, and adding 1 to the leaf node codes in each level of nodes in the coding result from top to bottom and shifting them left to move the leaf nodes in each level of nodes in the coding result to the left side of the tree structure, thereby generating a normalized Huffman coding of the coding result as the compression result of the neural network data.

Description

Technical Field

[0001] The present invention relates to the field of neural network calculation, and in particular to a normalized Huffman encoding and decoding method and a neural network calculation chip. Background Art

[0002] Entropy coding is a branch of lossless compression algorithms. It uses the principle of entropy to prevent any information loss during the compression and decompression process. Entropy represents a measure of the uncertainty of the source. While there are many types of entropy coding algorithms, this article focuses on two representative ones: Huffman coding and arithmetic coding.

[0003] Huffman coding, first proposed by Huffman in 1952, is a variable-length coding method based on an optimal binary tree (also known as a Huffman tree). Its key concept is that each character (literal) in the sequence to be encoded is used as a leaf node in the binary tree, thus satisfying the property of prefix coding. That is, the encoding of each character is not a prefix of the encoding of any other character, thus reducing decoding complexity.

[0004] In practice, Huffman coding requires frequency statistics for each character, then repeatedly merging the two child nodes with the lowest frequency to form the frequency of their parent node, recursively until a root node is generated. This process of growing a binary tree also generates a Huffman code table, where the encoding of each character corresponds to the path from the root node to a leaf node. Branches to the left add 0, while branches to the right add 1. Figure 1 Given the Huffman tree corresponding to the sequence {AABBCCCDDDDEEEEE}, Figure 2 is the encoding result corresponding to this Huffman tree.

[0005] From the above Huffman tree construction process, we can see that this encoding has two characteristics:

[0006] (1) The character encoding length is shorter when the word frequency is higher, so the weighted encoding length of the whole character sequence is optimal. Therefore, Huffman coding is considered to be the entropy coding with the best compression ratio.

[0007] (2) The Huffman tree constructed for the same character sequence is not unique. For example, the parent nodes of characters A and B have the same frequency as the leaf nodes corresponding to character D. Therefore, there are many variations of Huffman coding, and the hardware overhead of different implementations may vary greatly. Summary of the Invention

[0008] Specifically, the present invention proposes a normalized Huffman coding method, comprising:

[0009] Step 1: Obtain the Huffman tree data to be standardized;

[0010] Step 2: Left-shift the leaf node codes in each level of the Huffman tree data by adding 1 from top to bottom to move the leaf nodes in each level of the Huffman tree data to the left side of the tree structure, and generate and store the canonical coding data of the Huffman tree data.

[0011] In the normalized Huffman coding method, wherein the Huffman tree data is a character sequence, step 2 includes:

[0012] Step 21: Arrange all characters to be encoded in the character sequence in descending order of frequency of occurrence to form a first table;

[0013] Step 22: extract all valid code lengths involved in the character sequence, and arrange all valid code lengths in ascending order to form a second table;

[0014] Step 23: extract the sort subscripts of the characters corresponding to the last codeword of each valid code length in the character sequence in the first table to form a third table;

[0015] Step 24: extract the value of the last code word of each valid code length in the character sequence to form a fourth table;

[0016] Step 25: Subtract the third table from the fourth table to form a fifth table;

[0017] Step 26: sequentially extract characters from the character sequence as the current character, obtain the sort order of the current character by accessing the first table; access the third table to obtain the first subscript index greater than or equal to the sort order; access the fifth table and the second table to obtain the base value and code length corresponding to the subscript; add the base value to the sort order as the code value of the current character; and express the code value with the code length as the standard encoding of the current character;

[0018] Step 27: Gather the canonical codes of all characters in the character sequence in order to obtain the canonical code data.

[0019] The normalized Huffman decoding method includes:

[0020] Step S1, obtaining the canonical coded data to be decoded, traversing the fourth table and the second table by accumulating subscripts starting from 0 until the value in the fourth table is greater than or equal to the first len ​​bits of the code stream, obtaining the current code word value, subscript value, len value, and the value with subscript 0 in the fourth table, where len value is the effective code length in the second table;

[0021] Step S2: access the fifth table, subtract the value in the fifth table corresponding to the subscript value from the current code word value to obtain a character order, access the first table with the character order, and obtain the character corresponding to the character order.

[0022] The present invention also proposes a neural network computing chip based on normalized Huffman coding, which includes: an input circuit, an operation circuit and a storage circuit; the operation circuit includes a master circuit and a slave circuit;

[0023] An input circuit for acquiring neural network data;

[0024] The arithmetic circuit performs run-length all-zero encoding on the neural network data in response to the quantization instruction to obtain run-length compressed data, wherein the run-length all-zero encoding includes run-length encoding only zero characters in the neural network data; performs Huffman encoding on the run-length compressed data, and shifts the leaf node codes in each level of nodes in the encoding result by adding 1 to the left from top to bottom, so as to move the leaf nodes in each level of nodes in the encoding result to the left side of the tree structure, thereby generating a canonical Huffman code of the encoding result as a compression result of the neural network data;

[0025] The storage circuit is used to store the compression result.

[0026] The neural network computing chip based on normalized Huffman coding, wherein the encoding result is a character sequence, and the operation circuit is used to arrange all the characters to be encoded in the character sequence in descending order of frequency of occurrence to form a first table;

[0027] Extract all valid code lengths involved in the character sequence, and arrange all valid code lengths in ascending order to form a second table;

[0028] Extract the sort subscripts of the characters corresponding to the last codeword of each valid code length in the character sequence in the first table to form a third table;

[0029] Extract the value of the last code word of each valid code length in the character sequence to form a fourth table;

[0030] Subtract the third table from the fourth table to form a fifth table;

[0031] Extract characters from the character sequence in order as the current character, obtain the sort order of the current character by accessing the first table; access the third table to obtain the first subscript index greater than or equal to the sort order; access the fifth table and the second table to obtain the base value and code length corresponding to the subscript, add the base value and the sort order to obtain the code value of the current character, and express the code value with the code length as the standard encoding of the current character;

[0032] The canonical codes of all characters in the character sequence are collected in order to obtain the canonical Huffman code.

[0033] The present invention also proposes a storage medium for storing a program of any one of the normalized Huffman coding methods.

[0034] From the above scheme, it can be seen that the advantages of the present invention are:

[0035] 1. In view of the sparsity of quantized neural network data, the present invention improves run-length coding and proposes run-length zero coding, which can more efficiently and losslessly compress neural network data;

[0036] 2. The run-length all-zero encoding of the present invention includes second-order character replacement, which further improves the compression efficiency while reducing the number of zeros in the data, leaving more compression space for subsequent Huffman encoding;

[0037] 3. The Huffman tree is reorganized from top to bottom, eliminating the need to store the complete Huffman tree structure and significantly reducing the complexity of the table lookup operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 This is an example diagram of the Huffman tree;

[0039] Figure 2 The table diagram of the encoding results corresponding to the Huffman tree;

[0040] Figure 3 and Figure 4 This is an example graph of a normalized Huffman tree. DETAILED DESCRIPTION

[0041] The present invention proposes a normalized Huffman coding method, comprising:

[0042] Step 1: Obtain the Huffman tree data to be standardized;

[0043] Step 2: Left-shift the leaf node codes in each level of the Huffman tree data by adding 1 from top to bottom to move the leaf nodes in each level of the Huffman tree data to the left side of the tree structure, and generate and store the canonical coding data of the Huffman tree data.

[0044] In the normalized Huffman coding method, wherein the Huffman tree data is a character sequence, step 2 includes:

[0045] Step 21: Arrange all characters to be encoded in the character sequence in descending order of frequency of occurrence to form a first table;

[0046] Step 22: extract all valid code lengths involved in the character sequence, and arrange all valid code lengths in ascending order to form a second table;

[0047] Step 23: extract the sort subscripts of the characters corresponding to the last codeword of each valid code length in the character sequence in the first table to form a third table;

[0048] Step 24: extract the value of the last code word of each valid code length in the character sequence to form a fourth table;

[0049] Step 25: Subtract the third table from the fourth table to form a fifth table;

[0050] Step 26: sequentially extract characters from the character sequence as the current character, obtain the sort order of the current character by accessing the first table; access the third table to obtain the first subscript index greater than or equal to the sort order; access the fifth table and the second table to obtain the base value and code length corresponding to the subscript; add the base value to the sort order as the code value of the current character; and express the code value with the code length as the standard encoding of the current character;

[0051] Step 27: Gather the canonical codes of all characters in the character sequence in order to obtain the canonical code data.

[0052] The normalized Huffman decoding method includes:

[0053] Step S1, obtaining the canonical coded data to be decoded, traversing the fourth table and the second table by accumulating subscripts starting from 0 until the value in the fourth table is greater than or equal to the first len ​​bits of the code stream, obtaining the current code word value, subscript value, len value, and the value with subscript 0 in the fourth table, where len value is the effective code length in the second table;

[0054] Step S2: access the fifth table, subtract the value in the fifth table corresponding to the subscript value from the current code word value to obtain a character order, access the first table with the character order, and obtain the character corresponding to the character order.

[0055] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0056] Normalized Huffman coding. The Huffman coding corresponding to the same data distribution is not unique. This is because nodes with the same frequency may appear in the process of constructing the Huffman tree. To better apply Huffman coding, we need a fixed and efficient method for constructing the Huffman tree. The unique Huffman coding generated by this method is called normalized Huffman coding.

[0057] Huffman tree reorganization. Specifically, the present invention adopts HSF coding as the standardized Huffman coding. The main idea of ​​this coding is to reorganize the Huffman tree from top to bottom, and the leaf nodes among the nodes at the same level are preferentially moved to the left side of the binary tree. Under the premise of not changing the frequency distribution, all codes can be obtained by adding 1 and adding 1 to the left, and then use comparison and addition operations to replace complex binary tree traversal or complete table lookup operations, which greatly reduces the storage and computing overhead required for encoding and decoding. Figure 3 、 Figure 4 As shown in Table 1, given a character sequence {u1,u2,u3,u4,u5} and any two sets of Huffman trees, their normalized forms can be found using the above rearrangement method.

[0058] Table 1 Normalized Huffman coding example

[0059]

[0060]

[0061] Code table redefinition. As you can see, each Huffman tree has its own normalized form. At the same time, the normalized codewords follow a hardware-friendly arithmetic rule: for identical code lengths, add 1; for different code lengths, add 1 and then shift left 1. We can leverage this rule to redefine the code table, eliminating the need to store the complete Huffman tree structure and significantly reducing the complexity of table lookup operations.

[0062] Taking the above-mentioned sequence to be encoded (a) as an example, the encoding and decoding process first requires the following code tables:

[0063] (1) CharTable (first table), all characters to be encoded are arranged in descending order of frequency of occurrence. In example (a), it is {u1,u4,u5,u2,u3}. The encoding and decoding can be reused.

[0064] (2) LenTable (second table), all valid code lengths are arranged in ascending order. In example (a), the HSF code has only 2 bits and 3 bits, and the corresponding LenTable is {2,3}. The codec can be reused.

[0065] (3) RangeTable (third table), the sorting index of the character corresponding to the last codeword of each valid code length in CharTable. In example (a), the last codewords of 2 bits and 3 bits correspond to u5 and u3 respectively, so RangeTable is {2,4}, which is only used in the encoding stage.

[0066] (4) LimitTable (the fourth table), the value of the last codeword of each valid code length. In example (a), the last codewords of 2-bit and 3-bit are 10 and 111 respectively, so the LimitTable is {2, 7}, which is only used in the decoding stage.

[0067] (5) BaseTable (fifth table), LimitTable minus RangeTable. In example (a), the BaseTable is {0,3}, and the codec can be reused.

[0068] On this basis, given the character u4 in example (a) and the above encoding table, the process of generating its HSF encoding can be divided into the following three steps:

[0069] Lookup, access CharTable, get the rank of u4, that is, rank(u4) = 1;

[0070] Compare, access RangeTable, get the first subscript index greater than or equal to rank(u4), because rank(u4) ≤ 2, so index(u4) = 0;

[0071] Add, access BaseTable and LenTable, get the base value base and code length len corresponding to the subscript index, the sum of base and rank is the value of the code word, combined with len, we can get the HSF code of the character u4: base(u4)=0, len(u4)=2, code(u4)=0+1=1, so the final encoding result is 01.

[0072] Correspondingly, we can deduce the decoding process of parsing the first character u4 from a string of HSF code streams such as 01xxx:

[0073] Compare, access LimitTable and LenTable, traverse by an index starting from 0 and accumulating until limit ≥ the first len ​​bits of the code stream is satisfied. Because the value of limit with subscript 0 in LimitTable ≥ the first 2 bits of the code stream, so index = 0, limit = 2, len = 2, code = 1;

[0074] Sub, access BaseTable, use code value minus the base value corresponding to the subscript index to get the character order, that is, rank = 1-0 = 1;

[0075] Lookup accesses the CharTable and obtains the character corresponding to the sort rank. Therefore, the final decoding result is the item ranked 1 in the CharTable, which is the character u4.

[0076] The above is an introduction to the normalized Huffman coding algorithm. It can be seen that HSF coding can simplify the storage and operation structure of Huffman coding. At the same time, it naturally divides the encoding and decoding process into a three-stage pipeline, providing an efficient and reasonable implementation solution for hardware deployment.

[0077] This paper proposes a normalized Huffman coding. By reorganizing the Huffman tree and redefining the code table, a very compact entropy coding method is designed to further compress characters after run-length all-zero encoding.

[0078] The present invention also proposes a neural network computing chip based on normalized Huffman coding, which includes: an input circuit, an operation circuit and a storage circuit; the operation circuit includes a master circuit and a slave circuit;

[0079] An input circuit for acquiring neural network data;

[0080] The arithmetic circuit performs run-length all-zero encoding on the neural network data in response to the quantization instruction to obtain run-length compressed data, wherein the run-length all-zero encoding includes run-length encoding only zero characters in the neural network data; performs Huffman encoding on the run-length compressed data, and shifts the leaf node codes in each level of nodes in the encoding result by adding 1 to the left from top to bottom, so as to move the leaf nodes in each level of nodes in the encoding result to the left side of the tree structure, thereby generating a canonical Huffman code of the encoding result as a compression result of the neural network data;

[0081] The storage circuit is used to store the compression result.

[0082] The neural network computing chip based on normalized Huffman coding, wherein the encoding result is a character sequence, and the operation circuit is used to arrange all the characters to be encoded in the character sequence in descending order of frequency of occurrence to form a first table;

[0083] Extract all valid code lengths involved in the character sequence, and arrange all valid code lengths in ascending order to form a second table;

[0084] Extract the sort subscripts of the characters corresponding to the last codeword of each valid code length in the character sequence in the first table to form a third table;

[0085] Extract the value of the last code word of each valid code length in the character sequence to form a fourth table;

[0086] Subtract the third table from the fourth table to form a fifth table;

[0087] Extract characters from the character sequence in order as the current character, obtain the sort order of the current character by accessing the first table; access the third table to obtain the first subscript index greater than or equal to the sort order; access the fifth table and the second table to obtain the base value and code length corresponding to the subscript, add the base value and the sort order to obtain the code value of the current character, and express the code value with the code length as the standard encoding of the current character;

[0088] The canonical codes of all characters in the character sequence are collected in order to obtain the canonical Huffman code.

Claims

1. A normalized Huffman coding method, characterized in that: include: Step 1: Obtain the Huffman tree data to be standardized; Step 2: Left-shifting the leaf node codes in each level of the Huffman tree data by adding 1 from top to bottom to move the leaf nodes in each level of the Huffman tree data to the left side of the tree structure, thereby generating and storing canonical coding data of the Huffman tree data; Wherein the Huffman tree data is a character sequence, step 2 includes: Step 21: Arrange all characters to be encoded in the character sequence in descending order of frequency of occurrence to form a first table; Step 22: extract all valid code lengths involved in the character sequence, and arrange all valid code lengths in ascending order to form a second table; Step 23: extract the sort subscripts of the characters corresponding to the last codeword of each valid code length in the character sequence in the first table to form a third table; Step 24: extract the value of the last code word of each valid code length in the character sequence to form a fourth table; Step 25: Subtract the third table from the fourth table to form a fifth table; Step 26: sequentially extract characters from the character sequence as the current character, obtain the sort order of the current character by accessing the first table; access the third table to obtain the first subscript index greater than or equal to the sort order; access the fifth table and the second table to obtain the base value and code length corresponding to the subscript; add the base value to the sort order as the code value of the current character; and express the code value with the code length as the standard encoding of the current character; Step 27: Gather the canonical codes of all characters in the character sequence in order to obtain the canonical code data.

2. A normalized Huffman decoding method according to claim 1, characterized in that: include: Step S1, obtaining the canonical coded data to be decoded, traversing the fourth table and the second table by accumulating subscripts starting from 0 until the value in the fourth table is greater than or equal to the first len ​​bits of the code stream, obtaining the current code word value, subscript value, len value, and the value with subscript 0 in the fourth table, where len value is the effective code length in the second table; Step S2: access the fifth table, subtract the value in the fifth table corresponding to the subscript value from the current code word value to obtain a character order, access the first table with the character order, and obtain the character corresponding to the character order.

3. A neural network computing chip based on normalized Huffman coding, characterized in that: include: An input circuit, an operation circuit, and a storage circuit; the operation circuit includes a master circuit and a slave circuit; An input circuit for acquiring neural network data; The operation circuit performs run-length all-zero encoding on the neural network data in response to the quantization instruction to obtain run-length compressed data, wherein the run-length all-zero encoding includes run-length encoding only zero characters in the neural network data; Performing Huffman coding on the run-length compressed data, and shifting the leaf node codes in each level of the coding result left by adding 1 from top to bottom, so as to move the leaf nodes in each level of the coding result to the left side of the tree structure, and generating a canonical Huffman coding of the coding result as the compression result of the neural network data; The storage circuit is used to store the compression result; The encoding result is a character sequence, and the operation circuit is used to arrange all characters to be encoded in the character sequence in descending order of appearance frequency to form a first table; Extract all valid code lengths involved in the character sequence, and arrange all valid code lengths in ascending order to form a second table; Extract the sort subscripts of the characters corresponding to the last codeword of each valid code length in the character sequence in the first table to form a third table; Extract the value of the last code word of each valid code length in the character sequence to form a fourth table; Subtract the third table from the fourth table to form a fifth table; Extract characters from the character sequence in order as the current character, obtain the sorting of the current character by accessing the first table; access the third table to obtain the first subscript index greater than or equal to the sorting; Accessing the fifth table and the second table to obtain the base value and code length corresponding to the subscript, adding the base value to the order as the code word value of the current character, and expressing the code word value with the code length as the standard encoding of the current character; The canonical codes of all characters in the character sequence are collected in order to obtain the canonical Huffman code.

4. A storage medium for storing a program for executing the normalized Huffman encoding method according to claim 1 or the normalized Huffman decoding method according to claim 2.

Images