FPGA-based multipath parallel ZSTD data compression method and system

Abstract

The invention relates to a multi-channel parallel ZSTD data compression method and system based on an FPGA. The method comprises the steps that input data to be compressed are segmented according to a set block size; taking a character in the parallel processing window as a first character, taking a plurality of continuous characters after the first character is extracted as character strings, then moving the parallel processing window by a distance of one window length, and continuing to extract the character strings; the hash value of each character string is calculated in parallel, and the character strings are stored according to the hash values; matching the character string extracted from the current parallel processing window with a stored character string, determining a longest matching sequence in the character string extracted from the current parallel processing window, and trimming a matched and overlapped part in the longest matching sequence; the longest matching sequence is converted into triple data, FSE coding is carried out on the triple data, and Huffman coding is carried out on unmatched characters in character strings extracted by the parallel processing window; and integrating and packaging results of the FSE coding and the Huffman coding according to a ZSTD data format, and outputting final coded data.

Description

Technical Field

[0001] This invention relates to a multi-channel parallel ZSTD data compression method and system based on FPGA, belonging to the field of data compression. Background Technology

[0002] As the digital wave sweeps across the world, data is ubiquitous and has become central to driving applications and decision-making. This rapid growth in data volume forces people to rethink how to efficiently process and utilize this data; simply relying on increasing computing power and expanding storage resources is no longer sufficient to meet the requirements of modern data centers and applications.

[0003] Data compression technology, a core research area in information science, aims to efficiently represent data through encoding strategies to reduce its storage and transmission space requirements. Therefore, efficient data compression technologies are crucial, as they not only optimize resource utilization but also ensure the responsiveness and performance of data applications, meeting ever-increasing business demands. ZSTD (Zstandard, a data compression algorithm), as a modern lossless data compression algorithm, integrates the LZ77 algorithm (a data compression algorithm proposed by Abraham Lempel and Jacob Ziv in 1977), Huffman coding (a data compression algorithm), and FSE (First Stage Encoder) coding algorithms, achieving an optimized balance between high compression ratio and high decompression speed. The ZSTD algorithm first divides the source data into blocks, and then uses individual data blocks as the smallest processing unit during subsequent compression. For each compressed block, character matching is first performed using the LZ77 algorithm. Unmatched text data is then Huffman encoded, while matched data is FSE encoded. Finally, the complete compressed result is constructed according to the ZSTD compression format.

[0004] Due to the inherently serial nature of software, the software-based ZSTD algorithm can only be executed sequentially when processing large-scale data, which to some extent limits performance improvement. Summary of the Invention

[0005] The purpose of this invention is to provide a multi-channel parallel ZSTD data compression method and system based on FPGA, in order to solve the problem that the performance of software-based ZSTD algorithms is difficult to improve when processing large-scale data due to the inherent serial nature of software.

[0006] To achieve the above objectives, the present invention includes:

[0007] The present invention provides a multi-channel parallel ZSTD data compression method based on FPGA, comprising the following steps:

[0008] 1) Divide the input data to be compressed into segments according to the set block size;

[0009] 2) Process each data block using a parallel processing window: Starting with the character within the parallel processing window, extract the next few consecutive characters as a string. The string length is the same as the parallel processing window length. Then, move the parallel processing window by one window length and continue extracting strings. Calculate the hash value of each string in parallel and store the string based on the hash value. Before moving the parallel processing window a second time, match the string extracted by the current parallel processing window with the stored strings to determine the longest matching sequence with the most consecutive identical characters in the currently extracted string. Trim the overlapping parts of the current longest matching sequence with the string extracted by the previous parallel processing window. If the trimmed longest matching sequence meets the preset length requirement, convert the longest matching sequence into triplet data and perform FSE encoding. Perform Huffman encoding on the non-matching characters in the currently extracted string.

[0010] The results of FSE encoding and Huffman encoding are integrated and packaged according to the ZSTD data format, and the final encoded data is output.

[0011] Furthermore, in step 2), when calculating the hash value of each string in parallel, the hash value is obtained by performing an XOR operation on the first set number of characters in the string.

[0012] Furthermore, in step 2), the storage location of the corresponding string in the hash dictionary is calculated based on a portion of the hash value, the corresponding string is stored in that storage location, and the hash dictionary is dynamically updated based on the matching result when matching strings to ensure that the content of the hash dictionary is the latest processed data.

[0013] Furthermore, if different strings are stored in the same storage location when calculating the storage location of the corresponding string in the hash dictionary based on a part of the hash value, then the string corresponding to the character at the beginning of the current parallel processing window will be stored first.

[0014] Further, in step 2), when matching the string in the current parallel processing window with the stored strings, a portion of the hash value of the current string is used as an index, and storage units at different depth levels in the hash dictionary are accessed in parallel. Within a single clock cycle, candidate matching strings corresponding to different depth levels are retrieved, and the retrieved strings and position information are stored in a temporary structure for matching. During matching, the characters in each string in the current parallel processing window are compared with the characters in each candidate matching string in the temporary structure. If the same characters are found in the two compared strings, they are considered to be matched; otherwise, they are considered not to be matched.

[0015] Further, in step 2), the characters in the current parallel processing window other than the longest matching sequence converted into triplet data are sent to the Huffman encoding buffer. Characters that are considered to be mismatched are Huffman encoded, while characters that are considered to be matched are not Huffman encoded.

[0016] Furthermore, during Huffman encoding, the length of the parallel processing window buffered in the Huffman buffer is processed in parallel.

[0017] Furthermore, when outputting the corresponding Huffman encoded stream, the Huffman encoded result and the matched characters in the Huffman encoder that are not Huffman encoded are integrated into the set byte limit before being packaged and output.

[0018] Furthermore, in step 2), when performing FSE encoding, the triplet data is first mapped, then the mapped triplet data is encoded, and then the FSE encoding results are merged into an FSE encoded stream for output.

[0019] The present invention discloses an FPGA-based multi-channel parallel ZSTD data compression system, comprising a processor for executing instructions for implementing an FPGA-based multi-channel parallel ZSTD data compression method.

[0020] The beneficial effects of this invention are as follows:

[0021] This invention is a pioneering creation. It presents an FPGA-based multi-channel parallel ZSTD data compression method and system. In the character matching stage, by simultaneously performing character matching at multiple consecutive positions within a parallel window, a 16-channel parallel LZ77 character matching hardware architecture is proposed, solving the problem of low efficiency in traditional single-channel character matching. A hash dictionary with 32 banks and a 3-level depth is designed and the hash function is optimized, resolving character storage and storage conflict issues in parallel matching, improving matching coverage, and thus enhancing the compression ratio. In the character encoding stage, a 16-channel parallel Huffman static encoding hardware architecture is proposed, solving the problem of low encoding efficiency of the 16 unmatched character data streams output by the LZ77 module. This architecture achieves parallel data stream processing by deploying 16 independent encoders, and then utilizes shift registers to merge encoding results, significantly improving encoding speed and achieving a 16-fold increase in speed compared to traditional single-channel buffered encoding. FPGA platforms have the ability to execute in parallel, which can significantly improve data processing speed and ensure real-time response. Implementing the ZSTD algorithm on an FPGA platform can not only significantly accelerate the compression process and improve ZSTD compression performance by leveraging its hardware parallelism, but also achieve better performance in terms of power consumption, flexibility and response latency. It has important application value in fields such as data centers, cloud storage optimization and real-time video processing. Attached Figure Description

[0022] Figure 1 This is the overall architecture of the ZSTD data compression system of the present invention;

[0023] Figure 2 This is a diagram illustrating string partitioning;

[0024] Figure 3 This is a schematic diagram of hash storage;

[0025] Figure 4 This is a diagram illustrating a character storage conflict.

[0026] Figure 5 This is a schematic diagram of a hash dictionary structure;

[0027] Figure 6 This is a diagram illustrating byte-by-byte matching;

[0028] Figure 7 This is a bit reversal diagram;

[0029] Figure 8 This is a diagram illustrating the matching length;

[0030] Figure 9 This is a diagram illustrating trimming and matching overlaps. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0032] The present invention provides a multi-channel parallel ZSTD data compression method and system based on FPGA, which divides the data into 16 parallel windows and processes 16 strings simultaneously. By using a 16-channel parallel LZ77 character matching architecture and a Huffman static encoding architecture, the traditional single-channel matching and encoding speed is increased by 16 times, thus achieving 16-channel parallel ZSTD lossless data compression.

[0033] Method Implementation Examples:

[0034] like Figure 1 As shown, the FPGA-based multi-channel parallel ZSTD data compression method includes:

[0035] 1) The data partitioning module divides the input data to be compressed into blocks according to the set block size, ensuring that only a single data block is processed at a time. To facilitate knowing the end of each data block and processing the next block, each data block is structured. Each data block is divided into a data field and a flag field. The data field stores the data, and the flag field indicates whether the data is valid. The last data flag bit of each data block is set to 0, and the flag fields of the remaining fields are set to 1. For the last data block, the last data flag bit is set to 0, followed immediately by another data flag that is also 0, to indicate the end of the entire compressed file.

[0036] 2) The LZ77 character matching module is used to match these segmented data blocks. By replacing duplicate data fragments with triple sequences, preliminary data compression is achieved, generating unmatched text data and matched triple sequence data, specifically including:

[0037] 201) The hash value calculation submodule calculates the hash values ​​of 16 independent string segments in parallel:

[0038] like Figure 2 As shown, the data block processed in step 1) is divided into 16 strings according to bytes. The length of each processing window is 16, and each processing window corresponds to 16 consecutive characters. These 16 consecutive characters are the first characters of the 16 strings corresponding to the current processing window. The hash values ​​of the 16 strings are calculated in parallel using the following hash function:

[0039] hash=curr_window[i]xor curr_window[i+1]xor curr_window[i+2]xor curr_window[i+3]xor curr_window[i+4]

[0040] Where curr_window[i] represents the i-th character data of the current processing window, the hash value is calculated using the first five characters of the string, and xor represents the exclusive OR operation; after calculation by this function, a 14-bit hash result is obtained for each string; such as Figure 3 As shown, the specific storage location of each string in the hash dictionary is determined based on its corresponding hash value. First, the high 5 bits of the string's hash value determine which Bank (a Bank refers to a storage unit in the hash dictionary) the string should be stored in. Then, the low 9 bits of the hash value specify the exact index position of the string within its respective Bank. For example... Figure 4 As shown, for multiple strings that need to access the same Bank, a number priority strategy is adopted, that is, only the string with the smaller number is stored in the Bank.

[0041] Hash dictionary structure as follows Figure 5 As shown, the Bank employs a three-level deep caching structure. The first-level Bank focuses on storing the nearest matching candidate to the current string, performing a match with minimal positional difference. As the Bank level increases, the second-level Bank records relatively distant matching candidates, i.e., strings whose distance exceeds the matching range of the first-level Bank. The third-level Bank further broadens the search domain, covering the third nearest matching candidate.

[0042] 202) The string matching submodule performs character matching on the current window string to determine the optimal matching string:

[0043] The hash value of the current string is used as an index, and parallel access operations are performed on the three levels of the Bank. Candidate matching strings at the three levels of distance are retrieved within a single clock cycle. The retrieved strings and position information are then stored in a temporary structure called "potential match" for matching comparison in subsequent processing steps.

[0044] Each part of the dictionary is examined, and a flag array is used to determine which banks need to be updated. For banks marked as needing updates, the new string data and its corresponding position information are first stored at the top level of the dictionary; then, each level of the bank is updated through an iterative process, passing the string and position information from the previous level down to the next level.

[0045] For each potential match in the dictionary, compare it in detail with every byte of the current window data, such as... Figure 6 As shown, in this comparison mechanism, if the bytes of two sequences are the same at a specific position, that position is marked as "1" (match); otherwise, it is marked as "0" (mismatch). Based on this matching flag, for each Bank and its various depth levels within the dictionary, such as... Figure 7 The bitwise operation shown determines the total number of consecutive matching bytes from the beginning of the string to the first unmatched byte, thus calculating the string's matching length. The resulting match is shown below. Figure 8 As shown, the longest matching sequence is determined by comparing each depth level within each Bank.

[0046] Filter out matching sequences whose length does not meet the minimum matching length; define the string extension length as the sum of the string's starting byte position and its matching length, calculated using the following formula:

[0047] reach[i]=str_pos[i]+match_len[i]

[0048] Where reach[i] represents the extension length of string i, str_pos[i] represents the starting byte position of string i, and match_len[i] represents the matching length of string i. For strings with no match, its match_len value is 1; for strings with a match, its match_len represents the actual length of the matched string.

[0049] By comparing the extension positions of the 16 strings in the parallel window, the string that extends the furthest is selected and determined as the optimal matching string (i.e., the longest matching string) within that window.

[0050] 203) Eliminate matching overlaps between optimal matches for each string using the matching filtering submodule:

[0051] After performing longest match filtering on strings in a parallel window, the window currently being processed is marked as the current window, and the window processed in the previous cycle is considered the previous window. When a string match extends into the current window, the matching information from the previous window is preserved, while strings in the current window that overlap with matches in the previous window are appropriately pruned, retaining only the non-overlapping parts. For example... Figure 9 As shown, the specific steps are as follows:

[0052] In the current parallel processing window Window i, if the matching extension position of string Str 4 is the farthest, and the first valid position where the current window can start matching processing is set to first_pos, then (according to...) Figure 9 As shown, if the setting value of the first valid position is no greater than 7, then the matching start position of Str 4 needs to be trimmed from byte 4 to byte 6. After trimming, if the matching length is adjusted from the original 7 to 5, and this length exceeds the minimum matching length standard, then it is considered a valid match.

[0053] 204) The matching output submodule converts the processed data into the final output format to facilitate subsequent encoding:

[0054] Based on the aforementioned matching marker array, determine which characters are the matched string parts and which are unmatched characters. If the value in the marker array corresponding to a character is 01, it indicates that the character is an unmatched character. If the value is 10, it indicates that the character is the first matched character. If the value is 00, it indicates that the character is one of the remaining matched characters.

[0055] When a character is an unmatched character or one of the remaining unmatched characters, it is directly output to a 144-bit buffer and finally to `lit_stream_data`. The 144-bit buffer buffers 16 characters, each occupying 9 bits. The lower 8 bits represent the character data, and the higher 9 bits represent whether the character is an unmatched character. If it is, the value is 1; otherwise, it is 0. This facilitates the subsequent Huffman module to perform Huffman encoding based on the value of the higher 9 bits. In addition, when a character is an unmatched character, the unmatched characters need to be counted and accumulated. The value is assigned to `ll`, which represents the length of the previous unmatched character. When the character is the first matched character, a triplet (ll, ml, of) is calculated, where ml represents the length of the matched data and of represents the offset of the matched character. The triplet data is output to a 40-bit buffer and finally to `seq_stream_data` for subsequent processing by the FSE module.

[0056] 3) Unmatched text data is sent to the Huffman encoding module for Huffman encoding, specifically including:

[0057] 301) The unmatched character data stream obtained from the LZ77 character matching module is encoded in parallel using the parallel encoding submodule:

[0058] Efficient encoding is performed on a 144-bit unmatched character data stream obtained from the LZ77 compression algorithm. This data stream consists of 16 independent characters, each containing 8 bits of data and a 1-bit flag, the latter used to indicate whether the character matches. If the flag bit is 1, it indicates that the character is not matched and needs to be processed by Huffman coding; otherwise, characters with a flag bit of 0 are considered to be matched and do not require encoding.

[0059] 302) The Huffman encoded results of varying lengths are combined to a byte boundary using the encoded stream merging submodule:

[0060] During Huffman encoding, firstly, the encoding segment of each character is located in the encoding table based on its ASCII value. Next, the encoding value is adjusted according to the segment of the character's ASCII value: for characters with ASCII values ​​in the range of 0 to 143, the ASCII value is increased by 48 to obtain the character's encoding value; for characters with ASCII values ​​in the range of 144 to 255, the ASCII value is increased by 256 to obtain the character's encoding value. Finally, the adjusted encoding value is converted into binary data of a predetermined encoding length for the corresponding segment. A 144-bit shift register is used to pack the Huffman encoding results, outputting a 128-bit encoded stream huf_code_data.

[0061] 4) The matched triplet sequence data is sent to the FSE encoding module for FSE encoding, specifically including:

[0062] 401) The sequence mapping submodule is used to map the matching triplet data output by the LZ77 character matching module:

[0063] The mapping processing performed on the matching triplet data (ll, of, ml) output by the LZ77 character matching module is replaced with a new triplet sequence (ll_mape, of_mape, ml_mape) represented using fewer bits.

[0064] 402) Encoding is performed by the encoding submodule based on the mapped triples:

[0065] The current state (ll_state, ml_state, of_state) and the corresponding output bit width (ll_nbBitsOut, ml_nbBitsOut, of_nbBitsOut) are determined based on the mapped triples (ll_mape, ml_mape, of_mape).

[0066] 403) The bitstream merging submodule performs a masking operation based on the triplet sequence values, corresponding offsets, state values, and output bit widths to output the encoded result:

[0067] The lower ll_bias bits of the binary representation of the ll value are assigned to ll_mape_bits, and the lower ll_nbBitsOut bits of the binary representation of the ll_state are assigned to ll_stream_bits. The bitstreams generated by these two operations are then merged to form ll_encode_bits. The same process is applied to the ml and of values, generating the corresponding ml_encode_bits and of_encode_bits values. The ll_encode_bits, ml_encode_bits, and of_encode_bits values ​​are merged and stored in encode_bit. These encoding results are then merged in a specific order to form a result_code for output.

[0068] 404) The bit stream output submodule merges the data received from the bit stream merging module into an encoded stream for output:

[0069] The result_code data received from the bitstream merging module is merged into a 128-bit encoded stream fse_code_data and then output.

[0070] 5) Combine the huf_code_data output by the Huffman encoding module and the fse_code_data output by the FSE encoding module into the data packaging module, integrate and package them according to the ZSTD data format, and output the final encoded data.

[0071] System Implementation Example:

[0072] This embodiment provides an FPGA-based multi-channel parallel ZSTD data compression system, including a processor for executing instructions to implement the FPGA-based multi-channel parallel ZSTD data compression method. The FPGA-based multi-channel parallel ZSTD data compression method has been described clearly enough in the method embodiment and will not be repeated here.

Claims

1. A multi-channel parallel ZSTD data compression method based on FPGA, characterized in that, Includes the following steps: 1) Divide the input data to be compressed into segments according to the set block size; 2) Process each data block using a parallel processing window: Taking the character within the parallel processing window as the first character, extract several consecutive characters following the first character as a string. The length of the string is the same as the window length of the parallel processing window. Then, move the parallel processing window by one window length and continue extracting the string. Calculate the hash value of each string in parallel and store the string based on the hash value. Before moving the parallel processing window a second time, match the string extracted by the current parallel processing window with the stored string to determine the longest matching sequence in the string extracted by the current parallel processing window that has the most consecutive identical characters with the stored string. Then, trim the overlapping parts of the current longest matching sequence with the string extracted by the previous parallel processing window. If the trimmed longest matching sequence meets the preset length requirement, convert the longest matching sequence into triplet data and perform FSE encoding. Perform Huffman encoding on the non-matching characters in the string extracted by the current parallel processing window. 3) Integrate and package the results of FSE encoding and Huffman encoding according to the ZSTD data format and output the final encoded data.

2. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 1, characterized in that, In step 2), when calculating the hash value of each string in parallel, the hash value is obtained by performing an XOR operation on the first set number of characters in the string.

3. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 1, characterized in that, In step 2), the storage location of the corresponding string in the hash dictionary is calculated based on a portion of the hash value, the corresponding string is stored in that storage location, and the hash dictionary is dynamically updated based on the matching result when matching strings to ensure that the content of the hash dictionary is the latest processed data.

4. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 3, characterized in that, If the storage location of the corresponding string in the hash dictionary is calculated based on a portion of the hash value, and different strings are stored in the same storage location, then the string corresponding to the character at the beginning of the current parallel processing window will be stored first.

5. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 3, characterized in that, In step 2), when matching the string in the current parallel processing window with the stored string, a portion of the hash value of the current string is used as an index. At the same time, storage units at different depth levels in the hash dictionary are accessed in parallel. Within a single clock cycle, candidate matching strings corresponding to different depth levels are retrieved, and the retrieved strings and position information are stored in a temporary structure for matching. During matching, the characters in each string in the current parallel processing window are compared with the characters in each candidate matching string in the temporary structure. If the same characters are found in the two compared strings, they are considered to be matched; otherwise, they are considered not to be matched.

6. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 5, characterized in that, In step 2), all characters in the current parallel processing window except for the longest matching sequence converted into triplet data are sent to the Huffman encoding buffer. Characters that are considered to be mismatched are Huffman encoded, while characters that are considered to be matched are not Huffman encoded.

7. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 6, characterized in that, During Huffman encoding, the length of the parallel processing window buffered in the Huffman buffer is processed in parallel.

8. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 6, characterized in that, When outputting the corresponding Huffman encoded stream, the Huffman encoded result and the matched characters in the Huffman encoder that are not Huffman encoded are integrated into the set byte limit before being packaged and output.

9. The FPGA-based multi-channel parallel ZSTD data compression method according to claim 1, characterized in that, When performing FSE encoding in step 2), the triplet data is first mapped, then the mapped triplet data is encoded, and then the FSE encoding results are merged into an FSE encoded stream for output.

10. A multi-channel parallel ZSTD data compression system based on FPGA, comprising a processor, characterized in that, The processor is used to execute instructions for implementing the FPGA-based multi-path parallel ZSTD data compression method as described in any one of claims 1 to 8.

Images