Lz4 text compression method based on sliding dictionary

Abstract

This invention discloses an Lz4 text compression method based on a sliding dictionary, primarily addressing the problems of slow compression speed and difficulty in hardware implementation in current text compression methods. This invention establishes a sliding dictionary, a hash table, and a register for the number of unsuccessful matches. The text to be compressed is read into the sliding dictionary, hash addresses are calculated to store string position information, the sliding dictionary is used to find duplicate strings, and the information of duplicate strings is stored to compress the text information, finally outputting the compressed text stream. The parallel processing of hash address calculation is easily implemented in a pipelined hardware manner. The high probability of finding duplicate strings through pre-prediction reduces the number of hash address calculations. The compression ratio of the text to be compressed is determined by the number of unsuccessful string matches, and the input speed of the text to be compressed is adjusted accordingly, significantly improving the compression speed.

Description

Technical Field

[0001] This invention belongs to the field of communication technology, and further relates to an Lz4 text compression method based on a sliding dictionary in the field of lossless data compression technology. This invention uses a sliding dictionary to find repeating strings to process text information and predicts the position of repeating strings to complete text compression. It can be used to optimize hardware implementation methods for text compression encoding, and is particularly suitable for real-time text compression processing. Background Technology

[0002] With the increasing demand for massive data exchange in computer networks, the requirements for data transmission and storage are becoming increasingly stringent. Compressed data requires significantly less storage space than the original data and consumes less bandwidth during transmission, reducing server traffic and enabling faster transmission. Lz4, currently the most popular lossless compression method, is widely used in areas such as online data downloads and data backup. Lz4 is a compression method that observes and stores the position information of repeated strings by sliding a dictionary and comparing them with the currently viewed string.

[0003] In his paper "Real Time Data Compression: LZ4 Explained" (http: / / fastcompression.blogspot.com / 2011 / 05 / lz4-explained.html, 2011), Yann Collet proposed an Lz4 text compression method. This method is a variant of Lz77. Lz4 is a dictionary-based compression method implemented in software. It scans the text from the beginning, hashes the input 4 bytes to obtain a hash value, and stores and retrieves data addresses using this hash value in a hash table. When two calculated hash values ​​are the same, it indicates that the two input data are highly likely to be identical, thus achieving compression. The drawbacks of this method are: the hash table maintained by Lz4 frequently calculates the hash value of the input text data during encoding, resulting in slow compression speed; the size of the input data is limited by the data width of the hash table, which stores data addresses, and the maximum address value is limited by the data bit width, preventing continuous data compression; and the output delay is uncertain depending on the type of input data, making hardware implementation difficult.

[0004] Xi'an University of Electronic Science and Technology disclosed a text compression method in its patent application "Text Compression Method Based on Gzip Hardware Implementation" (Patent Application No.: 201710255484.8, Publication No.: CN 107135003 A). This method compresses text through dictionary lookup and Huffman coding. The original sequential processing of the text is optimized by trimming overlapping characters and implementing simultaneous string matching within the same window. The drawback of this method is that the dictionary lookup method still uses multiple hash calculations to build the lookup table and find duplicate characters, requiring a large number of clock cycles and resulting in a slow compression speed. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by proposing an Lz4 text compression method based on a sliding dictionary. This method aims to solve the problems of slow text compression speed and difficulty in hardware implementation.

[0006] To achieve the above objectives, the present invention optimizes the existing text compression process by processing the input text in parallel. During the reading of the file to be compressed, the content of the file is monitored by updating the hash table for lookup and comparison. The input data speed is controlled based on the number of unsuccessful matches, and the offset length of the last match is recorded to predict potential matching positions in advance. Finally, the compressed file is output, completing the compression process.

[0007] The specific steps to achieve the purpose of this invention are as follows:

[0008] Step 1: Create a hash table with all storage units containing 0, a sliding dictionary, a register with unsuccessful comparisons, and a compressed storage area;

[0009] Step 2: Read four unread characters sequentially from the text to be compressed to form a string, and store the string in a sliding dictionary; calculate the hash address of the string;

[0010] Step 3: Check if the content of the storage unit corresponding to the hash address is 0. If it is, store the position information of the sliding dictionary containing the string into the storage unit corresponding to the hash address and then execute Step 2; otherwise, execute Step 4.

[0011] Step 4: Address the storage unit content in the hash table to find the corresponding string in the sliding dictionary; compare the last string in the sliding dictionary with the string found by addressing to see if they are equal. If they are equal, the comparison is successful and proceed to step 6; otherwise, the comparison is unsuccessful and proceed to step 5.

[0012] Step 5: Determine if the number of unsuccessful matches in the register plus one is greater than 1% of the total number of characters in the text string to be compressed. If so, read one unread character from the text string to be compressed and store it in the sliding dictionary to speed up the input of the text string to be compressed, and then execute Step 2; otherwise, maintain the input speed of the text string to be compressed and directly execute Step 2.

[0013] Step 6: Clear the number of unsuccessful matches in the register to zero;

[0014] Step 7: Read the next character from the text to be compressed, and read the next character from the end of the position where the sliding dictionary string is successfully matched. Compare whether the two characters are equal. If they are equal, proceed to step 6; otherwise, proceed to step 8.

[0015] Step 8: Save the length of the successfully matched string, the offset length of the successfully matched string, the length of the unmatched string, and the encoding of the unmatched string to the compressed storage area;

[0016] Step 9: Predict the position of the text to be compressed based on the offset length of the previous match.

[0017] Step 9.1: Read one unread character from the text to be compressed and store it in the sliding dictionary to form a new string with the last four characters of the sliding dictionary;

[0018] Step 9.2: Based on the offset length of the successfully matched string in the compressed storage area, predict the position of this match, find the string in the sliding dictionary according to the offset length, and compare whether the two strings are equal. If they are equal, proceed to step 6; otherwise, proceed to step 10.

[0019] Step 10: Determine if there are any characters left in the text to be compressed. If yes, proceed to step 11; otherwise, proceed to step 2.

[0020] Step 11: Output all compressed storage area data.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] First, the present invention adopts a parallel processing method for hash calculation in compressed text data, which overcomes the shortcomings of the slow processing speed caused by the sequential execution structure processing in the prior art, making it easy for the present invention to complete the parallel processing of text compression in a pipeline structure through hardware.

[0023] Secondly, the present invention adopts a sliding dictionary to read the text to be compressed and predicts the matching position of the comparison string. When searching for a match in the text string to be compressed, the matching position of the comparison string is predicted in advance, and the matching strings can be searched continuously. This overcomes the shortcomings of the existing technology, which is slow due to multiple calculations of hash addresses. This invention speeds up the reading speed of the text to be compressed, increases the probability of successful matching of the comparison string, and greatly improves the speed of text compression. Attached Figure Description

[0024] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0025] The following is in conjunction with the appendix Figure 1 The present invention will be further described in detail with reference to the embodiments.

[0026] Step 1: Create a hash table with all storage units containing 0, a sliding dictionary, a register for the number of unsuccessful comparisons, and a compressed storage area.

[0027] In this embodiment of the invention, the hash table has an address width of 11 bits, the sliding dictionary has a set size of 4kb, and the sliding function is achieved by incrementing the address of the sliding dictionary each time the text data to be compressed is read. The register for the number of unsuccessful comparisons has a set size of 32 bits.

[0028] Step 2: Read four unread characters sequentially from the text to be compressed to form a string, and store the selected string in a sliding dictionary. Calculate the hash address of the string.

[0029] In this embodiment of the invention, the size of the text to be compressed is set to 4kb.

[0030] Step 3: Check if the content of the storage unit corresponding to the hash address is 0. If it is, store the position information of the sliding dictionary containing the string into the storage unit corresponding to the hash address and then execute Step 2; otherwise, execute Step 4.

[0031] Step 4: Address the storage unit content in the hash table to find the corresponding string in the sliding dictionary; compare the last string in the sliding dictionary with the string found by addressing to see if they are equal. If they are equal, the comparison is successful and proceed to step 6; otherwise, the comparison is unsuccessful and proceed to step 5.

[0032] Step 5: Determine if the number of unsuccessful matches in the register plus one is greater than 1% of the total number of characters in the text string to be compressed. If so, read one unread character from the text string to be compressed and store it in the sliding dictionary to speed up the input of the text string to be compressed, and then execute Step 2; otherwise, maintain the input speed of the text string to be compressed and directly execute Step 2.

[0033] Step 6: Clear the number of unsuccessful matches in the register to zero.

[0034] Step 7: Read the next character from the text to be compressed, and read the next character from the end of the position where the sliding dictionary string is successfully matched. Compare whether the two characters are equal. If they are equal, proceed to step 6; otherwise, proceed to step 8.

[0035] Step 8: Save the length of the successfully matched string, the offset length of the successfully matched string, the length of the unmatched string, and the encoding of the unmatched string to the compressed storage area.

[0036] Step 9: Predict the position of the text to be compressed based on the offset length of the previous match.

[0037] Read one unread character from the text to be compressed and store it in the sliding dictionary. Combine this character with the last four characters of the sliding dictionary to form a new string. Predict the position of the current match based on the offset length of the successfully matched string in the compressed storage area. Locate the string in the sliding dictionary based on the offset length and compare the two strings to see if they are equal. If they are equal, proceed to step 6; otherwise, proceed to step 10.

[0038] Step 10: Determine if there are any characters left in the text to be compressed. If yes, proceed to step 11; otherwise, proceed to step 2.

[0039] Step 11: Output all compressed storage area data.

[0040] The present invention will be further described below with reference to the embodiments.

[0041] The text data to be compressed is 01234567012345678.

[0042] The compression process is as follows:

[0043] Establish a hash table, a sliding dictionary, a register for the number of unsuccessful matches, and a compressed storage area. The hash table contains all 0s, the sliding dictionary contains 0s, the register for the number of unsuccessful matches contains 0s, and the compressed storage area contains all 0s.

[0044] Four characters, 0123, are read from the text to be compressed and stored in the sliding dictionary. The hash address of 0123 is calculated as follows: 0x0123 * 2654435761 = 0xD90F5433, 0x9E3779B1 >> 21 = 0x6C8. The resulting hash address is 0x6C8. At this point, the sliding dictionary data is 0123, the text to be compressed is 4567012345678, all addresses in the hash table are filled with 0, and the value of the unmatched count register is 0.

[0045] The hash address is used to find the corresponding storage content in the hash table, which is 0. The current position information of the string is 1, and it is stored in the storage content at hash address 0x6C8 of the hash table. At this time, the sliding dictionary data is 0123, the text to be compressed is 4567012345678, the storage content at hash address 0x6C8 is 1, the storage content at other addresses is 0, and the value of the unmatched count register is 1.

[0046] Four characters, 4567, are read from the text to be compressed, and the hash address of 4567 is calculated to be 0x4E0. At this time, the sliding dictionary data is 01234567, the text to be compressed is 012345678, and the value of the unmatched count register is 1.

[0047] Using hash address 0x4E0, the corresponding storage content in the hash table is found to be 0. The current position information of the string is 2, which is stored in the storage content at hash address 0x4E0 of the hash table. At this time, the sliding dictionary data is 01234567, the text to be compressed is 012345678, the storage content at 0x6C8 in the hash table is 1, the storage content at 0x4E0 is 2, the storage content at the other addresses is 0, and the value of the unsuccessful comparison count register is 2.

[0048] Four characters, 0123, are read from the text to be compressed. The hash address of 0123 is calculated to be 0x6C8. Using hash address 0x6C8, the corresponding storage content in the hash table is found to be 1. The string 0123 at position 1 in the sliding dictionary is found, and it matches successfully with 0123 in the text to be compressed. At this point, the sliding dictionary data is 01234567, the text to be compressed is 012345678, the storage content at 0x6C8 in the hash table is 1, the storage content at 0x4E0 is 2, the storage content at the other addresses is 0, and the value of the unmatched count register is 0.

[0049] The process continues with individual character comparisons. The characters 4, 5, 6, and 7 in the sliding dictionary are successfully matched with the characters 4, 5, 6, and 7 in the text to be compressed, until the character 0 in the sliding dictionary fails to match with the character 8 in the text to be compressed.

[0050] The length of the successfully matched string is 8, the offset length of the successfully matched string is 8, the length of the unmatched string is 9, and the unmatched string is 012345678. Save (8, 8, 9) and 012345678 to the compressed storage area.

[0051] The text to be compressed contains no strings, so the compression is complete. The output is (8, 8, 9) and 012345678.

Claims

1. An implementation of LZ4 text compression based on sliding dictionary, characterized in that, The text compression method comprises the following steps: Step 1, establishing a hash table with storage units whose contents are all 0, a sliding dictionary, a register of unmatching successful times, and a compressed storage area; Step 2, reading 4 characters from the text to be compressed to form a string, and storing the string in the sliding dictionary; Step 3, checking whether the content of the storage unit corresponding to the hash address is 0, if yes, storing the position information of the string in the sliding dictionary in the storage unit corresponding to the hash address, and then executing step 2; otherwise, executing step 4; Step 4, finding the corresponding string in the sliding dictionary through content addressing of the storage unit in the hash table; comparing the last string in the sliding dictionary with the found string, if they are equal, executing step 6; otherwise, executing step 5; Step 5, judging whether the number of unmatching successful times in the register plus 1 is greater than 1% of the total number of strings in the text to be compressed, if yes, reading 1 character from the text to be compressed and storing it in the sliding dictionary to speed up the input of the text to be compressed, and then executing step 2; otherwise, keeping the input speed of the text to be compressed, and then executing step 2 directly; Step 6, clearing the number of unmatching successful times in the register; Step 7, reading a subsequent character from the text to be compressed and a subsequent character from the end of the string in the sliding dictionary, and comparing the two characters, if they are equal, executing step 6; otherwise, executing step 8; Step 8, saving the length of the string matched successfully, the offset length of the string matched successfully, the length of the string not matched successfully, and the encoding of the string not matched successfully to the compressed storage area; Step 9, predicting the matching position of the text to be compressed according to the offset length of the last matching; Step 9.1, reading 1 character from the text to be compressed and storing it in the sliding dictionary to form a new string with the last 4 characters in the sliding dictionary; Step 9.2, predicting the matching position this time through the offset length of the string matched successfully in the compressed storage area, finding the string in the sliding dictionary through the offset length, and comparing the two strings, if they are equal, executing step 6; otherwise, executing step 10; Step 10, judging whether there are still characters in the text to be compressed, if yes, executing step 11; otherwise, executing step 2; Step 11, outputting all data in the compressed storage area. ​

Images