SNAPPY compression algorithm hardware accelerator

Abstract

The application discloses a SNAPPY compression algorithm hardware accelerator, comprising: a length generation module for receiving to-be-compressed data, obtaining a first address of the to-be-compressed data, a length of the to-be-compressed data and a compressed write-back data first address, sending the first address of the to-be-compressed data and the length of the to-be-compressed data to a reading module, sending the compressed write-back data first address to a write-back module, and converting the length of the to-be-compressed data into a preset format; the reading module is used for reading the to-be-compressed data in the length generation module and transmitting the read to-be-compressed data to a compression module; the compression module is used for compressing the to-be-compressed data transmitted by the reading module and transmitting the compressed data to the write-back module; and the write-back module is used for receiving and buffering the compressed data from the compression module and sending the compressed data to a host end. The SNAPPY compression is completed by using the hardware flow water parallel processing capacity, the CPU is not occupied for compression, and therefore the compression speed is fast and the efficiency is high.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit technology, and in particular to a hardware accelerator for the SNAPPY compression algorithm. Background Technology

[0002] The Snappy compression algorithm is a commonly used compression algorithm in big data applications. It compresses data in bytes as the smallest unit, balancing compression ratio and speed, achieving a relatively fast compression speed while maintaining a considerable compression ratio. The main principle of Snappy compression is to compare subsequent data with preceding data. If duplicate data segments are found, they are replaced with an offset and a length. The offset refers to the distance between the found duplicate data segment and the current data, and the length refers to the length of the duplicate data segment. By storing such an (offset, length) data pair to replace the longer duplicate data segment, compression is achieved. For the parts where no duplicate data is found, the original data is retained.

[0003] Currently, the SNAPPY compression algorithm divides data into compressed blocks of 64KB each, and each compressed block is compressed independently. That is, during the compression process, it will not cross the 64KB boundary to search for duplicate data segments, nor will it use metadata that crosses the 64KB boundary to generate data tags.

[0004] Existing SNAPPY compression algorithm software solutions consume significant CPU computing resources during operation, impacting the performance of the SNAPPY compression algorithm and the overall performance of the computer system. Overcoming this problem of excessive CPU resource consumption by SNAPPY compression algorithm software solutions is a technical issue that needs to be addressed. Summary of the Invention

[0005] To address some or all of the technical problems existing in the prior art, this invention provides a hardware accelerator for the SNAPPY compression algorithm. When compressing data, it is implemented by hardware circuitry, without occupying the CPU for compression, resulting in fast compression speed and high efficiency.

[0006] The technical solution of the present invention is as follows:

[0007] A hardware accelerator for the SNAPPY compression algorithm, comprising:

[0008] The length generation module is used to receive the data to be compressed, obtain the starting address of the data to be compressed, the length of the data to be compressed and the starting address of the compressed write-back data, send the starting address of the data to be compressed to the reading module, send the starting address of the compressed write-back data to the write-back module, convert the length of the data to be compressed into a preset format and then send the data in the preset format to the write-back module.

[0009] The reading module is used to read the data to be compressed from the host, receive the starting address and length of the data to be compressed from the length generation module, and transmit the read data to the compression module.

[0010] A compression module is used to compress the data to be compressed transmitted by the reading module and transmit the compressed data to the write-back module.

[0011] The write-back module is used to receive the length of the data to be compressed after being converted according to a preset format from the length generation module and the data compressed by the compression module, and to cache both types of data and send them to the host.

[0012] Furthermore, the preset format is a variant little-endian format.

[0013] Furthermore, the reading module includes a reading generation module, a reading control module, and a reading cache module.

[0014] The read generation module is used to cache the starting address and length of the data to be compressed transmitted by the length generation module, and generate a series of access requests starting from the starting address of the data to be compressed with an access length equal to the length of the data to be compressed, thereby completing the reading of the data to be compressed from the host.

[0015] The read control module is used to complete the interaction with the host and generate control signals to control the read process of the read module;

[0016] The read cache module is used to cache data read from the host and send the cached data to the compression module for compression.

[0017] Furthermore, the compression module includes: a string matching module, a copy marker generation module, and a data marker generation module.

[0018] The string matching module is used to cache the data of the reading module and write it into a preset data history table for storage. At the same time, the string matching module is used to perform hash operation on the cached data, read the corresponding hash item from the preset hash matching table according to the hash operation result for matching and comparison to find duplicate strings, and update the hash matching table as a new hash table item according to the hash operation result.

[0019] The copy marker generation module is used to complete the longest string matching and copy marker generation after the string matching module successfully matches the string;

[0020] The data tag generation module is used to process data for which no duplicate strings are found and generate data tags.

[0021] Furthermore, after the string matching module successfully matches a string, the method for completing the longest string matching and generating the copy marker is as follows:

[0022] After a successful initial string match, the string matching module directly passes the data obtained from the reading module to the copy marker generation module, which caches it. Simultaneously, the copy marker generation module uses the matching position segment of the corresponding hash table entry as the read address of the data history table, reads one byte of data from the data history table, and compares it with the cached data. If they are equal, the match is successful; otherwise, the match fails. The copy marker generation module also sets a longest match length counter for matching and copy marker generation.

[0023] Furthermore, at most one of the data tag generation module and the copy tag generation module can send data to the write-back module at any given time.

[0024] Furthermore, the data cached in the data tag generation module is no more than 60 bytes.

[0025] Furthermore, the write-back module includes a write-back data cache module, a write-back generation module, and a write-back control module.

[0026] The write-back cache module is used to cache the length of the data to be compressed after being converted according to a preset format and the compressed data transmitted in the compression module.

[0027] The write-back generation module is used to receive the starting address of the compressed write-back data transmitted by the length generation module, and generate a continuous write access request starting from the starting address of the compressed write-back data, thereby completing the write-back of the data in the write-back cache module to the host.

[0028] The write-back control module is used to interact with the host and generate control signals to control the write-back process of the write-back module.

[0029] Furthermore, the method for converting the length of the data to be compressed into a preset format is as follows:

[0030] Arrange the data to be compressed in a preset format;

[0031] The sorted data is divided into several fields starting from the least significant digit and according to a preset number of digits.

[0032] After padding the divided fields with high-order bits, they are arranged according to a preset format to complete the conversion of the length of the data to be compressed into a preset format.

[0033] The main advantages of the technical solution of this invention are as follows:

[0034] This invention discloses a hardware accelerator for the SNAPPY compression algorithm. When processing compressed data, it implements the SNAPPY compression algorithm through hardware circuitry, utilizing hardware pipelined parallel processing capabilities to complete the SNAPPY compression operation. Data tag processing and copy tag processing are performed by different hardware components, without interference. Data tags with a length exceeding 60 are split into multiple data tags with a length of 60 or less, reducing hardware complexity. The pipelined implementation allows data reading, compression, and write-back to occur simultaneously, further increasing the parallelism of data processing. This results in fast compression speed, high efficiency, and low CPU resource consumption. Attached Figure Description

[0035] The accompanying drawings, which are included to provide a further understanding of embodiments of the invention and constitute a part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0036] Figure 1 This is a schematic diagram of the overall structure of a hardware accelerator for the SNAPPY compression algorithm according to the present invention;

[0037] Figure 2 This is a schematic diagram of the compression module in a hardware accelerator for the SNAPPY compression algorithm of the present invention;

[0038] Figure 3 This is a schematic diagram of the reading module in a hardware accelerator for the SNAPPY compression algorithm of the present invention;

[0039] Figure 4 This is a schematic diagram of the write-back module in a hardware accelerator for the SNAPPY compression algorithm of the present invention.

[0040] Explanation of reference numerals in the attached figures:

[0041] 1. Length generation module; 2. Reading module; 3. Compression module; 4. Write-back module; 21. Read generation module; 22. Read control module; 23. Read cache module; 31. String matching module; 32. Copy marker generation module; 33. Data marker generation module; 41. Write-back data cache module; 42. Write-back generation module; 43. Write-back control module. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0043] The technical solutions provided by the embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0044] Specifically, the present invention provides a hardware accelerator for the SNAPPY compression algorithm. The SNAPPY compressed file consists of the original length, several copy markers, and several data markers. The original length is represented in little-endian variant format; the copy markers consist of metadata, offset, and copy length, representing duplicate data segments; and the data markers consist of metadata, data length, and data, representing non-duplicate data segments.

[0045] Specifically, the uncompressed length represented in variant little-endian format consists of a series of bytes arranged in little-endian. The last 7 bits of each byte are the numeric bits. The first bit of each byte is a flag bit, used to indicate whether the next byte belongs to this value. If the first bit of the byte is 1, it means that the next byte also belongs to this value; if the first bit is 0, the next byte does not belong to this value, that is, this byte is the next byte in the group of this value. For example, 2097149 (0x1FFFFD) should be represented in variant little-endian format as 0xFD0xFF 0x7F.

[0046] Specifically, depending on the data length, there are two formats for data tags. Format 1 can represent data lengths from 1 to 60. This data tag consists of one metadata byte and several bytes of data. The first 6 bits of the metadata byte indicate the data length minus 1, and the last 2 bits are fixed as "00". The metadata byte is followed by the data itself, and the length of the data is specified by the data length.

[0047] Specifically, format two can represent data tags with a length greater than 60 bytes. This data tag consists of one metadata byte, several data length bytes, and several bytes of data. The first four bits of the metadata byte are fixed as "1111" and the last two bits are fixed as "00". The remaining two middle bits are used to indicate the number of bytes in the data length byte, where "00" represents 1 byte, "01" represents 2 bytes, "10" represents 3 bytes, and "11" represents 4 bytes. The data length byte indicates the data length minus 1 and immediately follows the metadata byte, arranged in little-endian format. The data itself follows the data length byte, and the length of the data is specified by the data length byte.

[0048] Specifically, there are three formats for the copy marker. Format 1 is 2 bytes long. The first byte is the metadata byte, and the second byte is the offset byte. The first 3 bits of the metadata byte and all 8 bits of the offset byte form an 11-bit data set representing the offset. Bits 4 to 6 of the metadata represent the copy length minus 4, and the last two bits of the metadata are always "01". Format 2 is 3 bytes long. The first byte is the metadata byte, and the second and third bytes are the offset bytes. The first 6 bits of the metadata represent the copy length minus 1, and the last two bits are always "10". The two offset bytes represent the offset (in little-endian format). Format 3 is 5 bytes long. The first byte is the metadata byte, and bytes 2 to 5 are the offset bytes. The first 6 bits of the metadata represent the copy length minus 1, and the last two bits are always "11". The four offset bytes represent the offset (in little-endian format).

[0049] As attached Figure 1-4 As shown, this embodiment of the invention provides a hardware accelerator for the SNAPPY compression algorithm, the accelerator comprising:

[0050] Length generation module 1 is used to receive the data to be compressed, obtain the starting address of the data to be compressed, the length of the data to be compressed, and the starting address of the compressed write-back data. It sends the starting address of the data to be compressed and the length of the data to be compressed to the reading module 2, sends the starting address of the compressed write-back data to the write-back module 4, converts the length of the data to be compressed into a preset format, and sends it to the write-back module 4.

[0051] Reading module 2 is used to read the data to be compressed in the host, and receive the starting address and length of the data to be compressed from the length generation module 1, and transmit the read data to the compression module 3.

[0052] Compression module 3 is used to compress the data to be compressed transmitted by reading module 2 and transmit the compressed data to write-back module 4.

[0053] Write-back module 4 is used to receive the length of the data to be compressed after being converted according to a preset format from the length generation module 1 and the data compressed by the compression module 3, and cache them, and send both types of data to the host.

[0054] It should be noted that both types of data need to be sent. First, send the preset data, and then send the compressed data.

[0055] It should be noted that the data written back is first a variant little-endian format length (preset format), followed by a bunch of compressed data.

[0056] Specifically, the default format is a variant little-endian format.

[0057] Specifically, when processing data, the length generation module 1 receives the starting address of the data to be compressed, the length of the data to be compressed, and the starting address of the compressed write-back data from the host. It then forwards the starting address and length of the data to be compressed to the reading module 2 and sends the starting address of the compressed write-back data to the write-back module 4. Simultaneously, the length generation module 1 converts the length of the data to be compressed into a variant little-endian format, uses it as the "length before compression" field of the SNAPPY compressed file, sends it to the write-back module 4, and sends a start signal to the reading module 2 and the compression module 3 to initiate the reading, write-back, and compression operations.

[0058] Specifically, the write-back module 4 includes a write-back data cache module 41, a write-back generation module 42, and a write-back control module 43;

[0059] The write-back data caching module 41 is used to cache the length of the data to be compressed after being converted according to a preset format and transmitted in the length generation module 1, as well as the compressed data transmitted in the compression module 3;

[0060] The write-back generation module 42 is used to receive the starting address of the compressed write-back data transmitted by the length generation module 1, and generate continuous write access requests starting from the starting address of the compressed write-back data, thereby completing the write-back of the data in the write-back cache module 42 to the host.

[0061] The write-back control module 43 is used to interact with the host and generate control signals to control the write-back process of the write-back module 4.

[0062] Specifically, the method for converting the length of the data to be compressed into a preset format is as follows:

[0063] Arrange the data to be compressed in a preset format;

[0064] The sorted data is divided into several fields starting from the least significant digit and according to a preset number of digits.

[0065] After padding the divided fields with high-order bits, they are arranged according to a preset format to complete the conversion of the length of the data to be compressed into a preset format.

[0066] In some optional implementations of this embodiment, the process of converting data into a variant little-endian format is as follows:

[0067] (1) Represent a number in binary form, divide it into fields of 7 bits each starting from the least significant bit, and the last field (the highest field) is no more than 7 bits remaining in the most significant bit.

[0068] (2) Modify these fields to be 8-bit long. The modification method is as follows: for the highest field, fill all the high bits with 0; for other fields, fill the high bits with 1.

[0069] (3) Arrange these modified fields in little-endian format to complete the conversion of a number to its variant little-endian format.

[0070] It should be noted that the converted data in the preset format will also be sent to the write-back module (only once), and this will happen before the compression module 3 sends data to the write-back module 4. In addition to receiving the data compressed by the compression module 3, the write-back module 4 will also receive the data in the preset format, then cache it and send it to the host.

[0071] Specifically, the reading module 2 includes a reading generation module 21, a reading control module 22, and a reading cache module 23;

[0072] The reading generation module 21 is used to cache the starting address and length of the data to be compressed transmitted by the length generation module 1, and generates a series of access requests starting from the starting address of the data to be compressed with an access length equal to the length of the data to be compressed, thereby completing the reading of the data to be compressed from the host.

[0073] The read control module 22 is used to complete the interaction with the host and generate control signals to control the read process of the read module 2;

[0074] The read cache module 23 is used to cache the data read from the host and send the cached data to the compression module 3 for compression.

[0075] Specifically, such as Figure 1 and Figure 3 As shown, in this invention, when the reading module 2 processes the read data: the reading module 2 is responsible for reading and caching the data to be compressed from the host, and sending the read data to be compressed to the compression module 3 for compression. Because Snappy compression has a 64KB boundary, the reading module 2 counts the data sent to the compression module 3, sending a data block boundary signal EOB (end of block) to the compression module 3 every 64KB. When the sent data is the last byte of the data to be compressed, a data end signal EOD (end of data) is sent to the compression module 3.

[0076] The reading module 2 includes reading generation logic, a reading control module 22, and a reading cache module 23. The reading generation module 21 caches the starting address and length of the data to be compressed sent by the length generation module 1, and generates continuous access requests starting from the starting address of the data to be compressed with an access length equal to the length of the data to be compressed, thus completing the reading of the data to be compressed from the host. The reading control module 22 completes the specific interaction with the host, generating control signals to control the entire data reading process. The reading cache module 23 is responsible for caching the data read from the host and sending this data to the compression module 3 for compression.

[0077] Specifically, the read data buffer can usually be set to several read data bit widths, such as 64 bytes, the read data buffer can be set to 64 bytes, 128 bytes, 256 bytes, etc.

[0078] It should be noted that "end of data" or "end of block" indicates the end of data signal or the end of block signal, respectively. When an EOB signal is received, it means that the currently received data is the last byte of the block data; when an EOD signal is received, it means that the currently received data is the last byte of the data to be compressed.

[0079] Specifically, compression module 3 includes: string matching module 31, copy marker generation module 32, and data marker generation module 33.

[0080] The string matching module 31 is used to cache the data read by the reading module 2 and write it into a preset data history table for storage. At the same time, the string matching module 31 is used to perform hash operation on the cached data, read the corresponding hash item from the preset hash matching table according to the hash operation result for matching and comparison to find duplicate strings, and update the hash matching table as a new hash table item according to the hash operation result.

[0081] The copy marker generation module 32 is used to complete the longest string matching and copy marker generation after the string matching module 31 successfully matches the string;

[0082] The data tag generation module 33 is used to process data for which no duplicate strings are found and generate data tags.

[0083] Specifically, the data cached in the data tag generation module 33 is no more than 60 bytes.

[0084] Specifically, the hash matching table in the hardware accelerator of the SNAPPY compression algorithm of the present invention is shown in Table 1 below;

[0085] Table 1: Hash Matching Table

[0086] ABCD 30 1 BCDE 31 1 DDDD 65 1 MNOP 90 0 … … … ZZZZ 32500 1

[0087] Specifically, after the string matching module 31 successfully matches a string, the method for completing the longest string matching and generating the copy marker is as follows:

[0088] After a successful initial string match, the string matching module 31 directly passes the data obtained from the reading module 2 to the copy mark generation module 32, which caches it. Simultaneously, the copy mark generation module 32 uses the matching position segment of the corresponding hash table entry as the read address of the data history table, reads one byte of data from the data history table, and compares it with the cached data. If they are equal, the match is successful; otherwise, the match is unsuccessful. At the same time, the copy mark generation module 32 sets a longest matching length counter for matching and copy mark generation.

[0089] Specifically, in this invention, when processing data, the compression module 3 is responsible for compressing the data sent by the reading module 2, and then sending the compressed data to the write-back module 4 for data write-back. The compression module 3 includes a string matching module 31, a copy marker generation module 32, and a data marker generation module 33.

[0090] (1) String matching module 31

[0091] The string matching module 31 is responsible for caching data from the reading module 2 and writing it to a data history table. The data history table is 64KB in size and is used to sequentially store a 64KB data block; it can be stored in RAM or registers. Typically, the cached data should be 4 bytes in size. The string matching module 31 performs a hash operation on the cached data and reads the corresponding hash entry from the hash matching table based on the hash result to match and compare for duplicate strings. Simultaneously, based on the hash result, the cached data is updated as a new hash table entry in the hash matching table.

[0092] Specifically, the hash matching table contains three fields: a data segment, a match position segment, and a valid segment. The data segment stores 4 bytes of data, the match position segment stores the position of these 4 bytes of data within the 64KB data block, and the valid segment stores whether the hash table entry is valid, which can be indicated by one bit. Specifically, the string matching module 31 performs a hash operation on the cached 4 bytes of data. The hash operation can use the hash function recommended by the Snappy software compression program (i.e., multiplying the cached number by 0x1e35a7bd and then right-shifting it by a certain number of bits), or other hash functions can be used. If the final processed data is less than 4 bytes (e.g., when only 3 bytes or less remain in the current data block), subsequent string matching is not performed, or the match is forced to fail.

[0093] Specifically, after the string matching module 31 reads the corresponding entry from the hash matching table, if the entry is valid, it performs a preliminary string matching operation, which compares the data segment with the cached data to determine the cached data and checks if they are equal. If they are equal, the string matching is successful; otherwise, the matching fails. If the entry is invalid, the matching fails. Upon successful matching, the string matching module 31 sends a successful matching signal to the copy tag generation module 32 and the data tag generation module 33, activating the copy tag generation module 32 and sending the current cached data's position information and the corresponding hash table entry's matching position segment to the copy tag generation module 32 for subsequent copy tag generation. If the matching fails, the first byte of cached data is sent to the data tag generation module for subsequent data tag generation. If the matching fails again, the cached data in the string matching module 31 is updated by one byte, and the matching operation is repeated.

[0094] Specifically, when the string matching module 31 receives the EOB and EOD signals from the reading module 2, it indicates that the current data is the last data in the data block. After the data is processed, the hash matching table is reset, which can be done by setting the valid segment in the hash matching table to invalid. At the same time, the EOB and EOD signals are transmitted to the copy marker generation module 32 and the data marker generation module 33 for control of their respective modules.

[0095] (2) Copy mark generation module 32

[0096] The copy tag generation module 32 is responsible for completing the longest string matching and copy tag generation after the initial string matching of the string matching module 31 is successful.

[0097] Specifically, after a successful initial string match, the string matching module 31 sends the current position information of the current cached data and the matching position segment of the corresponding hash table entry to the copy marker generation module 32. Simultaneously, it directly transmits the data subsequently obtained from the reading module 2 to the copy marker generation module 32. The copy marker generation module 32 then begins the longest string matching operation, and the matching process is as follows:

[0098] Specifically, after a successful initial string match, the string matching module 31 directly passes the data obtained from the reading module 2 to the copy marker generation module 32, which then caches it. Simultaneously, the copy marker generation module 32 uses the matching position segment of the corresponding hash table entry as the read address of the data history table, reads one byte of data from the data history table, and compares it with the cached data. If they are equal, the match is successful; otherwise, the match fails. The copy marker generation module 32 sets a maximum match length counter, which resets to 4 each time. It automatically increments by 1 after a successful match and resets to 4 after a failed match.

[0099] Specifically, after a successful match, the copy marker generation module 32 increments the read address of the data history table by 1, and then continues to read data from the data history table. It compares the data with the next data directly passed from the read module 2 to the copy marker generation module 32 to complete the matching operation, until the match fails.

[0100] Specifically, when a match fails, the copy marker generation module 32 generates a copy marker and sends it to the write-back module 4 for write-back. The copy marker generation process is as follows: the copy marker generation module 32 uses the difference between the current position information of the current cached data and the matching position segment of the corresponding hash table entry as the offset, and uses the data before the counter is reset as the copy length to generate a copy marker. If the offset is less than 2048 and the copy length is less than 12, a format one copy marker with a length of 2 bytes is generated; otherwise, a format two copy marker with a length of 3 bytes is generated.

[0101] Specifically, when the string matching module 31 receives the EOB or EOD signal, it indicates that the last data in this data block has been processed. At this time, the copy marker generation module 32 generates a copy marker according to the current processing status. That is, the copy marker generation module 32 uses the difference between the current position information of the current cached data and the matching position segment of the corresponding hash table entry as the offset, and uses the data before the counter is reset as the copy length to generate a copy marker and send it to the write-back module 4 for write-back.

[0102] (3) Data tag generation module 33

[0103] Specifically, the data tag generation module 33 is responsible for processing the data for which no duplicate strings are found and generating data tags.

[0104] Specifically, when the initial string match fails, the string matching module 31 sends the first byte of cached data and a match failure signal to the data generation and tagging module. The data tagging module 33 sets up a data tagging data buffer. After receiving the match failure signal, the data generation module puts one byte of data into the data tagging data buffer. The buffer size should be at least 60 bytes, which is sufficient to meet the data size of a data tag format one. At the same time, the data tagging module 33 sets a data counter to count the data size in the data buffer. The initial value of the counter is 0, and it is incremented by 1 each time a buffered data is received. When the data counter is greater than or equal to 60, the data tagging module 33 retrieves the first 60 bytes from the data tagging data buffer, generates a data tag with a data length of 60 bytes, and sends it to the write-back module 4 for write-back. At the same time, the value of the data counter is decremented by 60.

[0105] Specifically, when the initial string match is successful, the string matching module 31 sends a matching success signal to the data generation module. After receiving the matching success signal, the data generation module retrieves all the data in the current data cache, generates a data tag with a data length equal to the data counter value, and sends it to the write-back module 4 for write-back. At the same time, the data counter value is reset to 0.

[0106] Specifically, when the string matching module 31 receives the EOB or EOD signal, it indicates that the last data in this data block has been processed. At this time, the data tag generation module 33 retrieves all the data in the current data buffer, generates a data tag with a data length equal to the data counter value, and sends it to the write-back module 4 for write-back. At the same time, the data counter value is reset to 0.

[0107] Specifically, at most one of the data tag generation module 33 and the copy tag generation module 32 can send data to the write-back module 4 at any given time.

[0108] It should be noted that (arbitration explanation): such as Figure 1 and Figure 2 As shown, the SNAPPY compression algorithm hardware accelerator in this invention processes data serially. At most, only one module (data marker generation module 33) or copy marker generation module 32 sends data to the write-back module 4 at any given time. The length generation module 1 only writes data once after system startup, and this process occurs before the data marker generation modules 33 and 32. Therefore, there are no conflicts between the length generation module 1, data marker generation module 33, and copy marker generation module 32. If data blocking during write-back causes the data marker generation modules 33 and 32 to simultaneously send data to the write-back module 4, the original data position information can be added when these two modules transmit data to the write-back module 4. Data with the earlier position information is processed first and written back to the host by the write-back module 4. After the copy marker generation module 32 and data marker generation module 33 receive the EOB or EOD signal and complete the current marker data transfer to the write-back module 4, the compression module 3 sends the EOB or EOD signal to the write-back module 4.

[0109] Specifically, such as Figure 1 and Figure 4As shown, in this invention, the write-back module 4 processes compressed data as follows: The write-back module 4 receives compressed data from the compression module 3 and caches it, then sends the compressed data back to the host. When the write-back module 4 receives an EOD signal from the compression module 3, it marks the last byte of the currently received data as the last byte. Once the last byte is written back to the host, the file compression is complete. The write-back module 4 includes a write-back data caching module 41, a write-back generation module 42, and a write-back control module 43.

[0110] Specifically, the write-back data buffer module 41 is used to buffer the "length before compression" field from the length generation module 1. This data is sent only once and is completed before the data tag generation module 33 and copy tag generation module 32 start. After that, it begins buffering the compressed data sent from the compression module 3. When the buffered data meets the write-back size or is the last byte of the current compressed block, the data is written back to the host. The write data buffer can typically be set to several write data bit widths, such as 64 bytes for a read data bit width, which can be set to 64 bytes, 128 bytes, 256 bytes, etc.

[0111] The write-back generation module 42 is used to receive the starting address of the compressed write-back data transmitted by the length generation module 1, and generate a continuous write access request starting from the starting address of the compressed write-back data, thereby completing the write-back of the data in the write-back cache module to the host.

[0112] Specifically, the write-back control module 43 is used to complete the specific interaction with the host, generate control signals, and control the entire data write-back process. When the write-back module 4 receives the EOB signal from the compression module 3, it indicates that the currently received compressed data is the last data in the current compressed block. At this time, a write-back request is immediately generated to complete the write-back of the current data block. When the write-back module 4 receives the EOD signal from the compression module 3, it indicates that the currently received compressed data is the last data in the compressed file. At this time, a write-back request is immediately generated, and the compression process ends after the data is written back.

[0113] Specifically, the steps and principles of the SNAPPY compression algorithm hardware accelerator of the present invention when compressing data are as follows:

[0114] The length generation module 1 receives the starting address of the data to be compressed, the length of the data to be compressed, and the starting address of the compressed write-back data from the host. It then sends the starting address and length of the data to be compressed to the reading module 2, and the starting address of the compressed write-back data to the write-back module 4. Next, the length generation module 1 converts the length to be compressed into a variant little-endian format, which becomes the uncompressed length data in the SNAPPY compressed file, and sends it to the write-back module 4, which writes this data back to the host. Afterward, the length generation module 1 sends a start signal to the reading module 2, the compression module 3, and the write-back module 4 to initiate subsequent data reading, compression, and write-back operations.

[0115] Reading module 2 reads data sequentially from the starting address of the data to be compressed and sends it to compression module 3 for data compression. After receiving the data from reading module 2, compression module 3 starts the compression operation and sends the compressed data to write-back module 4, which then writes the compressed data back to the host.

[0116] After reading the last data, the reading module 2 sends an EOD signal to the compression module 3. Upon receiving the EOD signal, the compression module 3 records the current data as the last data to be processed. After processing the last data, the compression module 3 sends an EOD signal to the write-back module 4, marking the current data as the last data. The compression process ends when the last data is written back to the host.

[0117] Therefore, the SNAPPY compression algorithm hardware accelerator of this invention implements the SNAPPY compression algorithm through hardware circuitry when processing compressed data. It completes the SNAPPY compression operation with hardware pipelined parallel processing capabilities. Data tag processing and copy tag processing are performed by different hardware components without interference. Data tags longer than 60 characters are split into multiple data tags of 60 characters or less, reducing hardware complexity. The pipelined implementation allows data reading, compression, and write-back to occur simultaneously. This means that data compression is performed through the hardware accelerator of this invention, fundamentally solving the problem of high CPU resource consumption in software-implemented SNAPPY compression. The hardware implementation of the compression algorithm does not run on the CPU, eliminating CPU usage issues and further increasing the parallelism of data processing. This results in fast compression speed, high efficiency, and low CPU resource consumption.

[0118] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Additionally, the terms "front," "back," "left," "right," "upper," and "lower" in this document refer to the placement shown in the accompanying drawings.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A hardware accelerator for the SNAPPY compression algorithm, characterized in that, include: The length generation module is used to receive and obtain the starting address of the data to be compressed, the length of the data to be compressed, and the starting address of the compressed write-back data. It sends the starting address of the data to be compressed and the length of the data to be compressed to the reading module, sends the starting address of the compressed write-back data to the write-back module, converts the length of the data to be compressed into a preset format, and sends the data in the preset format to the write-back module. The reading module is used to read the data to be compressed from the host, receive the starting address and length of the data to be compressed from the length generation module, and transmit the read data to the compression module. A compression module is used to compress the data to be compressed transmitted by the reading module and transmit the compressed data to the write-back module. The write-back module receives the length of the data to be compressed (converted to a preset format) and the data compressed by the compression module from the length generation module, caches them, and sends the compressed data to the host. The compression module includes: a string matching module, a copy marker generation module, and a data marker generation module. The string matching module is used to cache the data of the reading module and write it into a preset data history table for storage. At the same time, the string matching module is used to perform hash operation on the cached data, read the corresponding hash item from the preset hash matching table according to the hash operation result for matching and comparison to find duplicate strings, and update the hash matching table as a new hash table item according to the hash operation result. The copy marker generation module is used to complete the longest string matching and copy marker generation after the string matching module successfully matches the string; The data tag generation module is used to process data for which no duplicate strings are found and generate data tags. After a string matching module successfully matches a string, the method for completing the longest string matching and copy marker generation is as follows: After a successful initial string match, the string matching module directly passes the data obtained from the reading module to the copy marker generation module, which caches it. Simultaneously, the copy marker generation module uses the matching position segment in the corresponding hash table entry as the read address of the data history table, reads one byte of data from the data history table, and compares it with the cached data. If they are equal, the match is successful; otherwise, the match fails. The copy marker generation module also sets a longest match length counter for matching and copy marker generation.

2. The SNAPPY compression algorithm hardware accelerator according to claim 1, characterized in that, The preset format is a variant little-endian format.

3. A hardware accelerator for the SNAPPY compression algorithm according to claim 1, characterized in that, The reading module includes a reading generation module, a reading control module, and a reading cache module. The read generation module is used to cache the starting address and length of the data to be compressed transmitted by the length generation module, and generate a series of access requests starting from the starting address of the data to be compressed with an access length equal to the length of the data to be compressed, thereby completing the reading of the data to be compressed from the host. The read control module is used to complete the interaction with the host and generate control signals to control the read process of the read module; The read cache module is used to cache data read from the host and send the cached data to the compression module for compression.

4. A hardware accelerator for the SNAPPY compression algorithm according to claim 1, characterized in that, At most one of the data tag generation module and the copy tag generation module can send data to the write-back module at any given time.

5. A hardware accelerator for the SNAPPY compression algorithm according to claim 1, characterized in that, The data cached in the data tag generation module is no more than 60 bytes.

6. A hardware accelerator for the SNAPPY compression algorithm according to claim 1, characterized in that, The write-back module includes a write-back data caching module, a write-back generation module, and a write-back control module. The write-back data caching module is used to cache the length of the data to be compressed after being converted according to a preset format and transmitted in the length generation module, as well as the compressed data transmitted in the compression module. The write-back generation module is used to receive the starting address of the compressed write-back data transmitted by the length generation module, and generate a continuous write access request starting from the starting address of the compressed write-back data, thereby completing the write-back of the data in the write-back cache module to the host. The write-back control module is used to interact with the host and generate control signals to control the write-back process of the write-back module.

7. A hardware accelerator for the SNAPPY compression algorithm according to claim 1, characterized in that, The method to convert the length of the data to be compressed into a preset format is as follows: Arrange the data to be compressed in a preset format; The sorted data is divided into several fields starting from the least significant digit and according to a preset number of digits. After padding the divided fields with high-order bits, they are arranged according to a preset format to complete the conversion of the length of the data to be compressed into a preset format.

Images