Hardware acceleration device and method for high-throughput lossless data compression

Abstract

The invention discloses a hardware acceleration device and method for high-throughput lossless data compression, and relates to the field of integrated circuit design. The device comprises a front-end LZ77 compression module, a middle buffer module and a rear-end entropy coding module, wherein the front-end LZ77 compression module is configured to search and match input data; the intermediate buffer module is configured to realize decoupling of front and rear end processing time domains through an asynchronous symbol buffer; the back-end entropy coding module is configured to receive an output of the intermediate buffer module, construct a microcode packet intermediate representation containing codeword information, and parse the microcode packet to generate a serial compressed bit stream. According to the method, invalid memory reading and assembly line pause can be effectively eliminated, the throughput and the energy efficiency ratio of the hardware compressor are remarkably improved on the premise of not sacrificing the compression rate, and the method is suitable for unloading acceleration of a high-performance solid state disk controller, an intelligent network card and a cloud server.

Description

Technical Field

[0001] This invention relates to the fields of integrated circuit design and high-performance data processing technology, specifically to a lossless data compression hardware acceleration device for use in data centers, edge computing nodes, and high-bandwidth network interface controllers. Background Technology

[0002] With the explosive growth of cloud computing, artificial intelligence (AI), and large-scale data centers, the contradiction between data transmission bandwidth and storage costs has become increasingly prominent. To improve storage density and alleviate network bandwidth pressure, lossless data compression technology has become a standard feature in storage controllers, smart network interface cards (NICs), and edge computing nodes. Among these, the Deflate algorithm, widely used in GZIP, PNG, and HTTP compression standards, has become the mainstream choice in the industry.

[0003] However, with the evolution of network interface speeds, traditional pure software compression schemes based on general-purpose processors (CPUs) can no longer meet the real-time requirements of line-speed processing. Even with high-performance server CPUs, the single-core compression throughput is usually limited by instruction set efficiency, reaching only a few hundred MB / s, which is significantly lower than the bandwidth requirements of modern high-speed networks. Therefore, designing an efficient hardware acceleration scheme for lossless compression algorithms has become a critical technical problem that urgently needs to be solved.

[0004] While most existing hardware compression accelerators have improved throughput to some extent, they still face three major technical bottlenecks when dealing with high-throughput scenarios, which restrict further performance improvements and energy efficiency optimization: 1. The core of the LZ77 algorithm lies in performing the longest string match within a historical sliding window (e.g., 32KB). To improve throughput, current mainstream hardware architectures generally employ a multi-byte parallel window (PWS, e.g., processing 32 bytes per cycle) combined with a multi-bank hash table for searching. However, existing traditional hardware architectures typically use a "read-then-compare" mechanism: after locating all possible candidate positions using the hash index, the historical data for these positions must be read entirely from on-chip memory (e.g., BRAM) and then compared with the current data byte-by-byte. In high-parallelism designs, this means that multiple memory read operations may be initiated each clock cycle. Moreover, due to hash collisions, a large number of memory read operations will ultimately be judged as invalid matches. These invalid reads not only consume storage bandwidth but also lead to dynamic power consumption waste.

[0005] 2. The Deflate standard supports dynamic Huffman coding, a mechanism that requires scanning the entire data block first to statistically analyze symbol frequencies, constructing an optimal Huffman tree, and then using this tree to encode the same data block. This "statistics first, then tree building, then encoding" data dependency limits the data processing speed. In traditional pipeline designs, when the front-end LZ77 module outputs data, if the back-end is performing tree building operations, the front-end must stop working to wait for the back-end to be ready, or the back-end cannot encode until the front-end's statistics are complete. This pipeline pause caused by data dependency reduces the utilization of hardware resources, making it difficult for the system to maintain full throughput in dynamic encoding mode.

[0006] 3. Hardware compressors typically output highly parallel fixed-length symbols at the front end (e.g., 32 symbols per cycle), while the back-end Huffman encoder outputs a variable-length bitstream. Seamlessly concatenating multiple variable-length codewords with varying bit widths generated in each cycle into a compact bitstream within a single cycle requires extremely deep combinational logic levels of barrel shifters and bit-width alignment logic. As parallelism increases, the complexity of this logic rises, often becoming the critical path in the overall system design. This limits the maximum operating frequency achievable by the hardware system, making it difficult for the processing speed of the back-end encoding module to match the high throughput of the front-end LZ77 module, thus becoming a bottleneck restricting overall system performance. Summary of the Invention

[0007] This invention aims to solve the aforementioned technical problems by providing a hardware acceleration device and method for high-throughput lossless data compression. Through architectural restructuring and methodological design, this invention significantly improves system throughput, energy efficiency, and operating frequency while ensuring the required compression ratio.

[0008] The present invention specifically adopts the following technical solution: A hardware acceleration device for high-throughput lossless data compression, characterized in that it comprises: The front-end LZ77 compression module includes a near-field matching engine, a far-field matching engine, and a speculative arbitrator. The near-field matching engine is configured to search and match input data within the current and previous period windows. The far-field matching engine is configured to search and match input data within a historical sliding window. The far-field matching engine includes a hash calculation unit, a metadata active value table, and a multi-bank hash table data storage area. The speculative arbitrator is configured to selectively initiate read operations on the multi-bank hash table data storage area based on the near-field matching length output by the near-field matching engine and the far-field matching length output by the metadata active value table. The intermediate buffer module, connected between the front-end LZ77 compression module and the back-end entropy encoding module, is configured to decouple the front-end and back-end processing time domains through an asynchronous symbol buffer. The backend entropy encoding module is configured to receive the output of the intermediate buffer module, construct a micro-packet intermediate representation encapsulated with codewords and code lengths, and parse the micro-packet to generate a serial compressed bit stream.

[0009] The present invention also provides a method for implementing the Deflate algorithm based on the above-mentioned device, characterized in that it includes: Front-end matching steps: Combining near-field search with far-field inference logic based on metadata active value table, calculate the hash value of input data to index the metadata active value table, synchronously record the matching length as metadata when writing data back, and prioritize verifying this metadata during search to arbitrate whether to read the multi-bank hash table data storage area. Only initiate memory read when the metadata indicates a valid match; otherwise, reuse the near-field result. Intermediate transmission steps: The data stream is concurrently distributed to the statistics path and the storage path using a dual-path architecture. The current data block is written to the first statistics library and the asynchronous symbol buffer. When the data block ends, the process immediately switches to the second statistics library. At the same time, a Huffman tree is constructed using the data in the first statistics library, and data is read back from the asynchronous symbol buffer for encoding. The front-end data is continuously cached during the construction of the encoding tree in the back-end. Backend encoding steps: In a single cycle, multiple symbols are mapped in parallel into micro-code packets encapsulated with codewords and code lengths. A finite state machine is used to parse the micro-code packets slot by slot, and variable-length codewords are concatenated into an output bit stream through shift operations.

[0010] The present invention also provides an electronic device, characterized in that it includes: a memory configured to store computer programs, instructions or configuration bit streams; a processor coupled to the memory; and a communication interface configured to enable communication between the electronic device and other devices; wherein the processor is configured to run the computer programs, instructions or load the configuration bit streams to implement the steps of the above method.

[0011] Beneficial effects: This invention achieves on-demand reading of historical data by introducing a metadata inference arbitration mechanism, reducing invalid storage access and bandwidth consumption; it effectively eliminates pipeline pauses in dynamic encoding through an asynchronous symbol buffer transmission process, achieving decoupling of compression and encoding processing; and it separates parallel mapping from serial packing logic using micro-packet intermediate representation, shortening the critical path. Without sacrificing compression ratio, this invention significantly optimizes the processing speed for lossless compression of large volumes of data and the energy efficiency of the processing platform. It is highly versatile and applicable to computing scenarios such as high-performance solid-state drive controllers, smart network cards, and cloud server offloading acceleration. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the overall architecture of a hardware acceleration device for high-throughput lossless data compression according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the far-field matching method based on the metadata activity value table in an embodiment of the present invention; Figure 3 This is a diagram of the microcode packet data structure defined in an embodiment of the present invention; Figure 4 This is a schematic diagram of the electronic device structure provided in an embodiment of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below with reference to the accompanying drawings. These embodiments are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention.

[0014] Example 1 refer to Figure 1 This embodiment demonstrates a hardware acceleration device for high-throughput lossless data compression integrated on an FPGA (such as Xilinx UltraScale+) or ASIC chip. It mainly consists of three clock domains or logical partitions, including: a front-end LZ77 compression module, an intermediate buffer module, and a back-end entropy encoding module.

[0015] 1. Front-end LZ77 compression module: hybrid matching and metadata generation The core components are the near-field matching engine and the far-field matching engine, responsible for receiving the raw data stream. This module is configured with a parallel window size (PWS) of 32 bytes, meaning it processes 32 bytes of data per clock cycle. The core task of the front end is to find the longest matching string for each input byte. Unlike traditional designs that attempt to solve all problems with a single structure, this system physically decomposes the matching task. Near-field short matching is processed by a channel-based parallel comparison array, while far-field long matching is processed by a hash calculation unit in conjunction with a metadata active value table.

[0016] 2. Intermediate Buffer Module: Global Traffic Distribution and Isolation The core components are a global arbiter and an asynchronous symbol buffer. The global arbiter is responsible for simultaneously distributing the symbol stream output from the front end to two processing paths. The first processing path serves as a statistics path leading to the frequency accumulator. This is a high-speed, double-buffered memory used to calculate the symbol frequency of the current data block in real time. The second processing path serves as a storage path leading to the asynchronous symbol buffer.

[0017] 3. Backend Entropy Encoding Module: Microcode Decoupling and Streaming Output The core components are a microcode generator and a bitstream packer. They are responsible for reading data from the buffer and encoding it according to the constructed Huffman tree.

[0018] Specifically, such as Figure 1 As shown, the front-end LZ77 compression module includes a near-field matching engine, a far-field matching engine, and a speculative arbitrator. To address the bandwidth bottleneck and resource mismatch issues of traditional LZ77 algorithm implementations in high-throughput scenarios, this architecture physically decouples the search task into two physically independent sub-engines: a near-field matching engine for short-distance matching and a far-field matching engine for long-distance historical matching. These sub-engines work together in conjunction with a lossless arbitration mechanism based on metadata.

[0019] Specifically, the near-field matching engine is configured to capture short-range, high-frequency repetitive data within the parallel window size (PWS) and its immediate neighboring historical windows. Unlike traditional designs that use block random access memory (BRAM) as the hash table cache medium, the near-field matching engine in this embodiment is built entirely on register logic to achieve zero-latency data access.

[0020] Specifically, the near-field matching engine includes a shift register group with a depth of 2×PWS_WIDTH. This shift register group is logically divided into two parts: the first part (Reg_Window_Curr) stores the input data for the current clock cycle; the second part (Reg_Window_Prev) stores the historical data from the previous clock cycle. This double-window-width register design ensures that the compression algorithm can continuously detect boundary matches across clock cycles, effectively solving the "boundary effect" problem commonly found in traditional block compression architectures, and allowing repetitive patterns in the data stream at the split points (such as run-length encoded RLE or short-distance overlap) to still be fully identified.

[0021] Specifically, the near-field matching engine employs a channel-based parallel comparison array to detect all possible matching distances (offsets) in parallel. Taking a PWS width of 32 bytes as an example, the parallel comparison array includes multiple comparator units CMP[i][offset], where i represents the index of the current byte and offset represents the distance to look up. The system pre-defines a set of frequently occurring matching distances. For any fixed offset value δ, a dedicated channel δ is constructed for the short offsets that frequently occur in LZ77. A set of fixed offsets Δ = {1, 2, 3, 4, 5, 6, 7, 8, 16, 32} is selected. For each δ∈Δ, the circuit instantiates a set of parallel comparators CMP. i,δ=(Reg_Curr[i]==Reg_Curr[i-δ])?1:0 (Note: When i-δ<0, the index points to the corresponding position of Reg_Prev). Each δ channel directly outputs a set of matching feature vectors, which contain valid bits, match length, and offset. All channels work in parallel without interfering with each other. For any specific channel (let the distance of interest of the current channel be K), it consists of a set of parallel comparator units. For each byte index i (from 0 to PWS-1) in the current window, the channel directly compares the current byte Data[i] with the byte Data[ik] at the historical position. When index i<k, the comparison logic is automatically mapped to the corresponding high bit of the data (Reg_Prev) latched in the previous cycle, thereby ensuring the continuity across window boundaries. This channel contains 32 (i.e., PWS) comparators, which output the comparison result, i.e., the near-field match length, simultaneously in a single cycle.

[0022] Based on the parallel comparison array described above, the identification of specific data patterns becomes direct and requires no state machine intervention: For continuous repetition detection: the output is directly from the channel with Offset=1. If the matching mask vector of this channel contains consecutive "1"s, it means that continuous repetitions of the pattern "AAAA..." have been detected.

[0023] For short-range overlap detection: the channels with offset > 1 are responsible. For example, if the channel with offset = 3 outputs a continuous match, it means that a short-range loop of the pattern "ABCABC..." has been detected.

[0024] Specifically, the far-field matching engine is configured to search for matches within a large historical window (e.g., 32KB to 128KB) and includes: hash calculation units, metadata active value tables, and multi-bank hash table data storage areas.

[0025] The hash calculation unit is used to generate hash index signals. Its input is connected to the raw data stream of the parallel processing window (PWS), and its output is connected to the address input of the metadata active value table and the multi-bank hash table data storage area, respectively.

[0026] The metadata activity value table, acting as a multi-port simulator, has its input connected to the output of the hash calculation unit. Unlike traditional table structures that only store pointers, in this embodiment, the table entry structure of the metadata activity value table is expanded to include three key fields: Bank ID, True Match Length, and Validation Tag. The Bank ID points to the BRAM location where historical data is actually stored, and the True Match Length stores the byte length value calculated during the last data write to that hash location, i.e., the far-field match length.

[0027] The multi-bank hash table data storage area is divided into M independent storage banks to support parallel writes without blocking. When writing data, the system selects the target storage bank based on the low-order bits of the hash value or other mapping rules. The data input end is connected to the original data stream, and the read port is controlled by a speculative arbitrator.

[0028] Specifically, the speculative arbitrator is connected to the metadata activity value table and multi-bank hash table data storage area in the near-field matching engine and the far-field matching engine. It is used to receive data output by the near-field matching engine and the far-field matching engine and control the read operation of BRAM.

[0029] The working principle of the front-end LZ77 compression module is as follows: First, by utilizing the write-back feature of the LZ77 algorithm, the actual matching length is pre-calculated during the data writing phase and stored as metadata in the metadata activity value table; during the search phase, the metadata activity value table is queried to obtain metadata, and the metadata is used to make predictions and arbitrations without reading the multi-bank hash table data storage area; only after confirming that the match is valid will a delayed retrieval of the multi-bank hash table data storage area be initiated.

[0030] To address the pipeline stall problem in dynamic Huffman coding, this embodiment employs the following intermediate buffer module: Figure 1 The hardware architecture shown includes: a streaming symbol generation unit, a global arbiter, a first processing path (statistics path), and a second processing path (storage path).

[0031] like Figure 1 The streaming symbol generation unit shown takes the output of the LZ77 compression module as input, performs pruning and encoding operations on the compressed output sequence, and is finally configured to continuously and parallelly output a compressed symbol sequence containing literal and distance length pairs according to the clock cycle.

[0032] like Figure 1 The global arbitrator shown is connected to the output of the streaming symbol generation unit. It serves as the control hub for the data stream, used to clone the input compressed symbol sequence in real time and distribute it to the first processing path and the second processing path.

[0033] like Figure 1 As shown, the first processing path serves as a statistical path, containing a real-time frequency accumulator. This module internally integrates two independent statistical storage banks (the first statistical bank Bank A and the second statistical bank Bank B). This is used to synchronously capture symbol sequences during data stream transmission and to calculate the frequency of each character within the current data block in real time, generating a symbol frequency histogram. The second processing path serves as a storage path, containing an asynchronous symbol buffer, used for deep caching of the original symbol sequence.

[0034] In this embodiment, the backend entropy coding module is as follows: Figure 1 As shown, it includes a microcode generator and a bitstream packer. The microcode generator is coupled to the first processing path and the second processing path, respectively. It is used to read the statistical results of the first processing path to construct a Huffman tree and send a read request to the second processing path to obtain historical symbol data. Combined with the newly built Huffman tree, the final bitstream encoding output is performed through the microcode packer and the bitstream packer.

[0035] This embodiment describes a hardware acceleration device for high-throughput lossless data compression, which can be integrated into an FPGA (Field Programmable Gate Array), ASIC (Application-Specific Integrated Circuit), or SoC (System-on-a-Chip) and applied to solid-state drive (SSD) controllers, smart network interface cards (SmartNICs), or data center server acceleration cards.

[0036] The hardware acceleration device for high-throughput lossless data compression in this embodiment provides two different matching engines: near-field and far-field. For different types of matching (short-distance repetition and long-distance reference) in the LZ77 algorithm, they are separated in the physical circuit: a channel-based parallel comparison array is used to process near-field matching; a hash calculation unit is used to process far-field matching. Single-cycle multi-hash value throughput is achieved through expanding cyclic polynomial calculations, and the location information is stored in a hash table using the hash value as an index.

[0037] This embodiment of the hardware acceleration device for high-throughput lossless data compression introduces a lossless speculative arbitration mechanism based on a metadata active value table. This invention innovatively extends the definition of the metadata active value table in the hash table, storing the bank index (Bank_ID), the true match length (True_Match_Length), and the validity tag (Validation_Tag). During the arbitration phase, the system directly uses this metadata for speculative decision-making, initiating a memory read only when a match in the hash table is confirmed to be superior to a match in the comparison array.

[0038] The hardware acceleration device for high-throughput lossless data compression in this embodiment employs an asynchronous symbol buffering transmission strategy. By introducing a dual-path intermediate buffer module between the LZ77 front-end and the entropy coding back-end, ping-pong buffer control logic completely separates the data front-end and back-end processes. When the back-end constructs the Huffman tree, the data generated by the front-end is sent to a large-capacity memory instead of blocking the pipeline.

[0039] This embodiment of the hardware acceleration device for high-throughput lossless data compression proposes a micro-packet decoupled coding technique. Addressing the timing bottleneck of parallel-to-serial conversion, this invention defines an intermediate representation format. The system first generates micro-packets containing codewords and code lengths through parallel table lookups, and then these packets are processed serially by a finite state machine. This separates the complex table lookup logic (parallel domain) from the shifting and packing logic (serial domain).

[0040] Example 2 This embodiment provides a method for implementing the Deflate algorithm based on the above-described device, including: Front-end matching steps: Combining near-field search with far-field inference logic based on metadata active value table, calculate the hash value of input data to index the metadata active value table. When writing data back, synchronously record the actual matching length as metadata. During the search, first verify the metadata to arbitrate whether to read the multi-bank hash table data storage area. Only initiate memory read when the metadata indicates a valid match; otherwise, reuse the near-field result. Intermediate transmission steps: The data stream is concurrently distributed to the statistics path and the storage path using a dual-path architecture. The current data block is written to the first statistics library and the asynchronous symbol buffer. When the data block ends, the process immediately switches to the second statistics library. At the same time, a Huffman tree is constructed using the data in the first statistics library, and data is read back from the asynchronous symbol buffer for encoding. The front-end data is continuously cached during the construction of the encoding tree in the back-end. Backend encoding steps: In a single cycle, multiple symbols are mapped in parallel into micro-code packets encapsulated with codewords and code lengths. A finite state machine is used to parse the micro-code packets slot by slot, and variable-length codewords are concatenated into an output bit stream through shift operations.

[0041] Specifically, the front-end matching step employs a far-field matching method based on a metadata activity value table, such as... Figure 2 As shown, this method uses the aforementioned far-field matching engine to process the data stream in the following steps: Step 1: When the hash calculation unit receives the Parallel Processing Window (PWS) signal of the input data stream, it calculates the hash value in parallel for each byte position in the window. This hash value will be used simultaneously as the index address for accessing the metadata active value table and the multi-bank hash table data storage area.

[0042] This design employs a cyclic polynomial hash algorithm improved for hardware architecture. The mathematical model for hash calculation is as follows: Let the window size be k, and the hash value H at the current position i be... i It can be derived from the previous position H i-1 We can quickly derive: H i =((H i-1 -Byte[i-1]×C k-1 The hash function is calculated as Σ(i+k-1) × C + Byte[i+k-1], where C is a constant base (usually a prime number). To obtain W hash values ​​in parallel within a single clock cycle in hardware, the hash calculation unit expands the recursive logic using prefix computation and maps it to a parallel DSP slice (such as the Xilinx DSP48E2). This unit pre-computes all polynomial coefficients using DSP resources and then outputs W parallel hash values ​​within one clock cycle using an adder tree.

[0043] Step 2: During the data write-back cycle, analyze the currently processed data. While the compressor writes the current data to the history window, the system concurrently executes operations on the data path and control path: Data path operations: Write the original data according to the hash mapping rules. Figure 1 The selected multi-bank hash table; Control path operation: Calculate the matching length L of the current data in the sliding window, and synchronously write the tuple containing {Storage index=B, Actual matching length=L} to the corresponding hash index position of the metadata active value table.

[0044] This step extracts key features (metadata) at the moment the write operation occurs, shifting the computational complexity from the latency-sensitive read path to the write path, so that the subsequent search process can directly utilize the cached decision information.

[0045] Step 3: During the search cycle, the far-field matching engine queries the metadata active value table based on the hash value of the current input data. The metadata active value table directly returns the actual match length metadata and storage index stored at that location. During this process, the system keeps the read ports of the multi-bank hash table silent and does not access the data storage.

[0046] Step 4: The speculative arbitrator receives the true match length (corresponding to the far-field match length) value returned by the metadata active value table and evaluates it. For example, it determines whether the length meets the minimum match threshold, or compares the length values ​​of the candidates in the case of multiple hash collision candidates. Since the length value comes directly from the metadata active value table register or cache, this determination process involves only very low-latency numerical comparison logic.

[0047] Step 5: Perform delayed retrieval based on the evaluation results of Step 4: If the evaluation results confirm a valid and sufficiently long match, the speculative arbiter generates a read request pointing to a specific data storage bank based on the storage bank index to retrieve the original historical data for final output; if the evaluation results show an invalid match or insufficient length, no read operation is generated for the data storage bank. If an invalid match is detected, it means that although there is a record at the current hash position, the match quality is low, and the system directly ignores the path.

[0048] In this embodiment, the intermediate transmission step employs a zero-pause ping-pong buffer control method, processing the data stream according to the following steps: Step 1, Concurrent Distribution and Data Cloning: When the streaming symbol generation unit outputs a symbol stream belonging to the current data block (Block N), the global arbiter initiates concurrent distribution mode. At this time, the system is in cycle T1 operating mode: the global arbiter controls the first processing path to use Bank A as the current working bank. Bank A performs high-speed read, write, and accumulation operations on the input symbols of Block N, updating the frequency table in real time; simultaneously, Bank B is in a hold or reset state, preparing for the next stage. Meanwhile, a copy of the data stream is synchronously written to the asynchronous symbol buffer of the second processing path for queuing.

[0049] Step 2, Wait-Free Switching and Continuous Throughput: After the last symbol of the current data block (Block N) is output by the streaming symbol generation unit, the streaming symbol generation unit does not perform any waiting or pause operations and immediately and seamlessly switches to the processing state of the next data block (Block N+1) in the next clock cycle. The input stream of the statistical path is seamlessly directed to Bank B to begin statistical processing of the data in Block N+1. Bank A is immediately locked and a read-only port is opened to the backend microcode generator. At this time, although the encoding of Block N has not yet started, the data of Block N+1 has been continuously written into the subsequent address space of the asynchronous symbol buffer by the global arbiter, forming a continuous data stream backlog.

[0050] Step 3: Asynchronous Tree Construction and Parallel Processing. While the streaming symbol generation unit begins processing Block N+1, the first processing path has already locked the complete frequency data of Block N. The microcode generator uses this locked frequency data to initiate the Huffman tree construction process. During this process, the hardware system is in a "dual-stream parallel" state: the front end is writing data to Block N+1 at high speed, while the back end is calculating the coding tree based on the statistical information of Block N. The two processes do not interfere with each other and are completely parallel.

[0051] Step 4, Delayed Readback and Catch-up Encoding: Once the Huffman tree for Block N is constructed, the microcode generator immediately sends a burst read request to the asynchronous symbol buffer. Starting from the head pointer of the asynchronous symbol buffer, the microcode generator reads the historical data of Block N at high speed. Since the Huffman tree is ready at this point, the read data is immediately mapped to Huffman code and output. The asynchronous symbol buffer is configured to support either a "write-first" or "read-write concurrent" access strategy, ensuring that while the backend is reading back Block N data, the frontend's write operations to Block N+1 or even Block N+2 are unaffected.

[0052] The asynchronous symbol buffer described in this embodiment is not a general FIFO (First-In-First-Out) queue, but a high-capacity asynchronous dual-port buffer device. The key to its capacity design is that the storage capacity C is configured to accommodate at least two complete maximum-length compressed blocks (i.e., C ≥ 2 × L). max This specific capacity ratio allows the front-end write pointer to lead the back-end read pointer by at least one data block. This strategy effectively solves the pipeline bubble problem caused by the dependency of the "statistics first, then encoding" algorithm logic, significantly improving the overall data throughput of the system. This design concept is fundamentally different from the traditional small-capacity FIFO that only serves as an interface buffer, and can effectively cope with the high-concurrency data flow pressure in high-speed network storage or in-memory computing scenarios.

[0053] In this embodiment, the back-end encoding step employs a two-level micro-packet encoding method to address the timing bottleneck in the conversion process from high-parallel data streams to serial bit streams. This method introduces standardized micro-packets as intermediate representations, logically decoupling the complex Huffman lookup table operation from the variable-length bit stream concatenation operation.

[0054] To efficiently transmit variable-length encoded information between stages of the hardware pipeline, this embodiment first defines a fixed-width microcode packet format. This microcode packet is designed as an atomic transmission unit, completing transmission within one bus clock cycle. For example... Figure 3 As shown, its data structure is divided into three independent functional fields in terms of physical bits: Valid bitmask field: Located in the packet header, it indicates the number of valid symbols in the current microcode packet and their distribution in the slots. This field is usually implemented as a Bitmap for subsequent pipeline-level fast gating; Metadata payload field: As the core of the data packet, it contains N parallel fixed slots, where N strictly corresponds to the number of parallel paths of the front-end LZ77 compression module. Each slot encapsulates a standardized binary data structure: {Huffman_Code, Code_Length}, i.e., the Huffman codeword and its length. This field masks the encoding differences between different symbols, providing a unified interface to the subsequent layers; Extension bit field: Dedicated to storing non-Huffman encoded data generated by the LZ77 algorithm, such as the uncompressed portion of a literal or the extension bit for matching length / distance. This field is associated with a slot in the metadata payload field via a predefined offset.

[0055] In this embodiment, the two-level microcode packet encoding method divides the processing flow into two cascaded hardware pipeline stages: a first-level parallel generation stage and a second-level serial streaming packetization stage, and processes the data according to the following steps: Step 1: Parallel microcode generation in the first-stage pipeline. This stage aims to match the high throughput of the front-end LZ77 module and achieve "multiple-input, one-output" parallel mapping. Within one clock cycle, the system simultaneously receives multiple raw compressed symbols from the front end via a wide-bit-width bus. Using a pre-built multi-port lookup table or parallel register array in the hardware, Huffman coding mapping is performed simultaneously on all received symbols. This step obtains the Huffman codeword, code length, and any possible extension bits for each symbol in parallel. Subsequently, the encoding logic generates a corresponding effective bitmask based on the attributes of each symbol, fills the obtained multiple sets of codeword and code length data into the corresponding slots, and finally packages them in hardware according to the bit-width format defined in the "microcode packet," outputting a complete microcode packet on the rising edge of the clock.

[0056] Step 2: The second-stage pipeline performs serial streaming packetization. This stage aims to solve the timing convergence problem of variable-length bitstream splicing by implementing fine-grained bit-level operations through state machine control. Micropackets are received from the first-stage pipeline and temporarily stored in the input buffer register. A high-frequency finite state machine (FSM) is started, and the valid slots in the micropackets are quickly scanned one by one according to the indication of the valid bitmask field.

[0057] Step 2.1: The state machine reads the "code length" information in the current slot and controls the barrel shifter to dynamically shift the "codeword". The shift amount is determined by the number of residual bits already buffered in the current accumulator. The shifted codewords are merged into the end of the bitstream accumulator through a logical "OR" operation, achieving seamless splicing of variable-length codewords.

[0058] Step 2.2: After each concatenation operation, check the number of valid bits in the accumulator. When the accumulated bit stream length exceeds the fixed word length of the system bus, immediately trigger an output operation to output the complete word to main memory or FIFO. At the same time, use truncation logic to retain the remaining bits exceeding the word length in the accumulator and update the accumulator's count pointer as the starting reference for concatenating the next micropacket or the next slot.

[0059] In summary, given that traditional hardware compression architectures are generally limited by the "memory wall" effect and pipeline timing bottlenecks in high-throughput scenarios, this invention proposes a novel hardware acceleration solution. To change the inefficient access pattern caused by the traditional "read-then-compare" approach, eliminate massive invalid memory read operations, and effectively overcome the chip's internal memory bandwidth bottleneck, this invention constructs a physically separated near-field and far-field hybrid matching architecture, coupled with a speculative arbitration mechanism based on a metadata activity value table. This significantly improves the system's core data throughput while reducing dynamic power consumption. Furthermore, to address the pipeline stall problem caused by dynamic Huffman tree construction, this invention introduces a dual-path asynchronous symbol buffer, achieving full-time decoupling between the compression front-end and the encoding back-end. Simultaneously, by combining parallel-to-serial conversion technology based on micro-packet intermediate representation, critical path timing is further optimized, ensuring the device can adapt to the line-rate processing requirements of high-bandwidth network interfaces.

[0060] Example 3 This invention also provides an electronic device, which includes, but is not limited to, a data center server, an edge computing node, a network switching device, a storage server, or a personal computer.

[0061] like Figure 4 As shown, the electronic device includes: a processor: the processor can be a general-purpose processor (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices. In this embodiment, the processor is specifically configured to execute or control the high-throughput data compression hardware acceleration logic described in the above embodiments. A memory: used to store instructions, configuration bitstreams, or data to be compressed / compressed. The memory can include high-speed random access memory or non-volatile memory (such as Flash, SSD, etc.). A communication interface: used to enable communication between the electronic device and other devices (such as a host, network). A bus: used to connect the processor, memory, and communication interface to enable information transmission between the components.

[0062] Specifically, when the processor is an FPGA or ASIC, the above data compression method is implemented inside the processor through hardware logic circuits synthesized by a hardware description language (such as Verilog / VHDL), and the line-speed data compression is completed by utilizing hardware parallelism; when the processor is a general-purpose CPU, the externally mounted hardware acceleration card or the software simulation of the above method can be controlled by calling computer program instructions stored in memory.

[0063] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

[0064] It should be noted that the above description is merely a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. For those skilled in the art, any improvements and modifications made without departing from the principle of the present invention (such as adaptation to different process nodes, adjustment of buffer depth, or expansion of parallel paths) should also be considered within the scope of protection of the present invention. All technical solutions falling within the scope of the present invention should be included within the scope of protection of the present invention.

Claims

1. A hardware acceleration device for high-throughput lossless data compression, characterized in that, include: The front-end LZ77 compression module includes a near-field matching engine, a far-field matching engine, and a speculative arbitrator; The near-field matching engine is configured to search and match input data within the current period and the previous period window; the far-field matching engine is configured to search and match input data within a historical sliding window; the far-field matching engine includes a hash calculation unit, a metadata active value table, and a multi-bank hash table data storage area; the speculative arbitrator is configured to selectively initiate read operations on the multi-bank hash table data storage area based on the near-field matching length output by the near-field matching engine and the far-field matching length output by the metadata active value table. The intermediate buffer module, connected between the front-end LZ77 compression module and the back-end entropy encoding module, is configured to decouple the front-end and back-end processing time domains through an asynchronous symbol buffer. The backend entropy encoding module is configured to receive the output of the intermediate buffer module, construct a micro-packet intermediate representation encapsulated with codewords and code lengths, and parse the micro-packet to generate a serial compressed bit stream.

2. The apparatus according to claim 1, characterized in that: The near-field matching engine adopts an architecture based on full register logic, including a shift register group with a depth twice the width of the parallel window, which is configured to store the current clock cycle data and the previous clock cycle data simultaneously to eliminate the cross-cycle boundary detection blind zone. The near-field matching engine utilizes a channel-based parallel comparison array to compute the matching results.

3. The apparatus according to claim 1, characterized in that: The hash calculation unit concurrently calculates the hash index of all byte positions within a parallel window in a single clock cycle, and accesses the metadata active value table and the multi-bank hash table data storage area accordingly; the compressed attribute metadata in the metadata active value table includes at least the storage index, the actual matching length, and the valid bits.

4. The apparatus according to claim 1, characterized in that: The asynchronous symbol buffer adopts ping-pong buffer control logic, which is configured to redirect the symbol stream output by the front-end LZ77 compression module to an idle storage unit when the back-end entropy coding module is in the Huffman tree construction state. The capacity of the idle storage unit is at least enough to accommodate two Deflate data blocks defined by the maximum transmission unit.

5. The apparatus according to claim 1, characterized in that: The data structure of the microcode packet includes an effective mask field and a multi-slot payload field; the back-end entropy coding module includes a microcode generator and a bitstream packer. The bitstream packer is configured as a finite state machine, which sequentially parses the effective mask field in the microcode packet, loads the codewords in the effective slots into a barrel shifter, and performs an accumulation shift operation according to the corresponding code length to generate a continuous compressed bitstream.

6. A method for implementing the Deflate algorithm based on the apparatus of any one of claims 1 to 5, characterized in that, include: Front-end matching steps: Combining near-field search with far-field inference logic based on metadata active value table, calculate the hash value of input data to index the metadata active value table, synchronously record the matching length as metadata when writing data back, and prioritize verifying this metadata during search to arbitrate whether to read the multi-bank hash table data storage area. Only initiate memory read when the metadata indicates a valid match; otherwise, reuse the near-field result. Intermediate transmission steps: The data stream is concurrently distributed to the statistics path and the storage path using a dual-path architecture. The current data block is written to the first statistics library and the asynchronous symbol buffer. When the data block ends, the process immediately switches to the second statistics library. At the same time, a Huffman tree is constructed using the data in the first statistics library, and data is read back from the asynchronous symbol buffer for encoding. Continuously cache front-end data while the encoding tree is being built on the backend; Backend encoding steps: In a single cycle, multiple symbols are mapped in parallel into micro-code packets encapsulated with codewords and code lengths. A finite state machine is used to parse the micro-code packets slot by slot, and variable-length codewords are concatenated into an output bit stream through shift operations.

7. An electronic device, characterized in that, include: Memory, configured to store computer programs, instructions, or configuration bit streams; The processor is coupled to the memory; A communication interface is configured to enable communication between the electronic device and other devices; wherein the processor is configured to run the computer program, instructions, or load the configuration bit stream to implement the steps of the method as described in claim 6.

Images