FPGA-based window-by-window full-pipelined LZ4 lossless compression method and system

Abstract

The application relates to the technical field of data compression, and provides a window-by-window full-pipeline LZ4 lossless compression method and system based on FPGA. The window-by-window full-pipeline LZ4 lossless compression method based on FPGA comprises the following steps: outputting a 4-byte sliding window and a corresponding hash value every clock cycle from the 4th beat; based on the 4-byte sliding window and the corresponding hash value, completing hash dictionary lookup, updating, validity verification and window content comparison in a 5-stage pipeline, and outputting the matching result of each window; performing operation classification in combination with historical states, outputting a classification result according to five rule classifications, counting a cumulative literal quantity and a matching quantity, and writing the literal quantity bytes into a distributed RAM buffer area and outputting to an internal FIFO; taking out the meta-information of a completed sequence from the FIFO, and generating a compression output in an LZ4 standard format sequence by sequence. Two feedback loops can be fundamentally eliminated.

Description

Technical Field

[0001] This invention relates to the field of data compression technology, and in particular to an FPGA-based window-by-window fully pipelined LZ4 lossless compression method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The LZ4 lossless compression algorithm, as one of the most widely used high-speed compression algorithms in big data and real-time systems, plays a core role in scenarios such as data center network transmission, storage system log compression, embedded sensor data stream processing, and FPGA accelerator inter-communication. Based on the LZ77 dictionary compression concept, LZ4 uses a hash dictionary to find repeating byte sequences in the input data, representing these repeating sequences as references to historical data (offset + match length), thereby reducing data storage or transmission volume. In a typical LZ4 software implementation, the algorithm calculates the hash value of the current 4-byte window at each position in the input data stream and looks it up in the dictionary (hash table). If a match is found and the offset is less than 64KB, a "match" is generated. After a match is generated, it compares byte by byte along the historical direction to determine the actual match length, then skips the entire matched segment and continues processing subsequent data. This "match → determine length → skip matching segment" process is natural and efficient in serial software execution, but it forms two fatal feedback loops in the FPGA full pipeline hardware implementation: Feedback loop one, the matching length is determined by byte-by-byte forward comparison (i.e., while(data[i]==data[prev+i]) i++ logic), the size of the matching length directly determines the starting position of the next byte search (skipping the entire matching segment) and the timing of hash table updates. This "match length → next byte position → hash table update" feedback loop makes the processing of each byte unable to be independent, and the pipeline must wait for the current match to complete before continuing; Feedback loop two, the token byte format of the LZ4 sequence is {literal_length[3:0], match_length[3:0]}, that is, the literal length and the matching length must be encoded into the same token byte at the same time. In the actual data flow, after a literal segment ends, the matching segment begins. Before the matching segment ends, the matching length cannot be determined, and therefore the token cannot be constructed. This constitutes the second feedback loop of "literal segment → matching segment → matching length → token encoding".

[0004] Existing LZ4 compression implementations on FPGAs all retain the two core feedback loops mentioned above to varying degrees. Therefore, existing technologies suffer from the following systemic shortcomings: First, the fixed match length constitutes a data-dependent feedback loop, preventing independent and simultaneous processing of each byte. Pipeline throughput fluctuates with data content, making deterministic, fixed-latency processing impossible. Second, existing solutions either weaken the loop by limiting the upper limit of the match length (sacrificing compression ratio) or absorb latency fluctuations through multi-port memory / FIFO buffers (increasing resource overhead), neither of which completely eliminates the feedback loop at the data flow reconstruction level. Third, existing solutions generally rely on DSP hard cores (DSP48 Fibonacci multiplicative hash) and dual-port or multi-port memory, resulting in high resource consumption and limiting deployment flexibility on low-resource FPGA platforms. Summary of the Invention

[0005] To address the data-dependent negative feedback circuit problem in the FPGA hardware implementation of the LZ4 lossless compression algorithm, this invention provides an FPGA-based window-by-window fully pipelined LZ4 lossless compression method and system, which can fundamentally eliminate two feedback loops without limiting the upper limit of the matching length, without using DSP hard cores, and without relying on multi-port memory.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides an FPGA-based window-by-window fully pipelined LZ4 lossless compression method.

[0007] A window-by-window fully pipelined LZ4 lossless compression method based on FPGA includes: Input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and from the 4th clock cycle onwards, a 4-byte sliding window and its corresponding hash value are output each clock cycle. Based on a 4-byte sliding window and its corresponding hash value, the hash dictionary lookup, update, validity verification, and window content comparison are completed in a 5-stage pipeline, and the matching result of each window is output. The classification is performed based on the matching results and historical status of each window, according to the five rules, and the classification results are output. Each time, the literal count and the matching count are accumulated based on the classification results, and the literal bytes are written to the distributed RAM buffer and output to the internal FIFO according to the set conditions. Extract the metadata of the completed sequence from the FIFO and generate compressed output sequence by sequence according to the LZ4 standard format.

[0008] Furthermore, the input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and from the 4th clock cycle onwards, a 4-byte sliding window and its corresponding hash value are output each clock cycle; the method includes: Input data enters a 4-byte shift register at a rate of 1 byte per clock cycle. For the first 3 clock cycles, the data is accumulated but not shifted out. From the 4th clock cycle onwards, a complete 4-byte sliding window and the corresponding 12-bit hash value are output each clock cycle. The hash value is calculated using the following formula: mixed = window ⊕ (window>>12) ⊕ (window>>24), hash = mixed[11:0] ⊕ mixed[19:8].

[0009] Furthermore, the validity verification includes: the hash value of the current window has a valid record in the corresponding slot of the hash table, the distance between the current position and the record position in the table is less than 64KB, and the historical window value stored in the table is equal to the current window value bit by bit; if all conditions are met, it is determined to be a match hit, and when a match is hit, the matching offset is equal to the current position minus the position in the table.

[0010] Furthermore, the method involves performing classification based on the matching results and historical states of each window, classifying according to the five-rule classification, and outputting the classification results; the method includes: Rule A: The first window triggers, forcibly marking it as a literal window: lit_match=0, seq_start=1; Rule B: Highest priority, triggered when delay_flag ≠ 0: The current window is a delayed transition window, lit_match=1, marked as the matching quantity, delay_flag decrements by 1, when delay_flag is reduced to 2, seq_end=1 is marked, the old run ends, when delay_flag is reduced to 1, seq_start=1 is marked, the new run begins; Rule C: Triggered when offsets are the same and positions are consecutive: maintain the literal / match type of the previous window unchanged; Rule D: Triggered when the offset changes and the previous frame is a matching value, the current window is marked as a matching window lit_match=1, and the delay window counting with delay_flag=3 is started at the same time. The offset of this match run uses the old offset value of the previous frame. Rule E: Triggered when offset changes and the preceding frame is a literal, marking the current window as a matching window, switch=1 indicates switching from literal to matching; the matching length of the matching run is equal to the number of matching windows in that run.

[0011] Furthermore, in each iteration, the literal count and matching count are accumulated based on the classification result, and the literal bytes are written into a distributed RAM buffer and output to an internal FIFO according to set conditions; the method includes: Each time, the literal count and the matching count are accumulated based on the classification results, and the literal bytes are written to the distributed RAM buffer. When seq_end=1, the metadata of the completed sequence is pushed into the internal FIFO. When the cumulative literal exceeds 384 bytes, the sequence is forcibly split midway and pushed into the internal FIFO; When there is no input data for more than 256 frames and there is still residual sequence data, it is automatically flushed to the internal FIFO.

[0012] Furthermore, the method of retrieving the metadata of the completed sequence from the FIFO and generating compressed output sequence by sequence according to the LZ4 standard format includes: The EM state machine initially starts in the IDLE state. Upon receiving valid input data, it transitions to the TKN state. In the TKN state, a Token byte is generated. The high 4 bits of the Token byte represent the literal length field `lit_len[3:0]`, and the low 4 bits represent the matching length field `mat_len[3:0]`. When the value of the literal length field `lit_len[3:0]` is 15, it transitions to the LIT_EXT state; otherwise, it directly transitions to the LIT state. In the LIT_EXT state, subsequent extended bytes are read continuously until the total accumulated literal length reaches the actual value, after which it transitions to the LIT state. In the LIT state, the original literal data is continuously output byte by byte from the input data stream according to the total length of the literal. After the output is completed, it jumps to the OFS state. In the OFS state, 2 bytes of offset data are continuously output. After the output is completed, it jumps to the MAT_EXT state. In the MAT_EXT state, when the value of the matching length field mat_len[3:0] is 15, the subsequent extended bytes are continuously read until the accumulated total matching length reaches the actual value. Then, it jumps back to the IDLE state and waits for the next round of processing. If the value of the matching length field mat_len[3:0] is not 15, it directly jumps back to the IDLE state. The output interface is a 256-bit data bus with a 32-bit byte enable signal, supporting burst output of up to 32 bytes per frame.

[0013] A second aspect of the present invention provides an FPGA-based window-by-window fully pipelined LZ4 lossless compression system.

[0014] A window-by-window fully pipelined LZ4 lossless compression system based on FPGA, comprising: The window component and hash calculation module is configured to: input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and outputs a 4-byte sliding window and the corresponding hash value every clock cycle starting from the 4th clock cycle; The hash table lookup and matching verification module is configured to: perform hash dictionary lookup, update, validity verification and window content comparison in a 5-stage pipeline based on a 4-byte sliding window and the corresponding hash value, and output the matching result of each window; The Run classification processing module is configured to perform classification based on the matching results and historical status of each window, classify according to the five rules, and output the classification results. The encoding accumulation module is configured to: accumulate literal counts and matching counts based on the classification results at each step, write the literal bytes to the distributed RAM buffer, and output them to the internal FIFO according to the set conditions; The encoding and transmission module is configured to extract the metadata of the completed sequence from the FIFO and generate compressed output sequence by sequence according to the LZ4 standard format.

[0015] A third aspect of the present invention provides a computer device comprising: A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method described in the first aspect above.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and to execute steps in the FPGA-based window-by-window fully pipelined LZ4 lossless compression method described in the first aspect above.

[0017] The fifth aspect of the present invention provides a computer program product or computer program.

[0018] This invention provides a computer program product or computer program comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the steps of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method described in the first aspect above.

[0019] Compared with the prior art, the beneficial effects of the present invention are: This invention liberates byte processing from "depending on the matching length result" through window-by-window isochronous sliding processing, fixing the processing delay of each window to 1 clock cycle and eliminating feedback loop one. Window-counting matching length determination, combined with the delay_flag delay window mechanism, replaces variable-length forward byte-by-byte comparison with a fixed 3-clock delay transition process, completely eliminating byte-level comparison in matching length determination. The FIFO-decoupled accumulation-emit dual-stage encoder splits the encoding process into a non-blocking accumulation end and an independently operating emitter end, fundamentally eliminating feedback loop two. These three elements work together to achieve truly fixed-delay, fully pipelined processing while maintaining full compatibility with the LZ4 standard format and unlimited matching length, without requiring a DSP hard core or multi-port memory, and with resource consumption only 1 / 5 to 1 / 6 of existing solutions. Attached Figure Description

[0020] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0021] Figure 1 This is a flowchart illustrating the FPGA-based window-by-window fully pipelined LZ4 lossless compression method in an embodiment of the present invention. Figure 2 This is a pipeline timing diagram shown in an embodiment of the present invention; Figure 3 This is a structural diagram of an FPGA-based window-by-window fully pipelined LZ4 lossless compression system as shown in an embodiment of the present invention; Figure 4 This is a structural diagram of a computer device shown in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0023] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0024] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0025] To facilitate understanding of the technical solutions of this invention, some technical terms involved in this invention will be introduced below.

[0026] 1. LZ4 (Lempel-Ziv 4): refers to a fast lossless compression algorithm based on the LZ77 idea. The core principle of LZ4 is to find the repeating byte sequence in the input data through a hash dictionary and represent the repeating sequence as a reference to historical data (offset + matching length), thereby reducing the amount of data storage or transmission.

[0027] 2. Negative Feedback Circuit: In hardware pipeline design, a feedback loop formed by data dependencies occurs where the initiation of subsequent processing steps depends on the output of preceding steps, causing the pipeline to be unable to advance with a fixed delay.

[0028] 3. FPGA (Field-Programmable Gate Array): refers to a type of digital integrated circuit that can be programmed and configured using hardware description languages ​​(such as Verilog and VHDL). This invention is based on Xilinx Kintex UltraScale+ series FPGA chips.

[0029] 4. BRAM (Block RAM): This refers to the dedicated memory core integrated on the FPGA chip. Each BRAM Tile (RAMB36) in the Xilinx UltraScale+ series FPGA provides 36Kbit of storage capacity and can be configured in single-port or dual-port mode.

[0030] 5. Pipeline: This refers to a hardware design technique that breaks down complex data processing into multiple sequentially executed stages. Each stage completes its subtask within one clock cycle and then passes the result to the next stage. Ideally, an N-stage pipeline can increase system throughput by N times.

[0031] 6. Hash Table: A hash table is a data structure that maps keys to storage addresses using a hash function, supporting lookup and insertion operations with O(1) time complexity. In LZ4 compression, the hash table is used to record the last occurrence position of each 4-byte window, and the current window determines whether there is a historical duplicate (i.e., a match) by looking up the hash table.

[0032] 7. Sliding Window: In LZ4 compression, a new sliding window is generated at each input byte position (covering the current byte and the three bytes before it) for each 32-bit data segment consisting of four consecutive bytes. The window is advanced step by step using a 4-byte shift register.

[0033] 8. Literal: refers to the original byte that cannot be found in the historical data and needs to be output directly to the compressed stream.

[0034] 9. Match: refers to a byte segment with the same content that can be found in historical data. It is referenced by offset (distance to the historical position) and match length (number of identical bytes), thereby avoiding duplicate storage.

[0035] 10. Run (Run Segment): In this invention, a run is a sequence segment of the same type consisting of multiple consecutive windows. A literal run consists of consecutive literal windows, and a matching run consists of consecutive matching windows. Run is the basic unit of encoding. Each run is encoded as an LZ4 sequence (containing a token byte, a literal byte, an offset, and an extended length). The boundaries of a run are automatically determined by the Stage6 A / B / C / D / E five-rule classifier.

[0036] 11. Delay window flag (delay_flag): is a 2-bit counter (value 0 / 3 / 2 / 1). When the offset of the matching run changes (triggered by rule D), a fixed 3-beat delay transition window (decrementing from 3 to 1) is automatically inserted to achieve a smooth transition at the boundary of the matching run.

[0037] 12. READ_FIRST mode: This refers to a working mode of FPGA BRAM. When reading and writing operations are performed on the same BRAM address in the same clock cycle, the read operation returns the old data (the value before writing) at that address, rather than the newly written data.

[0038] 13. FIFO (First-In-First-Out): refers to a queue structure used to buffer and decouple data streams between different processing stages.

[0039] 14. DSP (Digital Signal Processor): refers to the dedicated multiply-accumulate hard core integrated on the FPGA chip.

[0040] 15. LUT (Look-Up Table): refers to the basic programmable logic unit of an FPGA, which implements arbitrary combinational logic functions through a pre-stored truth table.

[0041] 16. Register (flip-flop / register): refers to the basic unit in an FPGA used to store sequential logic state values.

[0042] 17. LZ4 Block Format: This refers to the standard format specification for LZ4 compressed data. Each LZ4 block consists of several sequences. Each sequence contains: Token bytes (high 4 bits encode the literal length, low 4 bits encode the match length), optional literal length extension bytes (when the literal length is ≥15), literal bytes (byte-by-byte raw data), 2-byte offset (little-endian, the distance to the historical match position), and optional match length extension bytes (when the match length is ≥15).

[0043] 18. MNIB function (Minimum Number of Independent Bytes): refers to the length encoding function defined by the LZ4 specification.

[0044] 19. Calgary Corpus: refers to the international standard lossless compression test dataset.

[0045] Figure 1 This is a flowchart illustrating an FPGA-based window-by-window fully pipelined LZ4 lossless compression method according to an embodiment of the present invention; see reference. Figure 1 The method includes: Input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and from the 4th clock cycle onwards, a 4-byte sliding window and its corresponding hash value are output each clock cycle. Based on a 4-byte sliding window and its corresponding hash value, the hash dictionary lookup, update, distance verification, and window content comparison are completed in a 5-stage pipeline, and the matching result of each window is output. The classification is performed based on the matching results and historical status of each window. The five-rule classification is executed in descending order of priority, and the classification results are output. Each time, the literal count and the matching count are accumulated based on the classification results, and the literal bytes are written to the distributed RAM buffer and output to the internal FIFO according to the set conditions. Extract the metadata of the completed sequence from the FIFO and generate compressed output sequence by sequence according to the LZ4 standard format.

[0046] The proposed LZ4 compression method based on window-by-window isochronous processing employs an 8-stage pipeline architecture, reconstructing the LZ4 compression data processing flow from the traditional "jump-by-match segment processing" to "window-by-window isochronous sliding processing." Each input byte position independently constructs a 4-byte sliding window, undergoing a fixed 8-stage pipeline of hash calculation → hash lookup → match verification → run classification → encoding accumulation → encoding emission. Regardless of whether the byte ultimately belongs to a literal segment or a match segment, it occupies the same processing time per clock cycle. The matching length is determined by window counting (replacing byte-by-byte forward comparison), reads and writes simultaneously using the hash table READ_FIRST (eliminating the feedback dependency of lookup-update), and uses a FIFO-decoupled accumulation-emission two-stage encoder (eliminating the feedback wait for token construction).

[0047] At the overall architecture level, the 8-level pipeline is divided into five processing modules according to their functions.

[0048] Stage 1 is the window construction and hash calculation module: input data enters a 4-byte shift register at a rate of 1 byte per clock cycle. For the first 3 clock cycles, data is accumulated but not shifted out. From the 4th clock cycle onwards, a complete 4-byte sliding window and its corresponding 12-bit hash value are output each clock cycle. The hash function uses pure XOR and shift combinational logic (without multipliers), and the calculation formula is mixed = window ⊕ (window>>12) ⊕ (window>>24), hash = mixed[11:0] ⊕ mixed[19:8]. This invention uses a 4096-depth (12-bit address) single-port BRAM hash table, with each entry storing 57 bits of information (1 bit valid flag + 32 bits window value + 24 bits position value). The hash function uses pure XOR and shift combinational logic, without multipliers, and does not consume DSP resources.

[0049] Stages 2 through 5 are hash table lookup and matching verification modules, completing hash dictionary lookup, update, validity verification, and window content comparison in a 5-stage pipeline. The pipeline hardware operations are as follows: Stage 2 input capture → Stage 3 BRAM simultaneous read / write → Stage 4 Tcko isolation → Stage 5 matching comparison and output. The hash table depth is 4096 (12-bit address), and each entry stores 57 bits (1-bit valid flag + 32-bit window value + 24-bit position value).

[0050] Stage 2 latches the hash, window, and pos signals from Stage 1 into the input capture register (r1), cutting off the direct coupling between the upstream hash calculation combination logic and the downstream BRAM address input, while ensuring that the READ_FIRST read and write operations in Stage 3 share the same stable address.

[0051] In Stage 3, the BRAM is configured in READ_FIRST mode. Within the same always block, the semantic order of Verilog non-blocking assignment, which is "first evaluate all right-hand expressions (read old data), then execute left-hand assignment (write new data)," is used to achieve "old data lookup + new data update" in the same clock cycle of a single-port BRAM, thereby eliminating the feedback dependency of "hash table update → next lookup" in the traditional LZ4.

[0052] Stage 4 latches the BRAM read data into the Tcko isolation register (r2b), performing pure register forwarding without performing combinational operations. This stage isolates the clock-to-output delay of the BRAM36 hard core (Tcko, approximately 1.0~1.5ns, with overlay traces totaling approximately 2.0ns) from the matching comparison combinational logic of Stage 5 (24-bit subtraction + 32-bit window equality comparison, approximately 1.8ns) into two independent clock cycles. If combined, the total path would be approximately 4.1ns, exceeding the 300MHz clock cycle by 3.33ns, resulting in a timing violation. After inserting r2b, each path is approximately 2.3ns, providing ample margin.

[0053] Stage 5 uses combinational logic based on the r2b latch value to perform validity verification. The validity verification is based on three conditional logical ANDs: the hash slot has been written with valid data, the difference between the requested position and the position in the table is less than 64KB (LZ4 offset 16-bit encoding hard constraint), and the window value in the table is equal to the requested window value bit by bit (excluding hash false positives). The comparison result is latched and output to Stage 6. The 5-stage pipeline split isolates the BRAM Tcko and the window comparison combinational logic into different pipeline stages, ensuring timing convergence above 300MHz.

[0054] In this invention, the hash table dictionary is stored in a single-port BRAM configured in READ_FIRST mode. The simultaneous operation of "reading old data" and "writing new data" in the same clock cycle is achieved through Verilog non-blocking assignment semantics, avoiding the dependence on dual-port memory in traditional solutions.

[0055] The matching logic is as follows: the hash value of the current window has a valid record in the corresponding slot of the hash table, the distance between the current position and the record position in the table is less than 64KB, and the historical window value stored in the table is equal to the current window value bit by bit. If all three conditions are met, it is considered a match, and the matching offset is equal to the current position minus the position in the table. This determination is implemented in Stage 5 using single-shot combinational logic.

[0056] Stage 6 is the Run classification module. This module classifies each window into Runs based on matching results and historical states, executing five rules (A / B / C / D / E) from highest to lowest priority. Rule B (highest priority, triggered when delay_flag ≠ 0): The current window is a delayed transition window, lit_match = 1 (marked as a match), delay_flag is decremented by 1. When delay_flag reaches 2, seq_end = 1 (old Run ends), and when delay_flag reaches 1, seq_start = 1 (new Run begins). Rule A (triggered on the first window): Forces it to be marked as a literal window (lit_match = 0), seq_start = 1. Rule C (triggered when offset is the same and position is consecutive): Maintains the literal / match type of the previous window unchanged. Rule D (triggered when offset changes and the previous frame is a matched value): Marks the current window as a matched value window (lit_match=1), and starts delay_flag=3 for delay window counting. The offset of this match run uses the old offset value of the previous frame. This rule makes the end processing of the match run independent of the known match length—it only needs to confirm that the offset has changed to trigger the end of the run, and completes the transition through a fixed 3-frame delay window. Rule E (triggered when offset changes and the previous frame is a literal value): Marks the current window as a matched value window, switch=1 indicates the switch from literal to matched value. The match length of the match run is equal to the number of matched value windows in that run (window counting method), which is completed by fixed-delay combinational logic in Stage 7, without the need for forward byte-by-byte comparison.

[0057] The specific execution flow of the Run category is as follows: (1) Rule A - Initialize the first window. If the current window is the first valid window in the world, it is forcibly marked as a literal window, and a sequence start signal is issued to start the first literal Run.

[0058] (2) Rule B – Delay Window Processing. If the delay_flag of the previous window is not zero, the current window enters a delayed transition state: it is marked as a matching window, and delay_flag is decremented by 1 based on the previous frame. When delay_flag is reduced to 2, a sequence end signal (seq_end) is issued, indicating that the previous Run has ended; when delay_flag is reduced to 1, a sequence start signal (seq_start) is issued, indicating that the new Run starts from the next window, and the type of the new Run is determined according to the matching offset of the current window.

[0059] (3) Rule C - Continuous same offset maintenance. If the matching offset of the current window is the same as that of the previous window and the position is continuous, then the type of the previous window remains unchanged: if the previous window was a literal, then continue with literal; if the previous window was a match, then continue with match. The number of consecutive windows with the same offset is the window count of this Run.

[0060] (4) Rule D – Match Run End Transition. If the matching offset of the current window is different from that of the previous frame, and the previous frame was a matching quantity window, it means that the offset of the matching run has changed (the original matching run has ended or switched to a new matching source): mark the current window as a matching quantity window, and start a delayed transition with delay_flag of 3. The offset of this matching run continues to use the old value of the previous frame. The delayed window enters the decreasing process of Rule B from the next frame, and completes the run boundary switching within a fixed 3 frames. This rule is the key to replacing byte-by-byte comparison with window counting method – the end of the matching run is determined only by detecting whether the offset has changed, without needing to know the actual byte length of the matching segment.

[0061] (5) Rule E - Literal to Match Switching. If the matching offset of the current window is different from that of the previous window, and the previous window was a literal window, it means that the data has switched from the literal segment to the matching segment: mark the current window as a matching window and issue a type switching signal.

[0062] The length of a matching run is automatically determined by window counting: a matching run consists of several consecutive matching windows (excluding delay windows), and the matching length of the run is equal to the cumulative number of matching windows within it. This counting is naturally accumulated in Stage 7 encoding as the window classification results are processed in each frame. When the seq_end signal arrives, the counting is completed and the data is packaged and pushed into the FIFO. The entire process is completed by combinational logic and registers with fixed delays, without involving any byte-by-byte comparisons.

[0063] Stage 7 is the encoding accumulation module: Each time, based on the classification results from Stage 6, it accumulates the literal count (lit_cnt) and the matching count (mat_cnt), writing the literal bytes into a distributed RAM buffer (16 rows × 32 bytes = 512 bytes). When seq_end = 1, the metadata of the completed sequence (lit_cnt, mat_cnt, offset) is pushed into an internal FIFO (8 sequences deep), decoupling the accumulation and transmission ends. A built-in Force-Split safety mechanism is included: when the accumulated literal count exceeds 384 bytes (LIT_MAX_SAFE), the sequence is forcibly split midway and pushed into the FIFO to prevent buffer overflow. An built-in idle timeout automatic flushing mechanism is also included: when there is no input data for more than 256 times and there is still residual sequence data, it is automatically flushed to the FIFO, ensuring that all data is eventually encoded and output.

[0064] In this invention, decoupling is achieved between Stage 7 (encoded accumulation) and Stage 8 (encoded transmission) through an internal FIFO with a depth of 8 sequences. The accumulation end can continue to work without waiting for the transmission end to complete, and the transmission end retrieves completed sequences from the FIFO at its own pace for LZ4 format encoding and output. This FIFO decoupling fundamentally eliminates the second feedback loop in traditional LZ4 where "the literal length and the matching length must be ready simultaneously to construct the token byte".

[0065] Stage 8 is the encoding and emission module: the EM (Emit) state machine operates independently of the accumulator, retrieving the metadata of the completed sequence from the FIFO and generating compressed output sequence by sequence according to the LZ4 standard format. The EM state machine goes through IDLE → TKN (generating Token bytes {lit_len[3:0], mat_len[3:0]}, encoded using the mnib function) → LIT_EXT (extended bytes when the literal length is ≥15) → LIT (outputting literals byte by byte) → OFS (2-byte offset, little-endian (low byte first, high byte last)) → MAT_EXT (extended bytes when the matching length is ≥15) → back to IDLE. The external output interface is a 256-bit data bus, coupled with a 32-bit wide byte enable signal, where each bit corresponds to one byte in the 256-bit data bus; within each clock cycle, the number of valid bytes allowed to be output is 0 to 32 bytes, and burst output for multiple consecutive clock cycles is supported, with the position of the valid byte output in each clock cycle dynamically indicated by the corresponding bit of the byte enable signal. Pure literal sequences (without matching) omit offsets and match length extensions. The .mnib length encoding function is defined as: When the value is 0 to 3, the encoding is: ; =4 to 18 are encoded as -4; When the value is ≥19, it is encoded as 15, and one or more extension bytes are followed by the Token (the lower 8 bits of each extension byte are accumulated to the total length; if the value is 255, the next extension byte is continued until an extension byte with a value less than 255 is encountered). The compressed LZ4 data stream is composed of alternating literal segments and matching segments. The Token byte encodes the literal length and the matching length, each 4 bits, into one byte.

[0066] The delay alignment mechanism ensures correct signal synchronization across all pipeline stages: Stages 1 through 5 have a cumulative 5-step delay. The literal byte is synchronized with the matching result to the Stage 6 input via a 5-stage shift register. The input end flag (last) is synchronized with the Run classification result (rs_valid) via a 6-stage shift register. The specific pipeline timing diagram is shown below. Figure 2 As shown. Figure 2 The overall timing diagram of the 8-stage fully pipelined LZ4 compressor is shown. This architecture is divided into five collaborative modules according to data flow: Stage 1 completes byte-level window construction and pure logical hash calculation; Stages 2–5 ensure high-frequency timing convergence through 4-stage pipeline splitting, completing hash table lookup and match hit determination; Stage 6, as the core innovation stage, uses a five-rule pure combinational logic classifier to accurately classify the running segments; Stage 7 is responsible for the encoding accumulation of literals and matches and pushing them into the internal FIFO (with built-in forced splitting and timeout flushing safety mechanisms); Stage 8 independently runs the EM state machine, transmitting compressed data sequence by sequence according to the LZ4 standard. This design achieves three core technological breakthroughs: halving storage resources by using a single-port BRAM READ_FIRST mode for simultaneous "read old, write new" operations; proposing a delay_flag mechanism to replace byte-by-byte forward comparison with a fixed 3-step delay transition, making the match length unlimited; and employing a FIFO-decoupled "accumulation-emission" two-stage architecture. The delay window flag mechanism decouples the end determination of the matching run from the specific matching length—only the offset change needs to be detected, without performing byte-by-byte forward comparisons. The matching length of the matching run is equal to the number of matching windows in that run (window counting method), which is accomplished by the fixed-delay combinational logic in the Stage7 encoding accumulation. This invention implements a fixed-delay full pipeline—regardless of changes in the input data content, each byte position experiences the exact same 8 clock cycles (26.7ns @300MHz) throughout the entire pipeline, with a constant throughput of 1 byte per clock cycle (300MB / s), meeting the deterministic requirements of hard real-time systems.

[0067] This invention reconstructs the traditional LZ4 data flow of "jumping processing by matching segment" into "isochronous sliding processing with 4-byte windows," ensuring that each input byte independently undergoes the same pipeline. This fundamentally eliminates the feedback dependency of matching length on processing timing, while not imposing an upper limit on the matching length, and does not rely on DSP hard cores or multi-port memory. It achieves a compact FPGA compressor using pure LUT and BRAM combinational logic. The core technical challenges are concentrated in three aspects. First, how to determine the matching length without performing byte-by-byte forward comparisons—in traditional methods, the matching length feedback determines the lookup starting point for the next byte, forming a core negative feedback loop; this invention uses an A / B / C / D / E five-rule Run classifier combined with a delay_flag delay window mechanism, replacing byte-by-byte comparisons with window counting, thus having no upper limit on the matching length. Second, how to achieve same-cycle table lookup and update on a single-port BRAM—existing solutions rely on dual-port or multi-port memory; this invention utilizes the READ_FIRST mode combined with Verilog non-blocking assignment semantics to complete reading old and writing new within the same clock cycle, reducing BRAM resources by nearly half. Thirdly, how to eliminate the second feedback loop in the encoding structure—traditional tokens require simultaneous encoding of literals and matching lengths, leading to encoding blockage; this invention splits the encoder into a FIFO-decoupled accumulator and transmitter, with the accumulator never blocking. Furthermore, all hash calculations (XOR + shift), matching determination, Run classification, and length encoding are completed in LUT + BRAM, achieving a zero-DSP pure logic solution.

[0068] This invention achieves significant performance and resource efficiency improvements on the Xilinx Kintex UltraScale+ xcku040-ffva1156-2 FPGA chip. The compressor is designed with a target clock of 300MHz (cycle 3.333ns), and after synthesis, it can reach approximately 294MHz (cycle approximately 3.398ns, Setup WNS of -0.065ns, 48 ​​failed endpoints representing 0.41% of all 11,600 endpoints, expected to converge to 300MHz after placement and routing). Based on the achievable frequency, the compressor processes input data at a constant rate of 1 byte per clock cycle, with a throughput of approximately 294MB / s. The total pipeline delay from input to encoded output is fixed at 8 clock cycles (approximately 27.2ns @ 294MHz), with 6 clock cycles from input to Run classification completion and 2 clock cycles from Run completion to encoded transmission initiation. The delay is completely independent of the data content; regardless of the length of the matching segment in the data, the processing delay per byte remains the same, meeting the deterministic requirements of a hard real-time system. In terms of FPGA resource usage, 2,590 LUTs are used (1.07% of the total 242,400 KU040), 6,055 Registers are used (1.25% of the total 484,800), 7.5 BRAM tiles are used (including 7 RAMB36 and 1 RAMB18, accounting for 1.25% of the total 600), 0 DSPs are used, and 44 CARRY8s are used (0.15%).

[0069] This invention uses the international standard Calgary Corpus test suite (consisting of 18 files (approximately 3.25MB in total), covering various data types including text files (bib, book1, book2, news, paper1 to paper6), binary object code files (obj1, obj2), image files (pic), geographic data (geo), program source code files (progc, progl, progp), and transcription data (trans)) for functional verification. All 18 files were compressed using FPGA, decompressed using the python-lz4 algorithm library (lz4.block.decompress), and compared byte-by-byte. The verification content showed a 100% match, proving that the compression format fully conforms to the LZ4 block format standard. The average compression rate was 64.4% (original 3,251,493 bytes → compressed 2,094,727 bytes). The original data is shown in Table 1.

[0070] Table 1. Compression Ratio of Calgary Corpus

[0071] This invention uses the international standard Calgary Corpus test suite (18 files, totaling 3,251,493 bytes, covering various data types such as text, binary object code, images, geographic data, and program source code) for functional verification. All 18 files were compressed using FPGA, decompressed using the python-lz4 algorithm library (lz4.block.decompress), and compared byte-by-byte. The verification content showed a 100% match, proving that the compression format fully conforms to the LZ4 block format standard. The average compression rate was 64.4% (original 3,251,493 bytes → compressed 2,094,727 bytes), with compression rates ranging from 22.8% (high-repetition image data pic) to 97.3% (high-entropy geographic data geo).

[0072] Technical contribution analysis shows that the performance advantage of this invention stems from system-level optimization brought about by data flow reconstruction: window-by-window isochronous processing eliminates feedback loop one, making the processing delay per byte fixed at 1 clock cycle and unaffected by the matching length, decoupling throughput from data content; window counting matching length determination replaces variable-length forward byte-by-byte comparison with a fixed 3-clock delay window transition, with no upper limit on matching length and no compression ratio loss; single-port BRAM READ_FIRST same-clock read / write eliminates the dependence on dual-port memory, reducing BRAM resource requirements by nearly half; FIFO decoupled accumulation-emission two-level encoding eliminates feedback loop two, ensuring the accumulation end never blocks; zero-DSP pure LUT+BRAM implementation (hash function XOR+shift replaces Fibonacci multiplication), with total resource consumption of only 1.07% for LUT, 1.25% for Register, and 1.25% for BRAM, saving approximately 5 to 6 times the LUT resources compared to the closest existing FPGA LZ4 solution (Benes 2019: 14,076 LUTs; HybriDC 2022: (12,336 LUTs) can be deployed on low-resource FPGA platforms or coexist on the same chip with other functional modules such as network protocol stacks and encryption engines.

[0073] The hash table depth parameter (HASH_BITS) of this invention is both universal and scalable. The main implementation uses a depth of 4096 (12-bit address), which can be scaled up to 8192 (13-bit address) on FPGA platforms with more abundant BRAM resources to achieve a higher hash hit rate, with an expected compression ratio improvement of 2 to 5 percentage points. On platforms with limited BRAM resources, the depth can be reduced to 1024 (10-bit address) or 512 (9-bit address), sacrificing compression ratio for further savings in BRAM resources. The hash table depth is a power of 2 with BRAM consumption and can be flexibly configured according to the total BRAM capacity of the target FPGA model. The core READ_FIRST single-port BRAM simultaneous read / write method, the five-rule Run classifier, and the FIFO decoupling encoder are all unaffected by changes in hash table depth.

[0074] This invention features an extremely low resource footprint for a single compressor core (LUT only 1.07%, BRAM only 1.25%), allowing multiple independent compression cores to be instantiated on the same FPGA chip to achieve linear throughput scaling. Each core has its own independent hash table (BRAM), run classifier, and encoder, with complete decoupling between cores—no shared state, no interconnect communication, and no memory access contention. Input data is distributed to each core through simple distribution logic (such as round-robin based on data blocks or segmentation based on a fixed number of bytes). After each core independently completes compression, the output stream is merged through an arbitrator. N cores operating in parallel can achieve N times the throughput, while the total resource footprint is approximately N times that of a single core (still far lower than the total chip resources). Taking the KU040 chip as an example, the LUT utilization rate is only 1.07%, theoretically allowing for the instantiation of dozens of cores, with a total throughput reaching several GB / s to tens of GB / s. This multi-core solution complements the single-core multi-byte parallel approach: the multi-core solution is simple to implement, each core does not interfere with the others, and is suitable for block-level parallel scenarios; the single-core multi-byte solution is suitable for single-data-stream scenarios requiring low latency.

[0075] Porting and adaptation for different FPGA platforms. This invention is not limited to the Xilinx Kintex UltraScale+ series. On Intel (Altera) FPGA platforms, BRAM corresponds to M20K memory blocks, READ_FIRST mode corresponds to "Read OldData" mode, and Verilog non-blocking assignment semantics are consistent with the Xilinx platform, allowing for direct porting. On Lattice or domestic FPGA platforms, adaptation is required based on the specific BRAM primitives and read / write modes, but the core window-by-window isochronous processing architecture, five-rule Run classifier, FIFO decoupling encoder, and zero-DSP hash calculation method are platform-independent and reusable. On FPGAs of different speed levels, different timing convergence targets can be adapted by adjusting the number of hash table pipeline stages (increasing the number of intermediate register stages).

[0076] This invention employs a 256-bit output bus with a 32-bit byte enable. In scenarios with limited I / O resources, the output width can be reduced to 64-bit or 128-bit (synchronously reducing the byte enable width), at the cost of increasing the number of output cycles required for each LZ4 sequence at the transmitter. In scenarios requiring extremely rapid burst output, the width can be expanded to 512-bit, further increasing the maximum number of bytes output per cycle. Adjusting the output width only affects the EM state machine and output buffer logic of Stage 8, without affecting the processing architecture and compression rate of Stages 1 to 7.

[0077] Based on the compressor of this invention, an AES-GCM encryption module or a CRC32 checksum generation module can be cascaded after Stage8 encoded transmission to achieve integrated compression-encryption or compression-checksum processing. The encryption / checksum module receives compressed data through a 256-bit standard bus interface, shares the same operating clock (300MHz) with the compressor, and forms a compression + encryption / checksum composite pipeline. The total latency only increases the number of pipeline stages of the subsequent modules. This extension does not change the internal architecture of the compressor, and the modules use a standard bus interface, making it suitable for application scenarios with additional requirements for data security and integrity.

[0078] The above combination Figure 1 The window-by-window fully pipelined LZ4 lossless compression method based on FPGA provided in the embodiments of the present invention has been described in detail. Next, the window-by-window fully pipelined LZ4 lossless compression system based on FPGA provided in the embodiments of the present invention will be described in conjunction with the accompanying drawings.

[0079] Figure 3 This is a schematic diagram of the structure of an FPGA-based window-by-window fully pipelined LZ4 lossless compression system, as shown in an embodiment of the present invention. (Refer to...) Figure 3 The system described in this invention includes: The window component and hash calculation module is configured to: input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and outputs a 4-byte sliding window and its corresponding hash value every clock cycle starting from the 4th clock cycle; The hash table lookup and matching verification module is configured to: based on a 4-byte sliding window and its corresponding hash value, complete hash dictionary lookup, update, validity verification and window content comparison in a 5-stage pipeline, and output the matching result of each window; The Run classification processing module is configured to perform classification based on the matching results and historical status of each window, classify according to the five rules, and output the classification results. The encoding accumulation module is configured to: accumulate literal counts and matching counts based on the classification results at each step, write the literal bytes to the distributed RAM buffer, and output them to the internal FIFO according to the set conditions; The encoding and transmission module is configured to extract the metadata of the completed sequence from the FIFO and generate compressed output sequence by sequence according to the LZ4 standard format.

[0080] In some embodiments, the window assembly and hash calculation module is specifically configured as follows: input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and only accumulates without shifting out in the first 3 clock cycles. From the 4th clock cycle onwards, a complete 4-byte sliding window and the corresponding 12-bit hash value are output in each clock cycle. The formula for calculating the hash value is: mixed = window ⊕(window>>12) ⊕ (window>>24), hash = mixed[11:0] ⊕ mixed[19:8].

[0081] In some embodiments, the validity verification includes: the hash value of the current window has a valid record in the corresponding slot of the hash table, the distance between the current position and the record position in the table is less than 64KB, and the historical window value stored in the table is equal to the current window value bit by bit; if all conditions are met, it is determined to be a match hit, and when a match is hit, the matching offset is equal to the current position minus the position in the table.

[0082] In some embodiments, the Run classification processing module is specifically configured as follows: Rule A: The first window triggers, forcibly marking it as a literal window: lit_match=0, seq_start=1; Rule B: Highest priority, triggered when delay_flag ≠ 0: The current window is a delayed transition window, lit_match=1, marked as the matching quantity, delay_flag decrements by 1, when delay_flag is reduced to 2, seq_end=1 is marked, the old run ends, when delay_flag is reduced to 1, seq_start=1 is marked, the new run begins; Rule C: Triggered when offsets are the same and positions are consecutive: maintain the literal / match type of the previous window unchanged; Rule D: Triggered when the offset changes and the previous frame is a matching value, the current window is marked as a matching window lit_match=1, and the delay window counting with delay_flag=3 is started at the same time. The offset of this match run uses the old offset value of the previous frame. Rule E: Triggered when offset changes and the preceding frame is a literal, marking the current window as a matching window, switch=1 indicates switching from literal to matching; the matching length of the matching run is equal to the number of matching windows in that run.

[0083] In some embodiments, the encoding accumulation module is specifically configured to: accumulate literal counts and matching counts based on the classification results at each frame, and write literal bytes into a distributed RAM buffer; when seq_end=1, push the metadata of the completed sequence into an internal FIFO; when the accumulated literal count exceeds 384 bytes, force the sequence to be split midway and push it into the internal FIFO; when there is no input data for more than 256 frames and there is still residual sequence data, automatically flush it to the internal FIFO.

[0084] In some embodiments, the encoding and transmission module is specifically configured as follows: the EM state machine is initially in the IDLE state, and after receiving valid input data, it jumps to the TKN state; in the TKN state, a Token byte is generated, the high 4 bits of which are the literal length field lit_len[3:0], and the low 4 bits are the matching length field mat_len[3:0]; when the value of the literal length field lit_len[3:0] is 15, it jumps to the LIT_EXT state, otherwise it jumps directly to the LIT state; in the LIT_EXT state, subsequent extended bytes are read continuously until the total length of the accumulated literal reaches the actual value, and then it jumps to the LIT state; in the LIT state, according to the literal length field lit_len[3:0], it jumps to the LIT_EXT state; otherwise, it jumps directly to the LIT state; in the LIT state, it reads the subsequent extended bytes continuously until the total length of the accumulated literal reaches the actual value, and then jumps to the LIT state; in the LIT state, it reads the literal length field lit_len[3:0] according to the actual value. The total length of the literal is continuously output byte by byte from the input data stream. After the output is completed, it jumps to the OFS state. In the OFS state, 2 bytes of offset data are continuously output. After the output is completed, it jumps to the MAT_EXT state. In the MAT_EXT state, when the value of the matching length field mat_len[3:0] is 15, the subsequent extended bytes are continuously read until the accumulated total matching length reaches the actual value. Then, it jumps back to the IDLE state and waits for the next round of processing. If the value of the matching length field mat_len[3:0] is not 15, it directly jumps back to the IDLE state. The output interface is a 256-bit data bus with a 32-bit byte enable signal, supporting a burst output of up to 32 bytes per frame.

[0085] According to embodiments of the present invention, the FPGA-based window-by-window fully pipelined LZ4 lossless compression system can correspond to the execution of the methods described in the embodiments of the present invention, and the above and other operations and / or functions of each module of the FPGA-based window-by-window fully pipelined LZ4 lossless compression system are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the code will not be elaborated here.

[0086] Referring to the structural diagram of the computer device shown in Figure 4, the computer device includes a processor, a communication interface, and a computer-readable storage medium. The processor, communication interface, and computer-readable storage medium can be connected via a bus or other means. The communication interface is used to receive and send data. The computer-readable storage medium can be stored in the memory of the computer device. The computer-readable storage medium is used to store computer programs, including program instructions. The processor is used to execute the program instructions stored in the computer-readable storage medium. The processor (or CPU, Central Processing Unit) is the computing and control core of the computer device. It is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to implement the corresponding steps in the embodiment of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method.

[0087] This embodiment provides a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the processing system of the computer device.

[0088] Furthermore, this storage space also contains one or more instructions suitable for loading and execution by the processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM memory or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0089] In one embodiment, the computer-readable storage medium stores one or more instructions; the processor loads and executes one or more instructions stored in the computer-readable storage medium to implement the corresponding steps in the above embodiment of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method.

[0090] This embodiment provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the corresponding steps in the above-described embodiment of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method.

[0091] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0092] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0093] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0094] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0095] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A window-by-window, fully pipelined LZ4 lossless compression method based on FPGA, characterized in that, include: Input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and from the 4th clock cycle onwards, a 4-byte sliding window and its corresponding hash value are output each clock cycle. Based on a 4-byte sliding window and its corresponding hash value, the hash dictionary lookup, update, validity verification, and window content comparison are completed in a 5-stage pipeline, and the matching result of each window is output. The classification is performed based on the matching results and historical status of each window, according to the five rules, and the classification results are output. Each time, the literal count and the matching count are accumulated based on the classification results, and the literal bytes are written to the distributed RAM buffer and output to the internal FIFO according to the set conditions. Extract the metadata of the completed sequence from the FIFO and generate compressed output sequence by sequence according to the LZ4 standard format.

2. The FPGA-based window-by-window fully pipelined LZ4 lossless compression method according to claim 1, characterized in that, The input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and from the 4th clock cycle onwards, a 4-byte sliding window and its corresponding hash value are output each clock cycle; the method includes: Input data enters a 4-byte shift register at a rate of 1 byte per clock cycle. For the first 3 clock cycles, the data is accumulated but not shifted out. From the 4th clock cycle onwards, a complete 4-byte sliding window and the corresponding 12-bit hash value are output each clock cycle. The hash value is calculated using the following formula: mixed = window ⊕ (window>>12) ⊕ (window>>24), hash = mixed[11:0] ⊕ mixed[19:8].

3. The FPGA-based window-by-window fully pipelined LZ4 lossless compression method according to claim 1, characterized in that, The validity verification includes: the hash value of the current window has a valid record in the corresponding slot of the hash table, the distance between the current position and the record position in the table is less than 64KB, and the historical window value stored in the table is equal to the current window value bit by bit; if all conditions are met, it is determined to be a match hit, and when a match is hit, the matching offset is equal to the current position minus the position in the table.

4. The FPGA-based window-by-window fully pipelined LZ4 lossless compression method according to claim 1, characterized in that, The method involves performing classification based on the matching results and historical states of each window, classifying according to five rules, and outputting the classification results; the method includes: Rule A: The first window triggers, forcibly marking it as a literal window: lit_match=0, seq_start=1; Rule B: Highest priority, triggered when delay_flag ≠ 0: The current window is a delayed transition window, lit_match=1, marked as the matching quantity, delay_flag decrements by 1, when delay_flag is reduced to 2, seq_end=1 is marked, the old run ends, when delay_flag is reduced to 1, seq_start=1 is marked, the new run begins; Rule C: Triggered when offsets are the same and positions are consecutive: maintain the literal / match type of the previous window unchanged; Rule D: Triggered when the offset changes and the previous frame is a matching value, the current window is marked as a matching window lit_match=1, and the delay window counting with delay_flag=3 is started at the same time. The offset of this match run uses the old offset value of the previous frame. Rule E: Triggered when offset changes and the preceding frame is a literal, marking the current window as a matching window, switch=1 indicates switching from literal to matching; the matching length of the matching run is equal to the number of matching windows in that run.

5. The FPGA-based window-by-window fully pipelined LZ4 lossless compression method according to claim 1, characterized in that, Each time a literal value is counted and a matching value is counted based on the classification result, and the literal value byte is written to a distributed RAM buffer and output to an internal FIFO according to set conditions; the method includes: Each time, the literal count and the matching count are accumulated based on the classification results, and the literal bytes are written to the distributed RAM buffer. When seq_end=1, the metadata of the completed sequence is pushed into the internal FIFO. When the cumulative literal exceeds 384 bytes, the sequence is forcibly split midway and pushed into the internal FIFO; When there is no input data for more than 256 frames and there is still residual sequence data, it is automatically flushed to the internal FIFO.

6. The FPGA-based window-by-window fully pipelined LZ4 lossless compression method according to claim 1, characterized in that, The method involves retrieving the metadata of the completed sequence from the FIFO and generating compressed output sequence by sequence according to the LZ4 standard format; the method includes: The EM state machine initially starts in the IDLE state. Upon receiving valid input data, it transitions to the TKN state. In the TKN state, a Token byte is generated. The high 4 bits of the Token byte represent the literal length field `lit_len[3:0]`, and the low 4 bits represent the matching length field `mat_len[3:0]`. When the value of the literal length field `lit_len[3:0]` is 15, it transitions to the LIT_EXT state; otherwise, it directly transitions to the LIT state. In the LIT_EXT state, subsequent extended bytes are read continuously until the total accumulated literal length reaches the actual value, after which it transitions to the LIT state. In the LIT state, the original literal data is continuously output byte by byte from the input data stream according to the total length of the literal. After the output is completed, it jumps to the OFS state. In the OFS state, 2 bytes of offset data are continuously output. After the output is completed, it jumps to the MAT_EXT state. In the MAT_EXT state, when the value of the matching length field mat_len[3:0] is 15, the subsequent extended bytes are continuously read until the accumulated total matching length reaches the actual value. Then, it jumps back to the IDLE state and waits for the next round of processing. If the value of the matching length field mat_len[3:0] is not 15, it directly jumps back to the IDLE state. The output interface is a 256-bit data bus with a 32-bit byte enable signal, supporting burst output of up to 32 bytes per frame.

7. An FPGA-based window-by-window fully pipelined LZ4 lossless compression system, characterized in that, include: The window component and hash calculation module is configured to: input data enters a 4-byte shift register at a rate of 1 byte per clock cycle, and outputs a 4-byte sliding window and its corresponding hash value every clock cycle starting from the 4th clock cycle; The hash table lookup and matching verification module is configured to: based on a 4-byte sliding window and its corresponding hash value, complete hash dictionary lookup, update, validity verification and window content comparison in a 5-stage pipeline, and output the matching result of each window; The Run classification processing module is configured to perform classification based on the matching results and historical status of each window, classify according to the five rules, and output the classification results. The encoding accumulation module is configured to: accumulate literal counts and matching counts based on the classification results at each step, write the literal bytes to the distributed RAM buffer, and output them to the internal FIFO according to the set conditions; The encoding and transmission module is configured to extract the metadata of the completed sequence from the FIFO and generate compressed output sequence by sequence according to the LZ4 standard format.

8. A computer device, characterized in that, A processor, adapted to execute computer programs; A computer-readable storage medium storing a computer program, which, when executed by the processor, implements the steps of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the steps of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the FPGA-based window-by-window fully pipelined LZ4 lossless compression method as described in any one of claims 1-6.

Images