Compressed data regular expression matching acceleration method and device based on FPGA

Abstract

The invention discloses a compressed data regular expression matching acceleration method and device based on an FPGA (Field Programmable Gate Array), belongs to the technical field of deep packet detection, and achieves the effects of improving the matching throughput rate and reducing the matching time delay by quickly skipping the matching of coded data in compressed data. The device comprises a matching module, a state transition table module, an annular storage, a coded data queue, a coded data matching state verification module and an output queue. Experiments show that the technical scheme of the invention can significantly improve the matching throughput rate of the regular expression of the compressed data and reduce the matching time delay of the regular expression of the compressed data.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep packet inspection, and relates to related technologies such as regular expression matching, data encoding compression, and FPGA hardware programming, and specifically to a compressed data regular expression matching acceleration method and device based on FPGA. Background Art

[0002] Regular expressions are widely used in deep packet inspection systems, such as network intrusion detection systems, firewall systems, and network traffic management systems, due to their flexible and powerful expression capabilities. With the rapid development of mobile Internet and the continuous emergence of related services and applications, network traffic has shown explosive growth. In order to improve network bandwidth utilization and reduce data packet transmission delay, more and more network services use compression technology to encode and compress transmission data. However, the massive amount of compressed data has brought bottlenecks and challenges in performance such as throughput and latency to network deep packet inspection systems. Specifically, taking the Gzip compression encoding algorithm widely used by HTTP servers as an example, the compression rate of this algorithm is about 20%, which means that the space occupied by the data after Gzip compression is reduced to 1 / 5. If the Gzip compressed data is decompressed before matching, the system matching throughput will be reduced by at least 4 / 5. Considering the time overhead caused by the decompression process, the matching delay will also increase.

[0003] At present, the research on compressed data regular expression matching methods can be divided into two categories: CPU-based software methods and FPGA-based hardware acceleration methods. The CPU-based software method runs on the operating system. When the compressed data reaches the network card, the network card sends an interrupt signal to the operating system kernel. The operating system kernel copies the data from the network card to the kernel buffer. When the detection system reads the data through the system call, it is copied from the kernel buffer to the user buffer. The detection system then decompresses and matches it, and finally outputs the matching results. The delay of the above processing flow is often in microseconds or even milliseconds. Therefore, the existing CPU-based software method is facing performance bottlenecks such as delay and throughput.

[0004] The hardware acceleration method based on FPGA uses its hardware programmable, highly parallel, and low latency features to accelerate matching, aiming to break through the performance bottleneck faced by the CPU. For example, reference [1] Yi-Hua Yang and Viktor Prasanna. High-performance and compact architecture for regular expression matching on FPGA. IEEE Transactions on Compmers, 61(7): 1013-1025, 2011. The regular expression is compiled into a logically equivalent non-determined finite automaton (NFA), and then the NFA is converted into an RTL logic circuit to improve the throughput. Reference [2] V, M, J.NFA reduction for regular expressions matching using FPGA[C] / / 2013 International Conference on Field-Programmable Technology(FPT).IEEE, 2013: 338-341. and literature [3] Hieu TT, Thinh TN, Tomiyama S.ENREM: An efficient NFA-based regular expression matching engine on reconfigurable hardware for NIDS[J].Journal of Systems Architecture, 2013, 59(4-5): 202-212. is committed to reducing the consumption of logic resources of NFA-based regular expression matching circuits on FPGA. Reference [4] Yang J, Jiang L, Tang Q, et al. PiDFA: A practical multi-stride regular expression matching engine based on FPGA [C] / / 2016 IEEE International Conference on Communications (ICC). IEEE, 2016: 1-7. A path fusion and multi-step matching method based on deterministic finite state automaton (DFA) on FPGA is proposed. Reference [5] Sun, Mingqian, et al. "PTME: A Regular Expression Matching Engine Based on Speculation and Enumerative Computation on FPGA." ACM Transactions on Reconfigurable Technology and Systems (2024). A device based on DFA on FPGA that supports dynamic updating of regular expression rule sets is proposed. However, the above FPGA-based hardware acceleration method can only process uncompressed data. When compressed data is input, it needs to be fully decompressed in advance before matching, and the redundant characteristics of compressed encoded data are not utilized. In fact, the performance of compressed data regular expression matching methods based on FPGA still has a lot of room for improvement. Summary of the invention

[0005] The present invention provides a compressed data regular expression matching acceleration method and device based on FPGA, which utilizes the redundant characteristics of compressed coded data to quickly skip the matching of coded data, thereby achieving the purpose of improving the regular expression matching throughput and reducing the matching delay.

[0006] In order to achieve the above object, the present invention adopts the following technical solution:

[0007] A compressed data regular expression matching acceleration method based on FPGA includes the following steps:

[0008] Step 1: Initialize the FPGA-based compressed data regular expression matching acceleration device:

[0009] Step 2: Determine whether the input queue is empty:

[0010] If the input queue is empty, the matching process is suspended;

[0011] Otherwise, the matching module reads a data unit from the input queue, and if the data unit is uncoded data, dispatches it to the uncoded data processor and jumps to step 3; otherwise, dispatches it to the coded data processor and jumps to step 4;

[0012] Step 3: The uncoded data processor reads the matching state from the state transition table and updates the matching state curState; then appends the updated matching state curState and the uncoded data to the ring storage, and increases the index index by 1; if the matching state curState is in the accepting state, encapsulates the matching state curState and the index index and writes them to the output queue; jumps to step 2;

[0013] Step 4: The coded data processor reads the coded data, reads the dictionary data and its matching status referenced by the coded data from the ring storage, records the read content as decompressed data and its predicted matching status, and re-appends and writes it back to the ring storage; and encapsulates the coded data, matching status curState and index index and writes them into the coded data queue; then updates the matching status curState to the final matching status of the dictionary data, and increases the length field value in the coded data by the index index; jumps to step 2 and step 5 to execute in parallel;

[0014] Step 5: Determine whether the encoded data queue is empty:

[0015] If the encoded data queue is empty, the verification process is suspended;

[0016] Otherwise, the encoding data matching state verification module reads an encapsulation unit from the encoding data queue, assigns the internal matching state localState of the encoding data matching state verification module to the matching state in the encapsulation unit, and reads the decompressed data and its predicted matching state from the ring storage; the predicted matching state is verified one by one starting from the first character of the decompressed data; after the verification is completed, the corrected predicted matching state is traversed, and if there is an acceptance state, the acceptance state and the corresponding index index are encapsulated and written to the output queue; after the traversal is completed, continue to step 5.

[0017] Furthermore, in step 3, the uncoded data processor combines the uncoded data and the matching state curState into an address through bit splicing, reads the matching state from the state transfer table module; and determines whether the matching state of the uncoded data is an accepting state through combinational logic.

[0018] Furthermore, in step 4, the encoding data processor calculates an address by using the index index and the distance field in the encoding data, reads the dictionary data and its matching status referenced by the encoding data from the ring storage based on the address, and writes the referenced dictionary data and its matching status back to the ring storage, thereby realizing the functions of decompressing data and predicting matching status.

[0019] Furthermore, in step 5, reading the decompressed data and its predicted matching status from the ring storage includes: the encoded data matching status verification module uses the index index in the encapsulation unit as the address value to read the decompressed data and its predicted matching status from the ring storage.

[0020] Furthermore, in the step 5, checking the predicted matching status one by one starting from the first character of the decompressed data includes: checking the predicted matching status starting from the first character of the decompressed data, combining the character and the internal matching status localState of the encoded data matching status verification module into an address through bit splicing, reading the actual matching status trueState from the state transfer table module, and comparing trueState with the predicted matching status of the first character: if they are not equal, correcting the predicted matching status of the character, and continuing to check the matching status of the next character; if they are equal, the verification is completed, and the corrected predicted matching status is written back to the ring storage.

[0021] Furthermore, in step 1, initializing the FPGA-based compressed data regular expression matching acceleration device includes: clearing all data storage space except the input queue; initializing the storage content of the state transfer table module; setting the matching state curState in the matching module to the initial value and the index index to 0.

[0022] A compressed data regular expression matching device based on FPGA, comprising an input queue, a matching module, a state transfer table module, a ring storage, a coded data queue, a coded data matching state verification module and an output queue;

[0023] An input queue, used to store input data written externally to the device, the input data including uncoded data and coded data;

[0024] A matching module, for reading input data from an input queue, matching unencoded data, decompressing the unencoded data, and predicting a matching state of the unencoded data;

[0025] A state transition table module is used to store the matching state of the deterministic finite state automaton compiled from the regular expression rule set;

[0026] Ring storage, used to store dictionary data and its matching status required by the matching module to decompress the encoded data;

[0027] The coded data queue is used to store the coded data to be verified and related information written by the matching module;

[0028] The coding data matching state verification module is used to verify the predicted matching state of the coding data and correct the wrong predicted matching state;

[0029] Output queue, used to store the accepting matching status and its index.

[0030] Further, the matching module includes a classifier, an uncoded data processor and a coded data processor;

[0031] The classifier is used to dispatch uncoded data to an uncoded data processor and to dispatch coded data to a coded data processor; the uncoded data processor is used to match the uncoded data and append the uncoded data and its matching status to a ring storage; the coded data processor is used to read the dictionary data and its matching status referenced by the coded data from the ring storage, and realize decompression and status prediction by writing the dictionary data and its matching status back to the ring storage.

[0032] Furthermore, the state transfer table module is constructed by a dual-port BROM IP core instance, connected to a matching module and a coded data matching state checking module through an address bus and a data bus, and outputs a matching state of uncoded data.

[0033] Furthermore, the ring storage is constructed using a dual-port BRAM IP core instance, and reads and outputs decompressed data and its matching status through a read-write control bus, an address bus, and a data bus connected to a matching module and a coded data matching status verification module.

[0034] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0035] (1) High throughput and low latency performance

[0036] The FPGA-based compressed data regular expression matching acceleration method and device proposed in the present invention utilizes the redundant characteristics of compressed encoded data to quickly skip the matching of encoded data, matching one unencoded data unit or one encoded data unit every two clock cycles. The performance evaluation experimental results show that when the system operating frequency is set to 200MHz, the throughput can reach 1.6Gbps, and the processing delay is 2 clock cycles, i.e. 10ns. Compared with the Baseline method of decompressing first and then matching, the throughput performance of the present invention is improved by up to 7.6 times, and the matching delay is reduced to as low as 12%. By instantiating the device multiple times on the FPGA board, the throughput will increase linearly, while the processing delay remains at 2 clock cycles.

[0037] (2) Good compatibility and strong applicability

[0038] The FPGA-based compressed data regular expression matching acceleration device proposed by the present invention uses a BROM IP core instance to store a state transition table of a regular expression rule set, and supports dynamic updating of the regular expression rule set without modifying the engineering source code. In addition, the device receives LZ77 algorithm encoded data input and widely supports matching algorithms based on LZ77, such as data compressed by LZSS, Gzip, Brotli and other algorithms.

[0039] (3) Good scalability

[0040] The FPGA-based compressed data regular expression matching acceleration device proposed in the present invention can be embedded in the server motherboard through the PCIe interface, and combined with the general-purpose CPU in the server to realize a regular expression matching system based on a heterogeneous computing architecture, reduce the computing load of the CPU, and use the hardware programmable, highly parallel, and low-latency characteristics of FPGA to break through the performance bottleneck faced by the general-purpose CPU, thereby providing users with a good experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a schematic diagram of the Gzip algorithm compression process. The original data is compressed by LZ77 and Huffman encoded, and output as Gzip format data;

[0042] Figure 2 is used Figure 1 Schematic diagram of the original data and the data format after LZ77 compression;

[0043] Figure 3It is a state transition diagram of DFA constructed using the regular expression "(ab*c)|(bc+d)";

[0044] Figure 4 It is a framework diagram of the FPGA-based compressed data regular expression matching acceleration device proposed by the present invention;

[0045] Figure 5 is a throughput performance comparison chart between the method proposed in the present invention and the Baseline method;

[0046] Figure 6 This is a delay performance comparison chart between the method proposed in the present invention and the Baseline method;

[0047] Figure 7 It is a flow chart of a compressed data regular expression matching acceleration method based on FPGA proposed by the present invention.

[0048] Among them, 101, input queue; 102, matching module; 103, state transfer table module; 104, ring storage; 105, encoded data queue; 106, encoded data matching state verification module; 107, output queue. DETAILED DESCRIPTION

[0049] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0050] It should be noted that when an element is referred to as being "disposed on" another element, it may be directly on the other element or there may be another element centered thereon. When an element is considered to be "connected to" another element, it may be directly connected to the other element or there may be another element centered thereon at the same time. The terms "upper", "lower", "front", "back", "inside", "outside", etc. used herein to indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those generally understood by those skilled in the art of the present invention. The terms used herein in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The term "and / or" used herein includes any and all combinations of one or more related listed items.

[0052] The technical solution of the present invention is clearly and completely described below in conjunction with a simple specific embodiment. Based on this embodiment, all other embodiments obtained by those skilled in the art without making any creative work should fall within the scope of protection of the present invention.

[0053] In the description of the present invention, unless otherwise specified, "plurality" means two or more than two. In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0054] The present invention provides a compressed data regular expression matching method and device based on FPGA, aiming to accelerate the regular expression matching of compressed data with the help of FPGA.

[0055] Example 1

[0056] This embodiment proposes a compressed data regular expression matching acceleration device based on FPGA. The device is designed in modules, including an input queue 101, a matching module 102, a state transfer table module 103, a ring storage 104, a coded data queue 105, a coded data matching state verification module 106, and an output queue 107. The device accelerates regular expression matching by quickly skipping the matching of coded data in compressed data, and finally realizes high-throughput, low-latency compressed data regular expression matching on an FPGA board.

[0057] In order to further explain the specific content of the technical solution of the present invention, the terms, professional terms and related technologies involved in the present invention are first explained:

[0058] (1) Gzip, Deflate

[0059] Gzip is a content encoding method used by HTTP 1.1. Deflate is the compression method used by Gzip, which includes the LZ77 compression algorithm and the Huffman coding algorithm.

[0060] In the process of compressing the original data, the LZ77 algorithm uses a sliding dictionary of 32K as a dynamic dictionary. During compression, it tries to find the longest substring that is the same as the current position from the dynamic dictionary, and encodes the substring that appears repeatedly at the current position in the format of <length, distance>. Figure 1The following is a schematic diagram of the Gzip compression process. The original data is "Hello, boys. Hello, girls." After being compressed by the LZ77 algorithm, the second "Hello," is encoded as a tuple <7, 13>, indicating that the length of the repeated substring is 7 bytes and the distance between it and the dictionary data is 13 bytes. Figure 2 shown.

[0061] The data compressed by the LZ77 algorithm includes unencoded data and encoded data, and then the Huffman encoding algorithm continues to encode it to generate the final Gzip format data. Since Huffman is a variable-length encoding algorithm, the length of the original characters after encoding is uncertain and is expressed as a continuous binary stream. Therefore, the existing compressed data regular expression matching method must be matched after Huffman decoding.

[0062] (2) Regular expression matching

[0063] Generally speaking, pattern matching methods are divided into two categories: string matching and regular expression matching. KMP, Aho-Corasick, Wu-Manber, etc. are all classic string matching algorithms. Strings can be regarded as regular expression patterns without metacharacters, so regular expression matching can cover string matching.

[0064] In automata theory, regular expressions can be converted into non-determined finite automatons (NFA) by Thompson construction, and NFA can be converted into deterministic finite automatons (DFA) by construction. The definition of DFA is A = (Q, ∑, δ, q0, F), which is defined as follows:

[0065] Q: a non-empty, finite set of states;

[0066] ∑: a non-empty, finite set of characters, called the input alphabet;

[0067] δ: transfer function, the input parameters are state and character, and the output is state;

[0068] q0: initial state, satisfying q0∈Q;

[0069] F: The set of accepting states that satisfies

[0070] Initially, DFA is in state q0. Each time a character is read, a state jump occurs according to the transfer function. The new state is the matching state of the input character. If the matching state belongs to the accepting state set, it means that the string is successfully matched. Figure 3 As shown, when the DFA constructed using "(ab*c)|(bc+d)" is used to match the string "abccd", the matching state path obtained is "11345", where "3" and "5" are accepting states, and the string "abccd" is successfully matched twice.

[0071] (3) Technical solution

[0072] On the basis of the above basic concepts and technologies, the basic theories on which the method proposed in the present invention relies and the technical solutions of the present invention are further introduced. According to the compression principle of the LZ77 algorithm, the original data can be compressed because there are repeated redundant parts in the data itself. The purpose of compression can be achieved by encoding these repeated redundant parts into encoded data that occupies less space. At present, there are related methods dedicated to using these compression codes to quickly skip the matching of redundant data in the compressed data to achieve the effect of accelerated matching.

[0073] Finite State Automaton (FSA) is a context-independent model, that is, the next transition state is only related to the active state and the next input character, and is independent of other historical states. In other words, two FSAs with the same initial state will have the same final state after receiving the same input sequence. Based on this, previous methods use the context-independent characteristics of FSA to quickly skip the matching of repeated data.

[0074] The FPGA-based compressed data regular expression matching acceleration method and device proposed by the present invention utilizes the context-independent characteristics of FSA. When processing redundant coded data, the method and device decompress the coded data by copying the dictionary data referenced by it, and quickly skips the matching of the coded data by copying the matching state of the dictionary data referenced by it as the predicted matching state of the coded data. In order to ensure that the predicted matching state is consistent with the actual matching state, the device asynchronously verifies the predicted matching state of the coded data through the coded data matching state verification module 106.

[0075] Reference Figure 4 A compressed data regular expression matching acceleration device based on FPGA includes an input queue 101, a matching module 102, a state transfer table module 103, a ring storage 104, a coded data queue 105, a coded data matching state verification module 106 and an output queue 107.

[0076] The input queue 101 is used to store input data written externally to the device, including uncoded data and coded data.

[0077] The matching module 102 is used to read input data from the input queue 101, match uncoded data, decompress uncoded data and predict the matching status of uncoded data; the matching module 102 includes a classifier 1021, an uncoded data processor 1022 and a coded data processor 1023.

[0078] The classifier 1021 is used to dispatch the input data read by the device to different processors, dispatch the uncoded data to the uncoded data processor 1022, and dispatch the coded data to the coded data processor 1023; the uncoded data processor 1022 is used to match the uncoded data, and append the uncoded data and its matching status to the ring storage 104; the uncoded data processor 1022 is used to read the dictionary data and its matching status referenced by the coded data from the ring storage 104, and realize decompression and state prediction by writing the dictionary data and its matching status back to the ring storage 104.

[0079] The state transfer table module 103 is used to store the matching state of the deterministic finite state automaton compiled by the regular expression rule set. The state transfer table module 103 is constructed by a dual-port BROM IP core instance, connected to the matching module 102 and the coded data matching state verification module 106 through the address bus and the data bus, and outputs the uncoded data matching state.

[0080] The ring storage 104 is used to store the dictionary data and its matching status required by the matching module 102 when decompressing the encoded data. The ring storage 104 is constructed using a dual-port BRAM IP core instance, and reads and outputs the decompressed data and its matching status through a read-write control bus, an address bus, and a data bus connected to the matching module 102 and the encoded data matching status verification module 106.

[0081] The coded data queue 105 is used to store the coded data to be verified and related information written by the matching module 102 .

[0082] The coding data matching state checking module 106 is used to check the predicted matching state of the coding data and correct an erroneous predicted matching state.

[0083] The output queue 107 is used to store the accepting state matching state and its index.

[0084] Specifically, the matching module 102 reads one uncoded data unit or one coded data unit from the input queue 101 each time, and the classifier 1021 distributes the read data to different processors.

[0085] When the unit read is an uncoded data (referred to as literal, 1-byte ACSII code value), the uncoded data processor 1022 combines the matching state (referred to as curState) maintained in the matching module (102) with the literal into an address {literal, curState}, reads the state transition table module 103, obtains the matching state of the literal, and records the matching state of the literal as newState. Then, newState is assigned to curState, and the literal and its matching state curState are appended to the ring storage 104. If curState is in the accepting state, that is, it is successfully matched by the regular expression rule, the literal, the matching state curState, and the index index are encapsulated into the form of {literal, curState, index} and written into the output queue 107.

[0086] When the data read is a coded data unit (denoted as a pointer), its data structure is a tuple<length,distance> , where the length field indicates the byte length of the dictionary data referenced by the pointer, and the distance field indicates the distance between the pointer and the dictionary data referenced by it. The coded data processor 1023 first calculates the serial number of the uncoded data referenced by the pointer, i.e., index-distance, according to the index maintained in the matching module 102 and the distance field in the pointer. Then, the serial number is used as the address to read the referenced dictionary data and its matching state from the ring storage 104, and the matching state of the dictionary data is used as the predicted matching state of the pointer. Subsequently, the dictionary data and the matching state of the dictionary data are appended to the ring storage (104) to realize the functions of decompression and state prediction. Since the predicted matching state may not be completely consistent with the actual matching state of the pointer, the pointer, the matching state curState, and the index are further encapsulated into the {pointer, curState, index} format and output to the coded data queue 105. Finally, the current matching state curState is updated to the final matching state of the dictionary data. In the above processing flow, two clock cycles are consumed for each processing of an uncoded data unit or a coded data unit.

[0087] The state transition table module 103 stores the state transition table, which describes the state transition that occurs after the DFA compiled by the regular expression rule set reads the input. Specifically, the module uses the dual-port ROM IP core instance to store the matching state, and the initial value is read from the COE file. When the input address signal is {literal, curState}, it means that the DFA is in the curState state and reads the matching state that will jump to 1 literal.

[0088] The ring storage 104 stores the dictionary data and its matching status needed to decompress the coded data. Specifically, the module uses a dual-port BRAM IP core instance as a storage medium to store the dictionary data and its matching status that the pointer may reference. The dictionary size supports dynamic adjustment. Taking the LZ77 encoding algorithm used by Gzip as an example, the maximum distance is 32K, and the maximum length field is 258. It is necessary to maintain the dictionary data and its matching status with a dictionary size of 32K. The BRAM instance storing literals is configured as follows: the storage row width is configured to 1024 bits, each storage row can store 1024 / 8=128 literals, and 32K / 128=256 storage rows are required. Considering that when the length in the pointer is 258, the matching module 102 needs to read 1+[257 / 128]=4 storage rows at a time, and a BRAM instance can only be read by the matching module 102 at a time. One storage row, therefore, 4 dual-port BRAM instances are instantiated as storage media, each instance contains 256 / 4=64 rows, and the BRAM instance storing the matching status is configured in the same way.

[0089] The coded data queue 105 uses a FIFO IP core instance to store the coded data encapsulation unit to be checked in the format of {pointer, curState, index} output by the matching module 102, and is used to buffer the difference in processing speed between the matching module 102 and the coded data matching state checking module 106.

[0090] The coded data matching state verification module 106 reads a coded data encapsulation unit {pointer, curState, index} to be verified from the coded data queue 105 each time, and uses index as the address to read the decompressed data of the pointer and its predicted matching state from the ring storage 104. Then, the literal in the decompressed data is combined with the previous matching state in sequence to form the read state transfer table module 103 to obtain the actual matching state trueState of the literal. Then, the actual matching state trueState is compared with the corresponding matching state read from the ring storage 104: if different, the matching state of the literal is corrected to the actual matching state trueState, and the next literal is checked, and the process is repeated until the states are the same; if the states are the same, the matching state of the pointer is checked and corrected, and then the corrected matching state is written back to the ring storage 104. Finally, the matching state that has been checked is traversed, and if the acceptance state is detected, the matching state and the corresponding index index are encapsulated and output to the output queue 107.

[0091] The output queue 107 uses the FIFO IP instance as a storage medium to store the compressed data regular expression matching results, including fields such as the acceptance state and index.

[0092] (4) Example

[0093] In order to more intuitively illustrate the working process of the device, the present invention is Figure 3 The DFA shown, combined with the input data "xxabccdaa<5,7>yy" as an example, illustrates the workflow of the present device.

[0094] Initialization: The matching module 102 initializes the index index to 0 and the matching state curState to 0; the state transition table module 103 is initialized, and the ROM IP core example is stored Figure 3 The state transition table of the DFA shown is shown in Table 1. For the sake of simplicity, the content in this table is not the entire content of the state transition table, and other states that do not appear are defaulted to 0.

[0095] Table 1 State transition table

[0096]

[0097] 1st clock cycle: the matching module reads the input character "x", combines it with the matching state curState into the address {x, 0}, and reads the matching state from the state transition table module 103;

[0098] The second clock cycle: At the rising edge of the system clock, the matching module 102 reads the matching state of "x" from the state transition table module 103 as "0", updates the matching state curState to "0", appends the character "x" and its matching state "0" to the ring storage, and increases the index index to 1;

[0099] The third clock cycle: at the rising edge of the system clock, the matching module 102 reads the input character "x", combines it with the matching state curState into the address {x, 0}, and reads the matching state from the state transition table module 103;

[0100] The fourth clock cycle: At the rising edge of the system clock, the matching module reads from the state transition table module 103 that the matching state of "x" is "0", updates the matching state curState to "0", and appends the character "x" and its matching state "0" to the ring storage 104, and increases the index index to 2;

[0101] The fifth clock cycle: at the rising edge of the system clock, the matching module 102 reads the input character "a", combines it with the matching state curState into the address {a, 0}, and reads the matching state from the state transition table module 103;

[0102] The sixth clock cycle: At the rising edge of the system clock, the matching module 102 reads the matching state of "a" from the state transition table module 103 as "1", updates the matching state curState to "1", and appends the character "a" and its matching state "1" to the ring storage 104, and increases the index index to 3;

[0103] From the 7th to the 18th clock cycle: the matching module 102 reads the characters in the string "bccdaa" one by one, repeats the steps of combining addresses, reading matching status from the state transition table module 103, updating matching status, appending characters and their matching status to the ring storage 104, and increasing index index; in particular, the matching module 102 reads the matching status of "c" as "3" in the 10th clock cycle and the matching status of "d" as "5" in the 14th clock cycle. Both "3" and "5" are accepting states, so the matching module encapsulates the matching status and its index index and writes them to the output queue;

[0104] 19th clock cycle: At the rising edge of the system clock, the matching module 102 reads the input coded data <5, 7>, indicating that the length of the dictionary data referenced by the coded data is 5, and the distance from the current position is 7. The starting position of the referenced dictionary data is calculated to be index-7=2, and 2 is used as the address to read the referenced dictionary data and the dictionary data matching status from the ring storage;

[0105] 20th clock cycle: At the rising edge of the system clock, the matching module 102 reads the dictionary data "abccd" and its matching state "11345" from the ring storage 104, and writes the dictionary data and its matching state to the ring storage 104. Through the above steps, the coded data is decompressed; at the same time, the coded data, curState, and index index are encapsulated into {<5, 7>, 1, 9} and output to the coded data queue 105, and the matching state curState is updated to "5", and index is increased by 5 and updated to 14;

[0106] 21st clock cycle: At the rising edge of the system clock, the matching module 102 reads the input character "y", combines it with the current matching state curState to form the address {y, 5}, and reads the matching state from the state transition table module 103;

[0107] 22nd clock cycle: at the rising edge of the system clock, the matching module 102 reads the matching state of "y" as "0" from the state transition table module 103, updates the matching state curState to "0", appends the character "y" and its matching state "0" to the ring storage 104, and updates the index index to 15; at the same time, the coded data matching state check module 106 takes out {<5, 7>, 1, 9} from the coded data queue 105, uses the index index=9 as the address, and reads the decompressed data of the coded data and its predicted matching state from the ring storage 104;

[0108] 23rd clock cycle: at the rising edge of the system clock, the matching module 102 reads the input character "y", combines it with the current matching state curState into the address {y, 0}, and reads the matching state from the state transition table module 103; at the same time, the coded data matching state verification module 106 reads the decompressed coded data "abccd" and its predicted matching state "11345" from the ring storage, combines the first character "a" of the decompressed data with the previous matching state "1" of the coded data into the address {a, 1}, and reads the matching state from the state transition table module 103;

[0109] 24th clock cycle: at the rising edge of the clock cycle, the coded data matching state verification module 106 reads from the state transition table module 103 that the actual matching state of the first character "a" of the decompressed data is "1", and compares the actual matching state "1" with the predicted matching state "1" of the first character "a". The comparison result is equal, so the verification is completed. Since no wrong prediction state occurs during the matching state verification of the coded data, there is no need to write the matching state back to the ring storage 104.

[0110] 25th to 29th clock cycles: On the rising edge of the clock cycle, the coded data matching status verification module 106 checks the predicted matching status of the coded data after verification one by one. Status "3" and "5" are acceptance status, and the status value and index index are encapsulated and written into the output queue respectively;

[0111] It can be seen from the above examples that the compressed data regular expression matching acceleration device based on FPGA proposed in the present invention can complete the matching of 1 byte of unencoded data or 1 encoded data within 2 clock cycles.

[0112] (5) Performance evaluation

[0113] In order to evaluate the throughput of this device when matching real compressed data, we selected data sets such as web pages crawled from the world's top 100 sites listed by Alex.com, Google search results pages, malicious web pages Malicious, and open source regular expression rule sets Snort24, Snort31, and Snort34 of the well-known network intrusion detection system Snort. The characteristics of the above data sets and rule sets are shown in Table 2 and Table 3, respectively.

[0114] Table 2 Dataset characteristics

[0115] Dataset Number of Pages Compressed size (MB) Decompressed size (MB) Alexa 434 16.3 71 Google 876 62.6 217 Malicious 4531 30.3 145

[0116] Table 3 Regular expression rule set characteristics

[0117]

[0118]

[0119] The FPGA-based compressed data regular expression matching acceleration device proposed in the present invention reads the LZ77 encoded data after Huffman decoding. We first perform Huffman decoding processing on the compressed data in the above data set through a software method to obtain LZ77 encoded format data, and then respectively count the number of clock cycles, number of data units, data size and other information consumed by this device to process the LZ77 encoded data of the above data set. The throughput and average processing delay of the present invention are calculated based on this information. The calculation formula is: Throughput = LZ77 encoded data size / number of clock cycles*operating frequency, average processing delay = number of clock cycles / number of data units. At the same time, in order to evaluate the throughput and delay performance improvement of the present invention compared to the method (Baseline method) of complete decompression and then matching, we also statistically calculated the throughput and processing delay of the Baseline method. The results are as follows: Figure 5 and Figure 6 As shown, from Figure 5 and Figure 6It can be seen that the FPGA-based compressed data regular expression matching acceleration method proposed in the present invention can improve the throughput by up to 7.6 times compared with the Baseline method, while the lowest delay is only 12% of the Baseline method. In addition, unlike the Baseline method, the FPGA-based compressed data regular expression matching method and device proposed in the present invention has a processing delay of 2 clock cycles, which will not change with different input data characteristics.

[0120] Example 2

[0121] A compressed data regular expression matching acceleration method based on FPGA, the workflow is as follows Figure 7 Specifically, the method comprises the following steps:

[0122] Step 1, initialize the compressed data regular expression matching acceleration device based on FPGA: clear all data storage spaces except the input queue 101; initialize the storage content of the state transfer table module 103; set the matching state curState in the matching module 102 to the initial value and the index index to 0;

[0123] Step 2: Determine whether the input queue 101 is empty:

[0124] If the input queue 101 is empty, the matching process is suspended;

[0125] Otherwise, the matching module 102 reads a data unit from the input queue 101, and the classifier 1021 dispatches the read data unit to different processors; if the data unit is uncoded data, it is dispatched to the uncoded data processor 1022 and jumps to step 3; otherwise, it is dispatched to the coded data processor 1023 and jumps to step 4;

[0126] Step 3, the uncoded data processor 1022 reads the matching state from the state transition table module 103, and updates the matching state curState maintained inside the matching module 102; then, curState and the uncoded data are appended to the ring storage 104, and the index index maintained inside the matching module 102 is increased by 1; if curState is in the accepting state, curState and index are encapsulated and written to the output queue 107; finally, jump to step 2;

[0127] Step 4, the coded data processor 1023 reads the coded data, reads the dictionary data and its matching status referenced by the coded data from the ring storage 104, records the read content as decompressed data and its predicted matching status, and re-appends and writes it back to the ring storage 104; and encapsulates the coded data, matching status curState and index index and writes them into the coded data queue 105; then curState is updated to the final matching status of the dictionary data, and the index index increases the length field value in the coded data; finally, jumps to step 2 and step 5 for parallel execution;

[0128] Step 5: Determine whether the coded data queue 105 is empty:

[0129] If the coded data queue 105 is empty, the verification process is paused; otherwise, the coded data matching state verification module 106 reads an encapsulation unit from the coded data queue 105, assigns the internal matching state localState of the coded data matching state verification module 106 to the matching state in the encapsulation unit, and reads the decompressed data and its predicted matching state from the ring storage; then, the predicted matching state is verified one by one starting from the first character of the decompressed data; after the verification is completed, the corrected predicted matching state is traversed, and if there is an acceptance state, the acceptance state and the corresponding index index are encapsulated and written to the output queue 107; after the traversal is completed, continue to execute step 5.

[0130] Preferably, in the above step 3, the uncoded data processor 1022 combines the uncoded data and the matching state curState into an address through bit splicing, reads the matching state from the state transfer table module 103; and determines whether the matching state of the uncoded data is an accepting state through combinational logic.

[0131] Preferably, in the above step 4, the coded data processor 1023 calculates the address by using the index index and the distance field in the coded data, reads the dictionary data and its matching status referenced by the coded data from the ring storage 104, and writes the referenced dictionary data and its matching status back to the ring storage 104, thereby achieving the purpose of decompressing the data and predicting the matching status, and realizing the function of quickly skipping the matching of the coded data.

[0132] Preferably, in the above step 5, the encoded data matching state verification module 106 uses the index index in the encapsulation unit as the address value, and reads the decompressed data and its predicted matching state from the ring storage 104; then the verification starts from the first character of the decompressed data, and the character and the module internal matching state localState are combined into an address through bit splicing, and the actual matching state trueState is read from the state transfer table module 103, and the trueState is compared with the predicted matching state of the first character: if they are not equal, the predicted matching state of the character is corrected, and the matching state of the next character is continued to be verified; if they are equal, the verification is completed, and the corrected predicted matching state is written back to the ring storage 104.

[0133] The above content is a further detailed description of the present invention in combination with a specific preferred embodiment. It cannot be determined that the specific embodiments of the present invention are limited to this. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as belonging to the present invention and the scope of patent protection determined by the submitted claims.

[0134] The term "consisting of" describing a combination shall include the identified elements, ingredients, parts or steps and other elements, ingredients, parts or steps that do not substantially affect the basic novel characteristics of the combination. The use of the terms "comprising" or "including" to describe a combination of elements, ingredients, parts or steps herein also contemplates embodiments that consist essentially of these elements, ingredients, parts or steps. By using the term "may", it is intended to illustrate that any of the attributes described that "may" be included are optional.

[0135] Multiple elements, ingredients, parts or steps can be provided by a single integrated element, ingredient, part or step. Alternatively, a single integrated element, ingredient, part or step can be divided into separate multiple elements, ingredients, parts or steps. The disclosure "one" or "an" used to describe an element, ingredient, part or step is not intended to exclude other elements, ingredients, parts or steps.

[0136] It should be understood that the above description is for illustration and not for limitation. Many embodiments and many applications beyond the examples provided will be apparent to those skilled in the art upon reading the above description. Therefore, the scope of the present teachings should not be determined with reference to the above description, but rather with reference to the foregoing claims and the full scope of equivalents to which such claims are entitled. For the purpose of comprehensiveness, all articles and references, including disclosures of patent applications and publications, are incorporated herein by reference. The omission of any aspect of the subject matter disclosed herein in the foregoing claims is not intended to be a waiver of such subject matter, nor should it be considered that the applicant has not considered such subject matter to be part of the disclosed inventive subject matter.

Claims

1. A compressed data regular expression matching acceleration method based on FPGA, characterized in that: The following steps are involved: Step 1: Initialize the FPGA-based compressed data regular expression matching acceleration device: Step 2: Determine whether the input queue (101) is empty: If the input queue (101) is empty, the matching process is suspended; Otherwise, the matching module (102) reads a data unit from the input queue (101), and if the data unit is uncoded data, dispatches it to the uncoded data processor (1022) and jumps to step 3; otherwise, dispatches it to the coded data processor (1023) and jumps to step 4; Step 3, the uncoded data processor (1022) reads the matching state from the state transition table (103) and updates the matching state curState; then appends the updated matching state curState and the uncoded data to the ring storage (104), and increases the index index by 1; if the matching state curState is in the accepting state, encapsulates the matching state curState and the index index and writes them to the output queue (107); jumps to step 2; Step 4, the coded data processor (1023) reads the coded data, reads the dictionary data and its matching status referenced by the coded data from the ring storage (104), records the read content as decompressed data and its predicted matching status, and re-appends and writes it back to the ring storage (104); and encapsulates the coded data, the matching status curState and the index index and writes them into the coded data queue (105); then updates the matching status curState to the final matching status of the dictionary data, and increases the length field value in the coded data by the index index; Jump to step 2 and step 5 and execute them in parallel; Step 5: Determine whether the coded data queue (105) is empty: If the coded data queue (105) is empty, the verification process is suspended; Otherwise, the coded data matching state checking module (106) reads a package unit from the coded data queue (105), assigns the matching state localState inside the coded data matching state checking module (106) to the matching state in the package unit, and reads the decompressed data and its predicted matching state from the ring storage (104); and checks the predicted matching state one by one starting from the first character of the decompressed data; After the verification is completed, the corrected predicted matching state is traversed. If there is an acceptance state, the acceptance state and the corresponding index index are encapsulated and written into the output queue (107); after the traversal is completed, step 5 is continued.

2. The method for accelerating the matching of compressed data regular expressions based on FPGA according to claim 1, characterized in that: In step 3, the uncoded data processor (1022) combines the uncoded data and the matching state curState into an address by bit splicing, reads the matching state from the state transfer table module (103), and determines whether the matching state of the uncoded data is an accepting state through combinational logic.

3. The method for accelerating the matching of compressed data regular expressions based on FPGA according to claim 1, characterized in that: In step 4, the coded data processor (1023) calculates an address by using the index index and the distance field in the coded data, reads the dictionary data and its matching status referenced by the coded data from the ring storage (104) based on the address, and writes the referenced dictionary data and its matching status back to the ring storage (104) to achieve the functions of decompressing data and predicting matching status.

4. The method for accelerating the matching of compressed data regular expressions based on FPGA according to claim 1, characterized in that: In the step 5, reading the decompressed data and its predicted matching status from the ring storage (104) includes: the encoded data matching status verification module (106) uses the index index in the encapsulation unit as the address value to read the decompressed data and its predicted matching status from the ring storage (104).

5. The method for accelerating the matching of compressed data regular expressions based on FPGA according to claim 1, characterized in that: In the step 5, checking the predicted matching state one by one starting from the first character of the decompressed data includes: checking the predicted matching state starting from the first character of the decompressed data, combining the character and the internal matching state localState of the coded data matching state checking module (106) into an address by bit splicing, reading the actual matching state trueState from the state transfer table module (103), and comparing trueState with the predicted matching state of the first character: if they are not equal, correcting the predicted matching state of the character and continuing to check the matching state of the next character; if they are equal, the checking is completed, and the corrected predicted matching state is written back to the ring storage (104).

6. The method for accelerating the matching of compressed data regular expressions based on FPGA according to claim 1, characterized in that: In the step 1, initializing the compressed data regular expression matching acceleration device based on FPGA includes: clearing all data storage spaces except the input queue (101); initializing the storage content of the state transfer table module (103); setting the matching state curState in the matching module (102) to an initial value and the index index to 0.

7. A compressed data regular expression matching device based on FPGA, characterized in that: It includes an input queue (101), a matching module (102), a state transfer table module (103), a ring storage (104), a coded data queue (105), a coded data matching state verification module (106) and an output queue (107); An input queue (101), used for storing input data written externally by the device, wherein the input data includes uncoded data and coded data; A matching module (102), configured to read input data from the input queue (101), match uncoded data, decompress the uncoded data, and predict a matching status of the uncoded data; A state transition table module (103), used for storing the matching state of the deterministic finite state automaton compiled from the regular expression rule set; A ring storage (104), used for storing dictionary data and matching status required by the matching module (102) when decompressing the encoded data; A coded data queue (105), used for storing the coded data to be verified and related information written by the matching module (102); A coding data matching state checking module (106), used to check the predicted matching state of the coding data and correct an erroneous predicted matching state; The output queue (107) is used to store the receiving state matching status and its index.

8. The FPGA-based compressed data regular expression matching acceleration device according to claim 7, characterized in that: The matching module (102) includes a classifier (1021), an uncoded data processor (1022) and a coded data processor (1023); The classifier (1021) is used to dispatch uncoded data to an uncoded data processor (1022), and dispatch coded data to a coded data processor (1023); the uncoded data processor (1022) is used to match the uncoded data, and append the uncoded data and its matching status to a ring storage (104); the coded data processor (1023) is used to read dictionary data and its matching status referenced by the coded data from the ring storage (104), and realize decompression and status prediction by writing the dictionary data and its matching status back to the ring storage (104).

9. The FPGA-based compressed data regular expression matching acceleration device according to claim 7, characterized in that: The state transfer table module (103) is constructed through a dual-port BROM IP core instance, connected to the matching module (102) and the coded data matching state checking module (106) through an address bus and a data bus, and outputs the uncoded data matching state.

10. The compressed data regular expression matching acceleration device based on FPGA according to claim 7, characterized in that: The ring storage (104) is constructed using a dual-port BRAM IP core instance, and reads and outputs decompressed data and its matching status through a read-write control bus, an address bus, and a data bus connected to a matching module (102) and a coded data matching status verification module (106).

Images