FPGA-based Full Offload Regular Matching System and Method

Abstract

This application relates to an FPGA-based full-offload regular expression matching system and method. The system includes a regular expression rule compilation unit and an FPGA full-matching unit. The regular expression rule compilation unit receives a set of regular expression rules, extracts fixed feature substrings from each rule, compiles a uniformly formatted isomorphic NFA, and constructs a mapping table between substrings and corresponding NFAs. The FPGA full-matching unit includes a parallel high-speed string matching engine array, a string-NFA mapping table module, a full regular expression data storage, a reconfigurable general-purpose NFA engine module, and a matching result output module. It performs full-offload regular expression matching on input network traffic: scanning the traffic through the engine array, filtering traffic containing fixed feature substrings to be verified and outputting substring identifiers, querying the NFA identifiers through the mapping table module, and loading the NFA for matching by the reconfigurable engine module. This method ensures low latency and high throughput performance for regular expression matching in high-speed network environments.

Description

Technical Field

[0001] This application relates to the field of network security technology, and in particular to an FPGA-based full offload regular expression matching system and method. Background Technology

[0002] Current mainstream cybersecurity systems widely rely on regular expressions as the language for describing rule features. Compared to fixed strings, regular expressions have stronger pattern generalization and semantic expressive capabilities, accurately capturing the characteristics of complex attack behaviors such as buffer overflows, SQL injection, and cross-site scripting (XSS). However, regular expression matching has high computational complexity, especially with large-scale rule sets (such as the Snort rule set containing nearly ten thousand regular expressions), where its high computational complexity severely limits matching efficiency.

[0003] At the software level, Intel's open-source Hyperscan library is a high-performance regular expression matching engine widely used in the industry. Although it has improved performance by optimizing NFA matching, multi-string pattern matching, and vectorized instructions, its throughput is still limited by CPU computing power and memory bandwidth. The actual throughput fluctuates wildly between tens of Mbps and several Gbps, and the latency is highly uncertain, making it difficult to meet the needs of modern high-speed data centers or backbone networks for deterministic low-latency processing.

[0004] To overcome performance bottlenecks, academia and industry have begun exploring FPGA-based hardware acceleration solutions. Due to the advantages of FPGAs, such as parallel processing, low latency, and customizable logic, some work has attempted to map regular expression matching logic to FPGA hardware. However, after regular expressions are compiled into deterministic finite automata (DFAs) or nondeterministic finite automata (NFAs), the number of states and the size of the transition table grow rapidly with the number of rules, leading to a rapid depletion of hardware resources (LUTs, registers, BRAM). Therefore, existing FPGA solutions generally adopt a "partial offloading" strategy: only fixed substrings or simple segments (such as anchor prefixes) in the regular expression are offloaded to the FPGA for high-speed filtering; when traffic hits these segments, the original packet is then uploaded to the CPU for software to perform complete regular expression verification.

[0005] While such partial offloading architectures can reduce average CPU load, in high-trigger-rate scenarios (such as dense attack traffic or high rule sensitivity), frequent fragment hits will cause a large number of packets to be sent back to the CPU, creating a new performance bottleneck. The overall system throughput will drop sharply, and deterministic latency guarantees cannot be provided. In addition, data transfer between the CPU and FPGA introduces additional overhead, diminishing the benefits of hardware acceleration.

[0006] Therefore, there is an urgent need for a new architecture that can achieve full offload matching of large-scale regular expressions under the constraints of limited FPGA resources, completely eliminate the CPU backhaul path, and support efficient deployment and line-speed processing of tens of thousands of rule sets. Summary of the Invention

[0007] Therefore, it is necessary to provide an FPGA-based full offload regularity matching system and method to address the aforementioned technical problems.

[0008] A fully offloaded regular expression matching system based on FPGA, the system comprising:

[0009] Regular expression compilation unit and FPGA full matching unit;

[0010] The regular expression rule compilation unit is used to receive a set of regular expression rules, extract the fixed feature substrings of each regular expression rule in the set of regular expression rules, compile each regular expression rule into a uniform format isomorphic NFA, and construct a mapping table between the fixed feature substrings and the corresponding isomorphic NFAs.

[0011] The FPGA full matching unit is communicatively connected to the regular expression compilation unit, and the FPGA full matching unit includes:

[0012] A parallel high-speed string matching engine array is configured with multiple string matching engines to perform parallel scanning of input network traffic, filter out the traffic to be verified that contains the fixed feature substring, and output the identification information of the corresponding fixed feature substring.

[0013] The string-NFA mapping table module stores the mapping relationship table and is used to query the corresponding isomorphic NFA identifier based on the identifier information.

[0014] A full regularization data storage unit is used to store the isomorphic NFAs corresponding to each of the regularization rules;

[0015] A reconfigurable general NFA engine module is configured with a general NFA engine, which is used to dynamically load the corresponding NFA from the full regular expression data storage according to the homogeneous NFA identifier, and perform full regular expression matching on the traffic to be verified.

[0016] The matching result output module is used to output the matching results of the reconfigurable general NFA engine module to an external processing unit, thereby achieving full unloading of regular expressions.

[0017] A full offload regular expression matching method based on FPGA, the method comprising:

[0018] Receive a set of regular rules and extract fixed feature substrings from each regular rule in the set of regular rules;

[0019] Each of the regular expression rules is compiled into a uniform format isomorphic NFA, and a mapping table between the fixed feature substrings and the corresponding isomorphic NFAs is constructed.

[0020] The mapping table is stored in the string-NFA mapping table module of the FPGA, and each of the isomorphic NFAs is stored in the full regular data memory of the FPGA.

[0021] The parallel string matching engine array of the FPGA performs parallel scanning of the input network traffic, filters out the traffic to be verified that contains the fixed feature substring, and outputs the identification information of the corresponding fixed feature substring.

[0022] The corresponding isomorphic NFA identifier is obtained by querying the identifier information through the string-NFA mapping table module.

[0023] The FPGA's reconfigurable general-purpose NFA engine module dynamically loads the corresponding isomorphic NFA from the full regular expression data memory based on the isomorphic NFA identifier, and performs full regular expression matching on the traffic to be verified.

[0024] The matching result output module of the FPGA outputs the matching result to the external processing unit, realizing the full unloading of regular expressions.

[0025] The aforementioned FPGA-based full-offload regular expression matching system and method extracts fixed feature substrings of regular expression rules through a regular expression rule compilation unit, compiles homogeneous NFAs in a unified format, and constructs a mapping table. This provides accurate data support for efficient matching on the FPGA side. Leveraging the parallel string matching engine array of the FPGA full-matching unit, it can perform parallel scanning of input network traffic, quickly filtering out traffic containing fixed feature substrings to be verified. The string-NFA mapping table module quickly queries the corresponding homogeneous NFA identifier. Combining the storage capacity of the full regular expression data storage device with the dynamic loading matching function of the reconfigurable general-purpose NFA engine, it achieves closed-loop complete regular expression matching within the FPGA. Finally, the matching result output module only reports the results to the external processing unit, without needing to send back the traffic to be verified, truly achieving full offload of regular expression matching. This invention effectively overcomes the performance bottleneck of traditional software engine throughput limitations and CPU back-to-verification in FPGA partial offload schemes, improving hardware resource utilization efficiency and ensuring low latency and high throughput performance of regular expression matching in high-speed network environments. Attached Figure Description

[0026] Figure 1 This is a block diagram of a fully offloaded regularization matching system based on FPGA in one embodiment;

[0027] Figure 2 This is a schematic diagram of the regular expression compilation process in one embodiment;

[0028] Figure 3 This is a schematic diagram of the runtime matching pipeline in one embodiment;

[0029] Figure 4 This is a schematic diagram of the internal structure of a reconfigurable NFA engine in one embodiment.

[0030] Figure 5 This is a schematic diagram of the architecture of a full offload regular expression matching system based on FPGA in a specific embodiment. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0032] In one embodiment, such as Figure 1 As shown, an FPGA-based fully offloaded regular expression matching system is provided, including:

[0033] Regular expression compilation unit and FPGA full matching unit;

[0034] The regular rule compilation unit is used to receive a set of regular rules, extract the fixed feature substrings of each regular rule in the set, compile each regular rule into a uniform format isomorphic NFA, and construct a mapping table between the fixed feature substrings and the corresponding isomorphic NFAs.

[0035] The FPGA full-matching unit communicates with the regular expression compilation unit. The FPGA full-matching unit includes:

[0036] The parallel high-speed string matching engine array is configured with multiple string matching engines to perform parallel scanning of input network traffic, filter out the traffic to be verified that contains fixed feature substrings, and output the identification information of the corresponding fixed feature substrings.

[0037] The string-NFA mapping table module stores the mapping relationship table, which is used to query the corresponding isomorphic NFA identifier based on the identifier information.

[0038] A full regularization data storage unit is used to store the isomorphic NFAs corresponding to each regularization rule;

[0039] The reconfigurable general NFA engine module is configured with a general NFA engine, which is used to dynamically load the corresponding NFA from the full regular expression data storage according to the homogeneous NFA identifier and perform full regular expression matching on the traffic to be verified.

[0040] The matching result output module is used to output the matching results of the reconfigurable general NFA engine module to the external processing unit, thereby achieving full unloading of regular expressions.

[0041] In the aforementioned FPGA-based fully offloaded regular expression matching system, the regular expression rule compilation unit extracts fixed feature substrings of regular expression rules, compiles homogeneous NFAs in a unified format, and constructs a mapping table. This provides accurate data support for efficient matching on the FPGA side. Leveraging the parallel string matching engine array of the FPGA's full matching unit, input network traffic can be scanned in parallel to quickly filter out traffic containing fixed feature substrings to be verified. The string-NFA mapping table module quickly queries the corresponding homogeneous NFA identifier. Combining the storage capacity of the full regular expression data storage device with the dynamic loading matching function of the reconfigurable general-purpose NFA engine, a closed-loop complete regular expression matching is achieved within the FPGA. Finally, the matching result output module only reports the results to the external processing unit, without needing to send back the traffic to be verified, truly achieving full offload of regular expression matching. This invention effectively overcomes the performance bottleneck of traditional software engine throughput limitations and CPU back-to-verification in FPGA partial offload schemes, improving hardware resource utilization efficiency and ensuring low latency and high throughput performance of regular expression matching in high-speed network environments.

[0042] It is worth noting that the system of this invention is applicable to deep packet inspection (DPI) scenarios in high-speed network environments (such as 100Gbps and above). Typical applications include network intrusion detection / prevention systems (IDS / IPS), next-generation firewalls (NGFW), malicious traffic identification and content auditing, etc. Addressing the core pain points of traditional software regular expression matching engines (such as Hyperscan) being limited by CPU computing power and memory bandwidth, resulting in large throughput fluctuations and uncertain latency, as well as existing FPGA partial offloading schemes needing to send packets back to the CPU for complete verification, creating new performance bottlenecks in high-trigger-rate scenarios, and traditional FPGA pure hardware solutions easily exhausting hardware resources due to NFA state explosions, this invention overcomes the resource constraints of traditional hardware solutions and the CPU dependence of partial offloading schemes through a uniquely designed dynamic loading general-purpose NFA engine, a full regular expression data storage architecture of FPGA high-density on-chip storage (such as URAM / BRAM), and a uniform format homogeneous NFA design. This invention achieves fully offloaded matching of FPGA internal closed loop for regular expressions of thousands to tens of thousands, without requiring the CPU to participate in any regular expression matching calculations. As a result, it provides deep packet inspection capabilities with high throughput, low latency and determinism in high-speed network environments of 100Gbps and above, and completely solves the problem of limited system throughput caused by the performance bottleneck of regular expression matching in traditional solutions.

[0043] To achieve the above objectives, this invention proposes a three-layer pipelined regular expression matching architecture. Through a coherent process of filtering, locating, and matching, regular expression matching is fully offloaded within the FPGA. Each layer is sequentially connected according to the data processing timing. The core design is as follows:

[0044] The first layer is a high-speed traffic filtering layer, consisting of a high-speed string matching engine. Deployed on the FPGA front end, this engine performs parallel, line-speed substring matching on all network traffic. It employs a multi-instance structure of 64 parallel Aho-Corasick or Hash-Based string engines, supporting variable-length pattern matching with a throughput of at least 100Gbps. Its core function is to quickly filter out the vast majority of benign traffic that does not trigger any regular expression rules, selecting only potentially suspicious traffic containing substrings with fixed characteristics. This reduces data processing volume for subsequent matching processes, laying the foundation for efficient matching.

[0045] The second layer is the regular expression data location layer, integrating two core components: the string-regular expression mapping table (str2NFA_map) and the full regular expression data storage. The string-regular expression mapping table is built on high-density storage units such as on-chip BlockRAM or UltraRAM of the FPGA, storing the mapping relationship between string IDs (str_ID) and regular expression NFA identifiers (NFA_ID). It supports single-cycle or low-latency random access, ensuring that the regular expression rules to be verified can be quickly located after traffic filtering. The full regular expression data storage utilizes on-chip storage resources in the FPGA with a much higher area density than LUTs / registers (such as URAM in Xilinx UltraScale+, with a storage density 1–2 orders of magnitude higher than LUTs), storing the general NFA structure data corresponding to all regular expressions in a compact serialized format. Each NFA is indexed by a unique NFA_ID, enabling fast data retrieval based on the identifier.

[0046] The third layer is the complete regular expression matching layer, with a reconfigurable general-purpose NFA matching engine handling the core functions. This engine has a limited number (e.g., 8–32) and employs a homogeneous NFA design, possessing a unified state representation format, transition table structure, and matching control logic. It can dynamically load the corresponding NFA data from the full regular expression data memory based on the NFA_ID output from the second layer and perform a complete regular expression matching process on the filtered traffic to be verified. This design eliminates the need to customize dedicated hardware logic for each rule, ensuring both the engine's versatility and reusability, while also supporting efficient matching of tens of thousands of regular expression rules through a dynamic loading mechanism, ultimately achieving closed-loop, fully unloaded matching within the FPGA.

[0047] It is understood that the system of this invention can achieve true full offloading, with all regular expression matching logic completed in a closed loop within the FPGA, without any CPU involvement in the verification process. This design completely eliminates the performance bottleneck caused by CPU intervention in traditional partially offloading architectures, providing a stable line-speed processing foundation for high-speed network scenarios of 100Gbps and above. Based on this, leveraging the traffic statistics characteristic of "extremely few regular expressions triggered per packet" (typically <1 in actual measurements, <3 in extreme scenarios), the traditional approach of customizing dedicated hardware for each rule is abandoned. Only a small number of general-purpose NFA engines are deployed, and flexible support for tens of thousands of rules can be achieved through a dynamic loading mechanism, greatly improving the utilization efficiency of hardware resources. Simultaneously, addressing the core requirement of rule storage, high-density on-chip storage resources (URAM / BRAM) are used on the FPGA to carry the rule body, rather than consuming expensive LUT resources, significantly increasing rule capacity. BRAM combines high speed and stability. This storage optimization method avoids consuming expensive logic resources and significantly improves rule storage capacity. The uniform format of the isomorphic NFA design provides the underlying technical support for the above advantages. Its standardized state representation and transition logic not only support the fast loading of arbitrary regular expressions, but also effectively improve the reusability of hardware templates, reduce the complexity of rule updates and system maintenance, and realize a matching system that is fully unloaded, high-capacity, and highly reusable.

[0048] In one embodiment, extracting fixed feature substrings from each regular rule in the regular rule set includes: extracting high-discrimination, non-wildcard literal strings from each regular rule as fixed feature substrings and marking them as trigger anchors; for regular rules from which valid fixed feature substrings cannot be extracted, using manual injection to approximate fixed feature substrings and marking them as trigger anchors; assigning a globally unique string identifier to each trigger anchor, with the string identifier corresponding one-to-one with the fixed feature substring.

[0049] In this embodiment, highly distinguishable fixed feature substrings can effectively filter out the vast majority of benign traffic, reducing the processing pressure of subsequent matching stages. The method of manually injecting approximate substrings ensures that regular expression rules without effective fixed substrings can also participate in the filtering process, avoiding rule omissions. The globally unique str_ID provides the basis for the accurate mapping of subsequent strings and NFAs.

[0050] In one embodiment, compiling each regular expression rule into a uniform isomorphic NFA includes: compiling each regular expression rule into a uniform isomorphic NFA, and assigning a unique NFA identifier to each compiled isomorphic NFA; the isomorphic NFA includes a character activation table, a state transition table, and a metadata header; the character activation table is stored in the form of a character-state set; the state transition table is stored in the form of a state-state set, and the transition of each state corresponds to multiple target states; the metadata header contains an initial state vector, a report state vector, and an associated rule identifier.

[0051] In this embodiment, the uniform format isomorphic NFA supports dynamic loading of arbitrary regular expressions, providing a prerequisite for the reuse of reconfigurable general NFA engines. The character activation table and state transition table, with the index key-state set as the core of the structured storage form, ensure the accurate execution of the regular expression matching logic and can accurately verify whether the traffic meets the wildcard, repetition and other logic in the regular expression rules. The character activation table is stored in the form of character-state set, that is, with the input character as the index key, each character corresponds to a set of NFA states that can be activated by it. The state transition table is stored in the form of state-state set, that is, with the current state ID of the NFA as the index key, each state corresponds to a set of target states that can be transitioned to by it. The initial state vector contained in the metadata header serves as the starting point for NFA to perform regular expression matching. It defines the set of states that are activated by default at the start of matching in vector form. The report state vector is the core basis for determining a successful match, marking the set of accepting states of the NFA in vector form. The associated rule identifier is the unique ID of the regular expression rule bound to this NFA, used to trigger subsequent security policies after the matching result is reported. The structured design of the metadata header facilitates the engine's rapid reading of NFA configuration information, significantly improving NFA loading efficiency. Furthermore, combined with serialized storage and the well-organized NFA_ID, the overall architecture can adapt to high-density on-chip storage resources on FPGAs (such as URAM / BRAM), stably supporting the storage of tens of thousands of regular expression rules, breaking through the resource limitations of traditional hardware solutions.

[0052] In one embodiment, constructing a mapping table between fixed feature substrings and corresponding isomorphic NFAs includes: establishing a correspondence between the string identifier of each fixed feature substring and the NFA identifier of the corresponding isomorphic NFA, and constructing a mapping table; the mapping table supports one-to-many mapping of one fixed feature substring to multiple isomorphic NFAs.

[0053] In this embodiment, the mapping table serves as the core link between the high-speed filtering and dynamic loading stages. It can quickly query the corresponding NFA_ID based on str_ID without traversing the entire NFA data, significantly reducing the positioning latency. At the same time, this mapping relationship naturally supports one-to-many scenarios where a fixed feature substring corresponds to multiple isomorphic NFAs, adapting to the actual need for multiple regular expression rules to share the same trigger anchor point, ensuring the comprehensiveness of rule association. Combined with the low-latency access characteristics of FPGA on-chip storage, this mapping table can achieve single-cycle or near-single-cycle queries, forming efficient collaboration with the high-speed string matching engine and dynamic loading mechanism, ensuring the continuity and high throughput performance of the entire pipeline processing.

[0054] In one embodiment, the construction of multiple string matching engines includes: integrating all fixed feature substrings to form a pattern set, and using an Aho-Corasick automaton or hash algorithm to construct a variable-length string matching engine; mapping the string matching engine to n parallel string matching engine instances, each instance processing different byte-aligned input substreams, covering offsets from 0 to n-1 bytes respectively.

[0055] In this embodiment, the Aho-Corasick algorithm has linear time complexity and multi-pattern string parallel matching capability, while the hash algorithm has the advantage of fast lookup. Both can adapt to the efficient matching requirements of variable-length feature substrings, ensuring a fast response to massive rules. The design of n parallel instances not only greatly improves the overall throughput of the engine through parallel computing, which can easily adapt to the line-speed processing requirements of high-speed network traffic of 100Gbps and above, but also completely avoids matching omissions caused by byte alignment issues by fully covering the sub-stream processing method with different byte offsets, laying an efficient screening foundation for subsequent NFA dynamic loading matching.

[0056] In one embodiment, storing the isomorphic NFA data corresponding to each regularity rule includes: using the on-chip Ultra RAM or Block RAM of the FPGA as the storage medium, storing each isomorphic NFA data in alignment with the NFA identifier, wherein the address offset of each NFA data block is linearly related to the NFA identifier.

[0057] In this embodiment, URAM features high capacity and high throughput, while BRAM boasts low latency. Both are high-density, high-efficiency on-chip storage resources for the FPGA, capable of stably supporting centralized storage of thousands to tens of thousands of homogeneous NFA data. This breaks through the storage resource limitations of traditional hardware solutions. By aligning storage and linear address mapping according to NFA identifiers, direct address calculation and fast indexing based on NFA identifiers can be achieved. Target NFA data can be located without a complex search process, significantly reducing the loading latency of the reconfigurable general-purpose NFA engine. At the same time, the on-chip storage deployment avoids the high latency and bandwidth bottlenecks of external storage, ensuring the low latency characteristics of closed-loop matching within the FPGA.

[0058] In one specific embodiment, such as Figure 2As shown, a schematic diagram of a regular expression compilation process is provided. This process, as the core preprocessing step in the system deployment stage of this application, takes the original regular rule set containing network security detection features as input, and outputs standardized components adapted for FPGA full offload matching through a multi-step serial and parallel processing mode. After the process starts, each regular expression rule is first processed in parallel using two paths: one path precisely extracts high-discrimination, non-wildcard literal strings from the rule as trigger anchors. For rules without effective fixed feature substrings, approximate anchors are generated through manual injection. At the same time, a globally unique string identifier (str_ID) is assigned to each anchor, and all anchor strings are merged to build a unified high-performance variable-length string matching engine (strEngine); the other path is based on the Thompson construct or derived optimization algorithm to compile each rule into a unified format isomorphic NFA containing a character activation table, a state transition table, and a metadata header. The character activation table stores character activation state information in the form of a character-state set, the state transition table stores transition logic in the form of a character-state set, and the metadata header clearly marks key information such as the matching start point and end point. After compilation, a unique NFA identifier (NFA_ID) is assigned to each isomorphic NFA. After dual-path processing is completed, the system constructs a mapping table (str2NFA_map) between str_ID and NFA_ID, and organizes the serialized data of all isomorphic NFAs according to NFA_ID and stores it in the full regular expression data store (FullRegEx Data Store). Finally, it outputs an FPGA deployment package containing strEngine, str2NFA_map, FullRegExDataStore and reconfigurable NFA engine hardware template, providing standardized data and hardware support for high-speed traffic offloading matching during the runtime phase.

[0059] Specifically, the regular expression compilation process is executed during the system deployment phase and includes the following steps:

[0060] Step 1: Extract string fragments;

[0061] For each regular expression, extract one or more highly distinctive, fixed, non-wildcard substrings as trigger tokens. The extraction principle is to select literal strings that appear infrequently, are of moderate length, and are semantically critical.

[0062] Rule 1 → Extract "union select" (ignore the middle \s+, take its typical fixed form);

[0063] Rule 2 → Extract " <script>" 和 "< / script> (Two anchor points improve recall rate)

[0064] Rule 3 → Unable to extract valid fixed strings (all are repeated wildcards), special handling: use manually injected anchors such as "AAAAAAAAAA" (10 A's) as approximate triggers, or let NFA handle them all (but ensure that their proportion in the rule set is extremely low).

[0065] Assign a unique string ID (str_ID):

[0066] str_ID=0 → "union select";

[0067] str_ID=1 → " <script>"；

[0068] str_ID=2 → "< / script> ";

[0069] str_ID=3 → "AAAAAAAAAA";

[0070] Step 2: Compile the regular expression into an isomorphic NFA;

[0071] Each regular expression is compiled into an isogeneous NFA using an improved Thompson construction algorithm. An isogeneous NFA requires:

[0072] All status numbers are consecutive (0 ~ N) 1);

[0073] The activation of each state is stored in the form of "character → state set";

[0074] The transition of each state is stored in the form of "state → set of states";

[0075] The acceptance status is explicitly marked;

[0076] The entire NFA is serialized in a compact structure (character activation table + state transition table + metadata header).

[0077] Assign a unique NFA_ID to each rule:

[0078] NFA_ID=0 → Rule 1 NFA;

[0079] NFA_ID=1 → NFA of Rule 2;

[0080] NFA_ID=2 → Rule 3 NFA;

[0081] Step 3: Construct a mapping table from strings to NFAs (str2NFA_map);

[0082] Establish a mapping from str_ID to {NFA_ID} (supports one-to-many relationships):

[0083] Table 1. Examples of string-to-NFA mapping representations

[0084]

[0085] The mapping table is stored in the FPGA's BRAM and supports single-cycle reading.

[0086] Step 4: Build a unified string matching engine (strEngine);

[0087] All extracted strings ("union select", " <script>", "< / script> The strings ", "AAAAAAAAAA" are merged into an Aho-Corasick automaton and mapped to 64 parallel string matching engine instances, each processing a different byte-aligned input substream to achieve 100Gbps line-speed matching.

[0088] Step 5: Store the NFA data to the full regularization data storage;

[0089] The serialized data of the three NFAs is written to the FPGA's URAM array (FullRegEx Data Store), stored aligned by NFA_ID. For example:

[0090] Data (512 bytes) at addresses 0x0000 ~ 0x01FF: NFA_ID=0;

[0091] Data (512 bytes) at addresses 0x0200 ~ 0x03FF: NFA_ID=1;

[0092] Data (512 bytes) at addresses 0x0400 ~ 0x05FF: NFA_ID=2;

[0093] Each NFA data block contains:

[0094] Header: Initial state vector, Report state vector;

[0095] Character activation table: a list of (character, target state) pairs per line, indexed by the input character;

[0096] State transition table: indexed by state ID, each column is a list of (starting state, target state) pairs; associated rule_ID: used for final reporting.

[0097] In one embodiment, performing full regular expression matching on the traffic to be verified includes: detecting currently idle general NFA engines; if an idle engine exists, reading the corresponding isomorphic NFA from the full regular expression data storage and loading it; if no engine exists, temporarily storing the traffic to be verified in a FIFO queue; sending the traffic to be verified to the general NFA engine that has loaded the isomorphic NFA through caching or forwarding; the general NFA engine starts from the initial state and simulates the state transition process of the isomorphic NFA character by character to verify whether the traffic to be verified meets the logic in the regular expression rules; if the matching reaches the accept state, the corresponding rule identifier is output; if the end of the traffic is reached but the accept state is not reached, the matching is determined to be a failure, and the traffic to be verified is output.

[0098] In this embodiment, during runtime, after the traffic to be verified is filtered by the string matching engine and the corresponding NFA_ID is obtained by querying str2NFA_map, the system first checks the currently idle general NFA engine. If an idle engine exists, the corresponding isomorphic NFA is directly read from the full regular expression data storage and loaded. If no idle engine exists, the traffic to be verified is temporarily stored in a FIFO queue to ensure that the data is not lost. After the engine is released, it is processed according to the first-in-first-out principle. Then, the traffic to be verified is sent to the general NFA engine that has loaded the isomorphic NFA through caching or forwarding. The engine simulates the state transition process of the isomorphic NFA character by character from the initial state to accurately verify whether the traffic to be verified meets the logic in the regular expression rule. If the acceptance state is reached during the matching process, the corresponding rule identifier (rule_ID) is output and reported to the host CPU through the PCIe bus to support the execution of subsequent security policies such as blocking and logging. If the acceptance state is not reached after traversing to the end of the traffic, it is determined that the matching has failed and the traffic is regarded as a false alarm and directly allowed. This design achieves ordered caching and balanced scheduling of traffic through a FIFO queue, avoiding processing bottlenecks caused by engine resource conflicts. Combined with the character-by-character precise state transition and dynamic allocation mechanism of the isomorphic NFA, it ensures the accuracy and completeness of regular expression matching while maximizing engine resource utilization. At the same time, the differential processing (reporting / allowing) of matching results is completed entirely in a closed loop within the FPGA, without the CPU participating in the intermediate matching process. This further enhances the high throughput and low latency advantages of the fully offloaded architecture, perfectly adapting to the deep packet inspection requirements in high-speed network environments.

[0099] In one specific embodiment, such as Figure 3The diagram illustrates the runtime matching pipeline. The input is a continuous network packet stream. The network packet stream is first sent to the string matching engine to perform high-speed filtering of the entire traffic: if no fixed feature substring is matched, it is directly determined as benign traffic and allowed to pass, without occupying subsequent hardware resources, ensuring low-latency forwarding of benign traffic; if the corresponding fixed feature substring is matched and str_ID is obtained, the associated NFA_ID is obtained by querying the str2NFA_map mapping table. At this time, the system will check whether there is an idle NFA engine. If there is no idle engine, the trigger packet is temporarily stored in the FIFO queue. After the engine resources are released, it will be scheduled and processed according to the first-in-first-out principle to avoid packet loss; if there is an idle engine, the corresponding data is read from the full NFA memory according to NFA_ID and dynamically loaded into the engine. Then, the trigger packet is sent to the configured NFA engine to perform full regular expression matching. After matching is complete, the process branches further: if the NFA match is successful, the corresponding rule_ID is output and reported to the host CPU via the PCIe bus, providing a basis for subsequent security policies (such as traffic blocking and logging); if the NFA match is unsuccessful, it is determined to be a false trigger of the feature substring, and the packet is directly allowed to pass. At this point, the matching pipeline for a single packet is complete. This link, through branching and queuing resource scheduling and closed-loop matching processing, achieves accurate detection under high-speed traffic while ensuring efficient resource utilization and orderly packet processing.

[0100] Specifically, the regular expression matching process includes the following steps: The input message payload is: GET / search?q=union%20select%20user,password%20from%20users HTTP / 1.1. After URL decoding, this message contains the string "union select".

[0101] Step 1: Full-flow string matching: The message enters the string matching engine (strEngine). The engine detects the substring "union select" in the pipeline and outputs str_ID = 0.

[0102] Step 2: Query the mapping table: The system sends str_ID=0 into str2NFA_map and finds the corresponding NFA_ID =0.

[0103] Step 3: Dynamically Load NFA: The control logic checks if there are any idle units among the 16 reconfigurable NFA engines (assuming engine #5 is idle). Then, it reads the complete NFA data of NFA_ID=0 from address 0x0000 of the FullRegEx Data Store and loads it into the configuration register and state transition memory of engine #5.

[0104] Step 4: Perform NFA matching: The original message (or its payload portion) is fed into the configured engine #5. The engine starts from the initial state and simulates NFA character by character:

[0105] Matches "union select user...";

[0106] Verify whether wildcard logic such as \s+ and \w+ is satisfied;

[0107] Finally, the state of acceptance is reached;

[0108] Match successful, engine output rule_ID = 1.

[0109] Step 5: Result Reporting: rule_ID=1 is reported to the host CPU via the PCIe interface, triggering IPS policies (such as blocking connections or logging). The entire process does not require CPU participation in regular expression calculation.

[0110] In one specific embodiment, such as Figure 4 The diagram illustrates the internal structure of a reconfigurable NFA engine, which primarily implements state transition matching and dynamic rule adaptation for symbol sequences. Input characters are first converted into corresponding encoded signals by a decoder and then fed into a character activation module containing 32 states (S0-S31, where rows and columns correspond to the matching relationships between states and symbols). This generates an active state vector for the current character. Simultaneously, the state transition module (which stores the transition target table for each state) combines the initial state vector with the state transition logic to calculate the next round's transitional state vector. The active and transitional vectors are then ANDed to obtain the active state vector for the next cycle. When the active state vector matches the reported state vector, the engine outputs a hit signal to complete the matching feedback. The reconfiguration module on the right dynamically updates the matching table of the character activation module and the transition table of the state transition module. This allows the engine to adapt to isomorphic NFAs corresponding to different regularization rules without modifying the hardware architecture, achieving a combination of parallel state processing and dynamic reconfigurability. This enables flexible and efficient regularization matching in high-traffic scenarios.

[0111] In one embodiment, the matching result includes a rule identifier, a matching start position, and length information, and is output to an external processing unit via the PCIe bus.

[0112] In one specific embodiment, such as Figure 5As shown, a schematic diagram of an FPGA-based fully offloaded regular expression matching system is provided. The CPU provides a set of regular expression rules. After the compiler completes string extraction, strEngine construction, NFA automaton generation, and str2NFA mapping table construction, the relevant resources are distributed to the FPGA. After receiving the resources, the FPGA first enters 64 parallel string matching engines (StrEngine 1~64) for high-speed filtering. If a match is found, strID is output. The corresponding NFA_ID is obtained through Str2NFA_map mapping. Then, the NFA is dynamically loaded from the full regular expression data storage to the reconfigurable general NFA. The filtered verification traffic is fully matched and the result is output. The entire matching process is completed in a closed loop inside the FPGA. The CPU only participates in the initial rule compilation and resource distribution, realizing the full offload of regular expression matching.

[0113] In one embodiment, a full offload regular expression matching method based on FPGA is provided, comprising the following steps:

[0114] Step 10: Receive the set of regular rules and extract the fixed feature substrings of each regular rule in the set of regular rules;

[0115] Step 20: Compile each regular expression rule into a uniform format isomorphic NFA and construct a mapping table between fixed feature substrings and corresponding isomorphic NFAs;

[0116] Step 30: Store the mapping table to the string-NFA mapping table module of the FPGA, and store each isomorphic NFA to the full regular data memory of the FPGA.

[0117] Step 40: The input network traffic is scanned in parallel by the parallel string matching engine array of the FPGA to filter out the traffic to be verified containing fixed feature substrings and output the identification information of the corresponding fixed feature substrings.

[0118] Step 50: Obtain the corresponding isomorphic NFA identifier by querying the identifier information through the string-NFA mapping table module;

[0119] Step 60: The reconfigurable general-purpose NFA engine module of the FPGA dynamically loads the corresponding isomorphic NFA from the full regular expression data memory according to the isomorphic NFA identifier, and performs full regular expression matching on the traffic to be verified.

[0120] Step 70: The matching result is output to the external processing unit through the matching result output module of the FPGA to realize the full unloading of regular expressions.

[0121] For specific limitations on the FPGA-based full offload regular expression matching method, please refer to the limitations on the FPGA-based full offload regular expression matching system mentioned above, which will not be repeated here.

[0122] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A fully offloaded regular expression matching system based on FPGA, characterized in that, The system includes a regular expression compilation unit and an FPGA full matching unit; The regular expression rule compilation unit is used to receive a set of regular expression rules, extract the fixed feature substrings of each regular expression rule in the set of regular expression rules, compile each regular expression rule into a uniform format isomorphic NFA, and construct a mapping table between the fixed feature substrings and the corresponding isomorphic NFAs. The FPGA full matching unit is communicatively connected to the regular expression compilation unit, and the FPGA full matching unit includes: A parallel high-speed string matching engine array is configured with multiple string matching engines to perform parallel scanning of input network traffic, filter out the traffic to be verified that contains the fixed feature substring, and output the identification information of the corresponding fixed feature substring. The string-NFA mapping table module stores the mapping relationship table and is used to query the corresponding isomorphic NFA identifier based on the identifier information. A full regularization data storage unit is used to store the isomorphic NFAs corresponding to each of the regularization rules; A reconfigurable general NFA engine module is configured with a general NFA engine, which is used to dynamically load the corresponding NFA from the full regular expression data storage according to the homogeneous NFA identifier, and perform full regular expression matching on the traffic to be verified. The matching result output module is used to output the matching results of the reconfigurable general NFA engine module to an external processing unit, thereby achieving full unloading of regular expressions.

2. The system of claim 1, wherein, The extraction of fixed feature substrings from each regular rule in the regular rule set includes: Extract high-discrimination, non-wildcard literal strings from each regular expression rule as fixed feature substrings and mark them as trigger anchor points; For regularization rules that cannot extract valid fixed feature substrings, a manual injection method is used to approximate them as fixed feature substrings, which are then marked as trigger anchor points. Each trigger anchor point is assigned a globally unique string identifier, which corresponds one-to-one with a fixed feature substring.

3. The system of claim 1, wherein, Compiling the aforementioned regular expression rules into a uniform format isomorphic NFA includes: Each regular expression rule is compiled into a uniformly formatted isomorphic NFA, and a unique NFA identifier is assigned to each compiled isomorphic NFA. The isomorphic NFA includes a character activation table, a state transition table, and a metadata header. The character activation table is stored in the form of a character-state set. The state transition table is stored in the form of a state-state set, and the transition of each state corresponds to multiple target states. The metadata header contains an initial state vector, a report state vector, and an associated rule identifier.

4. The system according to claim 1, characterized in that, The construction of the mapping table between the fixed feature substring and the corresponding isomorphic NFA includes: Establish a correspondence between the string identifier of each fixed feature substring and the NFA identifier of the corresponding isomorphic NFA, and construct a mapping table.

5. The system according to claim 1, characterized in that, The construction methods of the multiple string matching engines include: Integrate all fixed feature substrings to form a pattern set, and use Aho-Corasick automata or hash algorithm to build a variable-length string matching engine; The string matching engine is mapped to n parallel string matching engine instances, each instance processing a different byte-aligned input substream, covering offsets from 0 to n-1 bytes respectively.

6. The system according to claim 1, characterized in that, The storage of isomorphic NFA data corresponding to each of the regularization rules includes: The FPGA uses on-chip Ultra RAM or Block RAM as the storage medium, and stores each isomorphic NFA data according to the NFA identifier. The address offset of each NFA data block is linearly related to the NFA identifier.

7. The system according to claim 1, characterized in that, Performing a full regular expression match on the traffic to be verified includes: Detect currently idle general NFA engines. If an idle engine exists, read the corresponding isomorphic NFA from the full regular expression data storage and load it. If no engine exists, temporarily store the traffic to be verified in the FIFO queue. The traffic to be verified is sent to a generic NFA engine that has loaded isomorphic NFA by caching or forwarding. The generic NFA engine starts from the initial state and simulates the state transition process of isomorphic NFA character by character to verify whether the traffic to be verified meets the logic in the regular expression rules. If a match is found to the accept state, the corresponding rule identifier is output. If the traffic is traversed to the end without reaching the accept state, the match is deemed to have failed, and the traffic to be verified is output.

8. The system according to claim 1, characterized in that, The mapping table supports a one-to-many mapping where a fixed feature substring corresponds to multiple isomorphic NFAs.

9. The system according to claim 1, characterized in that, The matching result includes rule identifier, matching start position and length information, and is output to an external processing unit via the PCIe bus.

10. A full offload regular expression matching method based on FPGA, characterized in that, The method includes: Receive a set of regular rules and extract fixed feature substrings from each regular rule in the set of regular rules; Each of the regular expression rules is compiled into a uniform format isomorphic NFA, and a mapping table between the fixed feature substrings and the corresponding isomorphic NFAs is constructed. The mapping table is stored in the string-NFA mapping table module of the FPGA, and each of the isomorphic NFAs is stored in the full regular data memory of the FPGA. The parallel string matching engine array of the FPGA performs parallel scanning of the input network traffic, filters out the traffic to be verified that contains the fixed feature substring, and outputs the identification information of the corresponding fixed feature substring. The corresponding isomorphic NFA identifier is obtained by querying the identifier information through the string-NFA mapping table module. The FPGA's reconfigurable general-purpose NFA engine module dynamically loads the corresponding isomorphic NFA from the full regular expression data memory based on the isomorphic NFA identifier, and performs full regular expression matching on the traffic to be verified. The matching result output module of the FPGA outputs the matching result to the external processing unit, realizing the full unloading of regular expressions.

Images