Parallel regular expression matchers

Abstract

The invention provides a parallel regular expression matcher, comprising: a software pre-compilation unit, configured to pre-compile multiple regular expression rules in a regular expression rule set, and convert them into regular expression rules in a format that conforms to hardware processing behavior, so as to be matched by a hardware matching circuit; and a hardware matching circuit, configured to match a message to be matched with the regular expression rules in the format that conforms to the hardware processing behavior, and obtain a matching result.

Description

Technical Field

[0001] The present invention relates to the technical field of regular expressions, and in particular to a parallel regular expression matcher. Background Art

[0002] A regular expression is a text pattern consisting of common characters (letters / numbers / symbols) and special characters (also called "metacharacters"), which is used to describe one or more strings to be matched when searching for text. It is widely used in various fields of data processing. In high-speed, high-bandwidth network and database applications, the processing speed of regular expressions is extremely high. This requirement mainly includes two aspects: (1) the processing bandwidth of the data to be matched; (2) the number of regular expressions matched in parallel.

[0003] The current products are mainly implemented by software. Well-known regular expression matching software (libraries) include: POSIX REAPI, PCRE, Google RE2, Oniguruma, Hyperscan, etc. The core of the implementation method is the finite state automaton, which can be divided into deterministic finite automaton (DFA), non-deterministic finite automaton (NFA) and their combination. For example Figure 1 The figure shows a finite automaton constructed for the regular expression a(b|c)*. Each circle represents a state, and the double-line circle represents the end (match) state. When the characters of the text are input into the automaton one by one, if the current character is the same as the character on the arrow, you can press the arrow to advance to the next state. If the end state can be reached, it indicates that a match has occurred. If the end state cannot be reached after all the characters of the text are scanned, it means that there is no match.

[0004] The regular expression matching feature of software is flexible use, but the disadvantage is slow speed and limited parallel processing capability. The above software (library) uses some methods such as state diagram merging of multiple regular expressions, multithreading, phased matching, and the use of single instruction multiple data (SIMD) instructions to improve the above shortcomings, but it still cannot meet the growing demand for data processing speed. For example, Hyperscan, which is currently the fastest and widely used by mainstream data center manufacturers, can achieve a single-threaded matching speed of 7Gbit / second for the Snort regular expression set. However, multithreading cannot speed up linearly, and the difference in the text to be scanned sometimes significantly affects the speed. Other commonly used regular expression matching software (libraries) often have a matching speed of only tens or hundreds of Mbit / second. In addition, if you want to embed the processing of regular expressions in network processing equipment or digital signal processing equipment, it is difficult for software algorithms to do so (the above devices often do not have CPUs or only have low-performance microcontrollers).

[0005] The existing digital circuit implementation of regular expressions is generally a simple DFA (deterministic finite state automaton) or NFA (non-deterministic finite state automaton) circuit, which can achieve a good processing speed (for example, several Gbit / s to tens of Gbit / s) for a single regular expression, but when the number of regular expressions increases, the speed drops rapidly in inverse proportion. When the number of regular expressions is on the order of 100, the speed is not as good as software and lacks practical application value.

[0006] The problems with current products / methods are that software regular expressions are flexible to use, but slow and cannot keep up with the data input speed of current high-speed networks (25Gbps, 100Gbps) or high-speed interfaces (PCIe Gen3x16). Existing digital circuit implementations cannot efficiently process multiple regular expressions at the same time. Summary of the invention

[0007] The object of the present invention is to provide a parallel regular expression matcher to solve the problem that the existing regular expressions are slow or cannot be processed in parallel.

[0008] In order to solve the above technical problems, the present invention provides a parallel regular expression matcher, comprising:

[0009] A software pre-compilation unit is configured to pre-compile multiple regular expression rules in the regular expression rule set and convert them into regular expression rules in a format that conforms to the hardware processing behavior, so as to be matched by the hardware matching circuit;

[0010] The hardware matching circuit is configured to match the message to be matched with the regular expression rule in the format that conforms to the hardware processing behavior to obtain a matching result.

[0011] Optionally, in the parallel regular expression matcher, the software pre-compilation unit includes a splitting module, a state merging module and a configuration table module, wherein:

[0012] The splitting module is used to split multiple regular expression rules into pure strings and regular units;

[0013] The state merging module is used to merge the pure character strings and regular units in state, extract the same pure character strings and regular units, and distinguish different pure character strings and regular units;

[0014] The configuration table module encapsulates pure character strings and regular units into a regular expression pattern library in a format that conforms to hardware processing behavior, so as to be used for parameter configuration of the hardware matching circuit and initialization of the state table.

[0015] Optionally, in the parallel regular expression matcher, the regular expression pattern library includes a pure string group, a regular unit group and a metadata group, wherein:

[0016] The pure character string group is used to store pure character strings, and configure the relationship between pure character strings according to whether the pure character strings are the same;

[0017] The regular unit group is used to store regular units and configure the relationship between regular units according to whether the regular units are the same;

[0018] The metadata group is used to store metadata and configure the relationship between the pure string and the regular unit according to the metadata.

[0019] Optionally, in the parallel regular expression matcher, the hardware matching circuit includes a pattern input module, a data input module, a regular matching core module and a matching result output module, wherein:

[0020] The pattern input module performs parameter configuration on the regular matching core module according to the pure character string in the pure character string group and the regular unit in the regular unit group, and initializes the state table of the regular matching core module according to the metadata in the metadata group;

[0021] The data input module is used to provide the message to be matched to the regular matching core module;

[0022] The regular expression matching core module is used to match the message to be matched with the regular expression rule to generate a matching result;

[0023] The matching result output module is used to output the matching result;

[0024] Among them, the pattern input module, the data input module and the matching result output module perform bus protocol bridging or format conversion with the regular matching core module.

[0025] Optionally, in the parallel regular expression matcher, the regular matching core module includes a string preprocessing module, an operation scheduling module and an execution engine module, wherein:

[0026] The string preprocessing module is used to scan the to-be-matched message according to the pure strings in the pure string group to generate a string queue;

[0027] The operation scheduling module is used to execute corresponding programs according to the string queue to generate a request queue;

[0028] The execution engine module scans the to-be-matched message according to the request queue, and determines whether the to-be-matched message matches.

[0029] Optionally, in the parallel regular expression matcher, the string preprocessing module includes a text scanning module and a text confirmation module, wherein:

[0030] The text scanning module uses a Shift-OR algorithm circuit to pre-match the pure character strings in the message to be matched according to the pure character strings in the pure character string group to obtain a rough screening result;

[0031] The text confirmation module uses a hardware hash algorithm to reconfirm the pure character strings in the coarse screening result according to the pure character strings in the pure character string group to obtain a fine screening result.

[0032] Optionally, in the parallel regular expression matcher, the Shift-OR algorithm circuit performs pipeline scanning on 16 characters simultaneously;

[0033] After being scanned by the Shift-OR algorithm circuit, the byte sequence number to be confirmed of the matched data packet is put into the first-in-first-out queue of the index data structure of the text confirmation module for use by the text confirmation module;

[0034] The text confirmation module reads the first-in-first-out queue of the index data structure, forms and queries the fine screening module index table, obtains the byte sequence number of the matching data packet, calculates the hash value, uses the hash value to search the string information table, and obtains the string length and string address;

[0035] Generate a string queue from string length and string address.

[0036] Optionally, in the parallel regular expression matcher, the operation scheduler is responsible for forming a corresponding program based on the string request information and selecting a corresponding execution engine module. Each program includes instructions, operation codes and operands, and each instruction is sent to a request queue for subsequent execution engine module calls.

[0037] Optionally, in the parallel regular expression matcher, the execution engine module includes a character comparison engine, an NFA engine, a reporting engine and a message buffer;

[0038] The request queue includes string comparison requests, regular NFA requests and result report requests;

[0039] The character comparison engine performs accurate comparison on the fine screening results according to the string comparison request, including final one-by-one comparison of ultra-long pure string matches;

[0040] The NFA engine performs corresponding NFA state machine operations on the message to be matched according to the regular NFA request, including selecting corresponding NFA in the corresponding field of the message to be matched according to the input request information, performing NFA state machine operations, and finally generating a matching result for use by a subsequent report engine;

[0041] The report engine outputs the final matching result according to the result report request, including calculating the matching position according to the output of the character comparison engine and the NFA engine, extracting the matched sequence number, obtaining the matching result and outputting it;

[0042] The message buffer is used to store and provide to-be-matched message data required by various execution engines.

[0043] Optionally, in the parallel regular expression matcher, the number ratio and operation frequency of the character comparison engine and the NFA engine are weighed according to the software performance evaluation and the physical characteristics of the hardware circuit;

[0044] Select regular expression rule sets and messages to be matched for typical applications to perform performance estimation, determine the size of the string preprocessing module under a certain matching probability, and determine the number and appropriate ratio of character comparison engines and NFA engines;

[0045] According to the physical characteristics of the hardware implementation, the parameter configuration of each module is further determined.

[0046] In the parallel regular expression matcher provided by the present invention, multiple regular expression rules in a regular expression rule set are precompiled by a software precompilation unit and converted into regular expression rules in a format that conforms to hardware processing behavior, so as to be matched by a hardware matching circuit, so that the hardware matching circuit can match the message to be matched with the regular expression rules in the format that conforms to the hardware processing behavior to obtain a matching result, thereby realizing a new type of parallel regular expression matching digital circuit, and being able to support parallel matching of multiple regular expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic diagram of constructing a finite automaton using the existing regular expression a(b|c)*;

[0048] Figure 2 is a schematic diagram of a parallel regular expression matcher in one embodiment of the present invention;

[0049] Figure 3 is a schematic diagram of a regular expression matching core in a parallel regular expression matcher hardware circuit in an embodiment of the present invention;

[0050] Figure 4 is a schematic diagram of a text scanning module of a regular expression matching core in an embodiment of the present invention;

[0051] Figure 5 It is a schematic diagram of a text confirmation module of a regular expression matching core in one embodiment of the present invention;

[0052] Figure 6 is a schematic diagram of an operation scheduler of a regular expression matching core in an embodiment of the present invention;

[0053] Figure 7 is a schematic diagram of a string comparison engine of a regular expression matching core in one embodiment of the present invention;

[0054] Figure 8 Schematic diagram of the NFA engine of the regular expression matching core in one embodiment of the present invention. DETAILED DESCRIPTION

[0055] In the present invention, each embodiment is only intended to illustrate the aspects of the present invention and should not be construed as limiting.

[0056] It should also be noted that in the embodiments of the present invention, for the sake of clarity and simplicity, only a portion of the parts or components may be shown, but those of ordinary skill in the art will understand that under the teachings of the present invention, the required parts or components may be added according to the needs of the specific scenario. In addition, unless otherwise specified, the features in different embodiments of the present invention may be combined with each other. For example, a feature in the second embodiment may be used to replace a corresponding or functionally identical or similar feature in the first embodiment, and the resulting embodiment also falls within the disclosure scope or recorded scope of the present application.

[0057] It should also be noted that within the scope of the present invention, the terms "same", "equal", "equal to" and the like do not mean that the values ​​of the two are absolutely equal, but allow a certain reasonable error, that is, the terms also cover "substantially the same", "substantially equal", "substantially equal to". By analogy, in the present invention, the terms "perpendicular to", "parallel to" and the like indicating directions also cover the meanings of "substantially perpendicular to" and "substantially parallel to".

[0058] In addition, the numbering of the steps of the methods of the present invention does not limit the execution order of the method steps. Unless otherwise specified, the method steps can be executed in different orders.

[0059] The parallel regular expression matcher proposed in the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become clearer according to the following description and claims. It should be noted that the drawings are all in a very simplified form and are not in precise proportions, and are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0060] In addition, unless otherwise specified, the features in different embodiments of the present invention may be combined with each other. For example, a feature in the second embodiment may be used to replace a corresponding feature in the first embodiment or a feature with the same or similar function, and the resulting embodiment also falls within the disclosure or description scope of the present application.

[0061] The core idea of ​​the present invention is to provide a parallel regular expression matcher to solve the problem that the existing regular expressions are slow or cannot be processed in parallel.

[0062] To realize the above idea, the present invention provides a parallel regular expression matcher, including: a software pre-compilation unit, configured to pre-compile multiple regular expression rules in a regular expression rule set, and convert them into regular expression rules in a format that conforms to hardware processing behavior, so as to be matched by a hardware matching circuit; a hardware matching circuit, configured to match a message to be matched with the regular expression rules in a format that conforms to hardware processing behavior, and obtain a matching result.

[0063] The present invention proposes a novel parallel regular expression matching digital circuit, which can support parallel matching of multiple regular expressions (for example, 1000 regular expressions) and improve the matching throughput bandwidth (greater than 50 Gbit / s).

[0064] The present invention proposes a novel digital circuit implementation method and device for regular expressions, which can efficiently process a large number of parallel regular expressions. Figure 2 As shown:

[0065] Split the regular expression rule into a pure string (literal string) and a regular unit (regex), and perform matching scans separately. For example, the regular expression is hello(e|f)*, which can be split into a pure string hello and a regular unit (e|f)*. If the substring hello of the regular expression fails to match, then the entire regular expression will definitely not match. Furthermore, substrings can also be divided into prefixes (prefix) and suffixes (suffix).

[0066] During the software compilation process, multiple regular expressions are merged. In actual applications, multiple regular expressions have a certain degree of similarity: they contain the same substring or the same regular finite automaton structure. Distinguishing and extracting the parts that can be "merged" is the key to improving the parallel processing capability of regular expressions.

[0067] Multi-level scanning method. In order to achieve the target processing speed with a smaller hardware circuit scale, it is divided into three levels of scanning: the first level pre-processes the pure string, which can determine whether the text (message) to be scanned and the substring "definitely does not match" or "may match" (the "definitely does not match" message will be directly skipped later), using a low-cost and high-efficiency hardware Shift-OR algorithm, called "coarse screening"; the second level is to confirm the "possibly matching" text (Confirm), which uses a hardware hash algorithm to further confirm the matching of the pure string and determine the "definitely does not match" and "highly likely to match" messages, called "fine screening"; the third level is a processor-like architecture. Its working principle is: the "operation and request co-processor" performs operation scheduling on the pre-processing results and selects the corresponding program (Program) for execution. The program (Program) execution unit contains three engines: string matching engine, regular NFA engine and result reporting engine. The number of string matching engines and regular NFA engines can be configured. Depending on the preprocessing results, the operation scheduler is responsible for generating the corresponding request sequence and may call one, two or three of the execution units.

[0068] An embodiment of the present invention provides a parallel regular expression matcher, comprising: a software pre-compilation unit, configured to pre-compile multiple regular expression rules in a regular expression rule set, and convert them into regular expression rules in a format that conforms to hardware processing behavior, so as to be matched by a hardware matching circuit; and a hardware matching circuit, configured to match a message to be matched with the regular expression rules in a format that conforms to the hardware processing behavior, and obtain a matching result.

[0069] In one embodiment of the present invention, in the parallel regular expression matcher, the software pre-compilation unit includes a splitting module, a state merging module and a configuration table module, wherein: the splitting module is used to split multiple regular expression rules into pure character strings and regular units; the state merging module is used to merge the pure character strings and regular units in state, extract the same pure character strings and regular units, and distinguish different pure character strings and regular units; the configuration table module encapsulates the pure character strings and regular units into a regular expression pattern library in a format that conforms to the hardware processing behavior, so as to be used for parameter configuration and state table initialization of the hardware matching circuit. Specifically, a software method is used to merge the states of multiple regular expressions (pre-compilation), and in the pre-compilation process, the regular expressions are split into pure character string groups and regular unit groups. Then, the regular expression pattern library (pattern database) is encapsulated in a format that conforms to the hardware processing behavior, so as to be used for parameter configuration and state table initialization of the regular expression matching core in the hardware circuit.

[0070] In one embodiment of the present invention, in the parallel regular expression matcher, the regular expression pattern library includes a pure string group, a regular unit group and a metadata group, wherein: the pure string group is used to store pure strings, and configure the relationship between pure strings according to whether each pure string is the same; the regular unit group is used to store regular units, and configure the relationship between regular units according to whether each regular unit is the same; the metadata group is used to store metadata, and configure the relationship between pure strings and regular units according to the metadata.

[0071] In one embodiment of the present invention, in the parallel regular expression matcher, Figure 3 As shown, the hardware matching circuit includes a pattern input module, a data input module, a regular matching core module and a matching result output module, wherein: the pattern input module performs parameter configuration on the regular matching core module according to the pure character string in the pure character string group and the regular unit in the regular unit group, and initializes the state table of the regular matching core module according to the metadata in the metadata group; the data input module is used to provide the message to be matched to the regular matching core module; the regular matching core module is used to match the message to be matched with the regular expression rule to generate a matching result; the matching result output module is used to output the matching result; wherein the pattern input module, the data input module and the matching result output module perform bus protocol bridging or format conversion with the regular matching core module. That is, the hardware circuit is composed of a pattern input module, a data input module, a matching result output module and a regular matching core. wherein the pattern input module, the data module module and the matching result output module perform bus protocol bridging and format conversion.

[0072] In one embodiment of the present invention, in the parallel regular expression matcher, Figure 3 As shown, the regular matching core module includes a string preprocessing module, an operation scheduling module and an execution engine module, wherein: the string preprocessing module is used to scan the to-be-matched message according to the pure strings in the pure string group to generate a string queue; the operation scheduling module is used to execute the corresponding program according to the string queue to generate a request queue; the execution engine module scans the to-be-matched message according to the request queue to determine whether the to-be-matched message matches.

[0073] In one embodiment of the present invention, in the parallel regular expression matcher, the string preprocessing module includes a text scanning module and a text confirmation module, wherein: the text scanning module uses a Shift-OR algorithm circuit to pre-match the pure string in the message to be matched according to the pure string in the pure string group to obtain a coarse screening result; the text confirmation module uses a hardware hash algorithm to re-confirm the pure string in the coarse screening result according to the pure string in the pure string group to obtain a fine screening result.

[0074] In one embodiment of the present invention, in the parallel regular expression matcher, the execution engine module includes a character comparison engine, an NFA engine, a report engine and a message buffer; the request queue includes a string comparison request, a regular NFA request and a result report request; the character comparison engine performs an accurate comparison of the fine screening results according to the string comparison request; the NFA engine performs corresponding NFA state machine operations on the to-be-matched message according to the regular NFA request; the report engine outputs the final matching result according to the result report request; and the message buffer is used to store and provide the to-be-matched message data required by various execution engines.

[0075] In one embodiment of the present invention, in the parallel regular expression matcher, the Shift-OR algorithm circuit performs pipeline scanning on 16 characters at the same time; after being scanned by the Shift-OR algorithm circuit, the byte sequence number of the matched data packet is placed in the first-in-first-out queue of the index data structure of the text confirmation module for use by the text confirmation module; the text confirmation module reads the first-in-first-out queue of the index data structure, forms and queries the fine screening module index table, and then calculates the hash value to obtain the byte sequence number of the matched data packet, uses the hash value to search the string information table, and obtains the string length and string address; generates a string queue based on the string length and string address; the operation scheduler is responsible for forming a corresponding program according to the string request information and selecting a corresponding execution engine module, each program includes instructions, operation codes and operands, and each instruction is sent to the request queue for subsequent execution engine module calls.

[0076] Specifically, Figure 4 As shown, in the string preprocessing step, the text scan uses a parallel Shift-OR circuit. In this example, it scans 16 characters at the same time. This scanning process is pipelined, so 16 characters (bytes) can be processed in each clock cycle. After scanning, the byte sequence number of the matched message will be placed in the byte sequence number FIFO to be confirmed for use by the subsequent text confirmation module. At this time, there may still be "false matches" in the match, which requires further confirmation. Figure 5As shown in the figure, the text confirmation module reads the byte sequence number generated by the previous level scan, queries the confirmation message index table, calculates the hash value, and finally uses the hash value to search the string information table to obtain the string length, address and other information. This information will eventually generate a string request, including the matching address, length and grouping confirmed by the fine screening, for use by subsequent modules. Figure 6 As shown in the figure, the operation scheduler is responsible for forming a program based on the string request information and selecting the corresponding execution unit. Each program consists of a series of instructions, each of which contains an opcode and an operand. Each instruction will be sent to the request queue for subsequent execution engine calls.

[0077] Program example:

[0078]

[0079]

[0080] Among them, min_offset, fail_jump, etc. are operation codes, followed by operands. Program StateMachine puts them into Instruction Buffer and organizes them into parallel request queues.

[0081] In one embodiment of the present invention, in the parallel regular expression matcher, the string comparison engine performs a final one-by-one comparison of the ultra-long pure string to obtain an accurate conclusion on whether it matches (because whether it is a coarse screening or a fine screening, efficiency is pursued and a certain probability of misjudgment is allowed, so each byte needs to be confirmed word by word in the end); the NFA engine selects the corresponding NFA in the corresponding field of the message to be matched according to the input request information, performs NFA state machine operations, and finally generates a matching result for use by the subsequent reporting engine; the reporting engine corrects the matching position according to the output of the string comparison engine and the NFA engine, and extracts the serial number of the matched rule to form a matching result and outputs it.

[0082] Specifically, Figure 7 As shown, in the program execution unit, the string comparison engine is responsible for the final one-by-one comparison of the overlong string (greater than 8 bytes). Figure 8 As shown in the figure, the regular NFA engine is responsible for implementing the NFA state machine. The engine will select the corresponding NFA in the corresponding field of the message according to the input request information and perform the NFA state machine operation. The state machine will eventually generate a matching result for use by the subsequent report engine. The result report engine calculates the matching position (offset) based on the output of the first two matching engines, extracts the matched rule number (ID), forms the matching result and outputs it.

[0083] In one embodiment of the present invention, in the parallel regular expression matcher, the scale of the string preprocessing module, the number ratio of the string comparison engine and the NFA engine, and the operating frequency are weighed according to the software performance evaluation and the physical characteristics of the hardware circuit; the regular expression rule set of typical applications and the message to be matched are selected for performance estimation, and the scale of the string preprocessing module is determined under a certain matching probability, and the number and appropriate ratio of the character comparison engine and the NFA engine are determined; according to the physical characteristics of the hardware implementation method, the parameter configuration of each module is further determined.

[0084] Specifically, the size of the string preprocessing module, the number ratio of the string comparison engine and the regular NFA engine, and the operating frequency need to be weighed according to the software performance evaluation and the physical characteristics of the hardware circuit. By selecting a set of regular expressions for typical applications and the text (message) to be scanned for performance estimation, the size of the string preprocessing module under a certain matching probability can be determined, and the number and appropriate ratio of the character comparison engine and the NFA engine can be determined. Then, according to the physical characteristics of the hardware implementation, for example, for FPGA, considering the limitations of circuit logic resources and the synthesizable frequency, and for ASIC chips, considering the characteristics of the tape-out process library and the integration of input and output interfaces, the parameter configuration (bus width, operating frequency, storage unit size, etc.) of each module can be further determined. For different goals, the technical solution described in the present invention will be presented as different parameter configurations.

[0085] In the regular expression matcher and matching method provided by the present invention, multiple regular expression rules in a regular expression rule set are precompiled by a software precompilation unit and converted into regular expression rules in a format that conforms to hardware processing behavior, so as to be matched by a hardware matching circuit, so that the hardware matching circuit can match the message to be matched with the regular expression rules in the format that conforms to the hardware processing behavior to obtain a matching result, thereby realizing a new type of parallel regular expression matching digital circuit, and being able to support parallel matching of multiple regular expressions.

[0086] The regular expression matching core consists of three parts: a) string preprocessing module; b) operation scheduling module; c) execution engine module. These three parts process the input message in a streaming manner and output the matching results according to the pre-set rules.

[0087] During the processing, the input message first passes through the string preprocessing module to determine whether the input message matches the string in the rule. If it does not match, it means that the current message will not match the rule, so it does not need to be sent to the subsequent module for processing; if it matches, subsequent processing is required. This module consists of two sub-modules, among which the text scan (Literal Scan) is responsible for the first step of coarse screening to determine the possible matching position of the string; the text confirmation (LiteralConfirm) module is responsible for calculating the hash value of the possible matching position and comparing it to complete further fine screening.

[0088] After the string preprocessing is confirmed, a series of possible matching location information and encoding information of the associated rules are obtained. They will be sent to the string queue. Then, the program scheduler will select the corresponding program to execute according to the information in the string queue. During the program execution, corresponding requests will be generated according to the rules. These requests include string comparison, regular NFA, result report, etc. All requests will be sent to the request queue, and once the subsequent execution unit is ready, the execution process will be entered.

[0089] The execution engine usually follows three steps: precise character-by-character comparison of strings, regular NFA, and result reporting. The character-by-character comparison of strings is responsible for confirming the matching results of super-long strings (more than 8 bytes), NFA is responsible for performing corresponding NFA state machine operations, and the reporting engine outputs the final matching results. During the operation, a dedicated message cache is responsible for providing the message data stream required by the engine.

[0090] The key points of the present invention are:

[0091] A regular expression matcher implemented by a hardware circuit, where:

[0092] Multi-level hardware scanning circuit for parallel matching of regular expressions: the first level is the fast text scanning of the Shift-OR algorithm; the second level is the text matching confirmation process using the Hash algorithm after obtaining possible matching results; the third level is the operation scheduler and the supporting execution engine: including the string comparison engine, the regular NFA engine and the result reporting engine.

[0093] The software pre-compilation method applicable to the above multi-level hardware scanning circuit is as follows: first, extracting pure string substrings and regular units when pre-compiling the regular expression pattern library. Second, extracting and merging the common parts of multiple rules when pre-compiling the regular expression pattern library, and encoding them into a format suitable for hardware memory.

[0094] In summary, the above embodiments describe in detail different configurations of regular expression matchers and matching methods. Of course, the present invention includes but is not limited to the configurations listed in the above embodiments. Any content that is transformed based on the configurations provided in the above embodiments belongs to the scope of protection of the present invention. Those skilled in the art can draw inferences based on the contents of the above embodiments.

[0095] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0096] The above description is only a description of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any changes or modifications made by a person skilled in the art in the field of the present invention based on the above disclosure shall fall within the scope of protection of the claims.

Claims

1. A parallel regular expression matcher, characterized in that: include: A software pre-compilation unit is configured to pre-compile a plurality of regular expression rules in a regular expression rule set and convert them into regular expression rules in a format that conforms to hardware processing behavior, so as to be matched by a hardware matching circuit; The hardware matching circuit is configured to match the message to be matched with the regular expression rule in the format that conforms to the hardware processing behavior to obtain a matching result. The software pre-compilation unit includes a splitting module, a state merging module and a configuration table module, wherein: The splitting module is used to split multiple regular expression rules into pure strings and regular units; The state merging module is used to merge the pure character strings and regular units in state, extract the same pure character strings and regular units, and distinguish different pure character strings and regular units; The configuration table module encapsulates pure strings and regular units into a regular expression pattern library in a format that conforms to the hardware processing behavior, so as to be used for parameter configuration and state table initialization of the hardware matching circuit. The regular expression pattern library includes a pure string group, a regular unit group and a metadata group, wherein: The pure character string group is used to store pure character strings, and configure the relationship between pure character strings according to whether the pure character strings are the same; The regular unit group is used to store regular units and configure the relationship between regular units according to whether the regular units are the same; The metadata group is used to store metadata and configure the relationship between the pure string and the regular unit according to the metadata. The hardware matching circuit includes a pattern input module, a data input module, a regular matching core module and a matching result output module, wherein: The pattern input module performs parameter configuration on the regular matching core module according to the pure character string in the pure character string group and the regular unit in the regular unit group, and initializes the state table of the regular matching core module according to the metadata in the metadata group; The data input module is used to provide the message to be matched to the regular matching core module; The regular expression matching core module is used to match the message to be matched with the regular expression rule to generate a matching result; The matching result output module is used to output the matching result; Among them, the pattern input module, the data input module and the matching result output module perform bus protocol bridging or format conversion with the regular matching core module.

2. The parallel regular expression matcher according to claim 1, wherein: The regular expression matching core module includes a string preprocessing module, an operation scheduling module and an execution engine module, wherein: The string preprocessing module is used to scan the to-be-matched message according to the pure strings in the pure string group to generate a string queue; The operation scheduling module is used to execute corresponding programs according to the string queue to generate a request queue; The execution engine module scans the to-be-matched message according to the request queue, and determines whether the to-be-matched message matches.

3. The parallel regular expression matcher according to claim 2, wherein: The string preprocessing module includes a text scanning module and a text confirmation module, wherein: The text scanning module uses a Shift-OR algorithm circuit to pre-match the pure character strings in the message to be matched according to the pure character strings in the pure character string group to obtain a rough screening result; The text confirmation module uses a hardware hash algorithm to reconfirm the pure character strings in the coarse screening result according to the pure character strings in the pure character string group to obtain a fine screening result.

4. The parallel regular expression matcher according to claim 3, wherein: The Shift-OR algorithm circuit performs pipeline scanning on 16 characters simultaneously; After being scanned by the Shift-OR algorithm circuit, the byte sequence number to be confirmed of the matched data packet is put into the first-in-first-out queue of the index data structure of the text confirmation module for use by the text confirmation module; The text confirmation module reads the first-in-first-out queue of the index data structure, forms and queries the fine screening module index table, obtains the byte sequence number of the matching data packet, calculates the hash value, uses the hash value to search the string information table, and obtains the string length and string address; Generate a string queue from string length and string address.

5. The parallel regular expression matcher according to claim 4, wherein: The operation scheduler is responsible for forming the corresponding program according to the string request information and selecting the corresponding execution engine module. Each program includes instructions, operation codes and operands. Each instruction is sent to the request queue for subsequent execution engine module calls.

6. The parallel regular expression matcher according to claim 5, wherein: The execution engine module includes a character comparison engine, an NFA engine, a report engine and a message buffer; The request queue includes string comparison requests, regular NFA requests and result report requests; The character comparison engine performs accurate comparison on the fine screening results according to the string comparison request, including final one-by-one comparison of ultra-long pure string matches; The NFA engine performs corresponding NFA state machine operations on the message to be matched according to the regular NFA request, including selecting corresponding NFA in the corresponding field of the message to be matched according to the input request information, performing NFA state machine operations, and finally generating a matching result for use by a subsequent report engine; The report engine outputs the final matching result according to the result report request, including calculating the matching position according to the output of the character comparison engine and the NFA engine, extracting the matched sequence number, obtaining the matching result and outputting it; The message buffer is used to store and provide to-be-matched message data required by various execution engines.

7. The parallel regular expression matcher of claim 6, wherein: The number ratio and operation frequency of character comparison engines and NFA engines are weighed based on software performance evaluation and the physical characteristics of hardware circuits; Select regular expression rule sets and messages to be matched for typical applications to perform performance estimation, determine the size of the string preprocessing module under a certain matching probability, and determine the number and appropriate ratio of character comparison engines and NFA engines; According to the physical characteristics of the hardware implementation, the parameter configuration of each module is further determined.

Images