Regular expression feature extraction and matching method and device, equipment and readable medium

Abstract

The invention provides regular expression feature extraction and matching methods and devices, equipment and a readable medium, and belongs to the technical field of communication. The regular expression feature extraction method comprises the steps that a regular expression is analyzed, and information of meta-characters in the regular expression is determined; performing feature extraction on the regular expression according to the information of the meta characters to obtain at least one continuous character substring; and determining a longest combined feature substring based on the at least one continuous character substring, and taking the longest combined feature substring as a feature corresponding to the regular expression. The method is used for improving the regular expression matching efficiency.

Description

Technical Field

[0001] The present disclosure relates to the field of communication technology, and in particular to a regular expression feature extraction and matching method, device, equipment and readable medium. Background Art

[0002] Regular expressions are a tool for describing strings with a fixed format. For example, the regular expression "ab+" means that one "a" is followed by at least one "b" ("+" means that the previous character is repeated at least once), so "ab", "abb", "abbb"... all conform to this format, and the regular expression "ab+" is said to match these characters.

[0003] Regular expressions are widely used in network data acceleration fields such as data processing units (DPUs), smart network cards, and accelerator cards, mainly for feature matching at the application layer. Large-scale regular expression matching usually uses feature filtering and hardware acceleration methods, where feature filtering is a method of matching features first and then invoking rules, with the aim of reducing the number of rule invocations. One rule is one regular expression, which improves the matching efficiency of regular expressions.

[0004] The current feature filtering method does not have a good filtering effect on rules, especially in large-scale regular set scenarios, which will evoke a large number of unnecessary regular expressions, resulting in a long data reading time for the rules and reduced matching efficiency. Summary of the invention

[0005] The present invention provides a regular expression feature extraction and matching method, device, equipment and readable medium.

[0006] The first aspect of the present disclosure provides a regular expression feature extraction method, comprising:

[0007] Parsing the regular expression to determine the information of the metacharacters in the regular expression;

[0008] Performing feature extraction on the regular expression according to the information of the metacharacter to obtain at least one continuous character substring;

[0009] The longest combined feature substring is determined based on the at least one continuous character substring, and the longest combined feature substring is used as the feature corresponding to the regular expression.

[0010] A second aspect of the present disclosure provides a regular expression matching method, comprising:

[0011] Get the string sequence to be matched;

[0012] Matching the string sequence using features corresponding to each regular expression in the rule set to obtain features that match the character segments in the string sequence; wherein the features corresponding to the regular expressions are determined using the method described in the first aspect;

[0013] The character string sequence is matched based on the regular expression corresponding to the matched feature to obtain a matching result.

[0014] A third aspect of the present disclosure provides a regular expression feature extraction device, comprising:

[0015] A parsing module, used for parsing a regular expression to determine information of metacharacters in the regular expression;

[0016] An extraction module, used for performing feature extraction on the regular expression according to the information of the metacharacter to obtain at least one continuous character substring;

[0017] A processing module is used to determine a longest combined feature substring based on the at least one continuous character substring, and use the longest combined feature substring as a feature corresponding to the regular expression.

[0018] A fourth aspect of the present disclosure provides a regular expression matching device, characterized by comprising:

[0019] An acquisition module is used to obtain a string sequence to be matched;

[0020] A processing module, used to match the string sequence using features corresponding to each regular expression in a rule set to obtain features that match the character segments in the string sequence; wherein the features corresponding to the regular expressions are determined using the method described in the first aspect;

[0021] The matching module is used to match the string sequence based on the regular expression corresponding to the matching feature to obtain a matching result.

[0022] A fifth aspect of the present disclosure provides an electronic device, including:

[0023] at least one processor;

[0024] A memory having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to the first aspect or the method according to the second aspect;

[0025] At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

[0026] A sixth aspect of the present disclosure provides a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method according to the first aspect or the method according to the second aspect is implemented.

[0027] The present disclosure has the following advantages:

[0028] The information of metacharacters in the regular expression is obtained by parsing the regular expression, and the feature extraction of the regular expression is performed according to the information of the metacharacters to obtain at least one continuous character substring; the longest combined feature substring is determined based on the at least one continuous character substring, and the longest combined feature substring is used as the feature corresponding to the regular expression. By using the maximum combined feature substring extracted from the regular expression as the feature of the regular expression, the probability of the same features corresponding to different regular expressions is reduced in the scenario of large-scale regular sets, and the number of regular expressions evoked by matching features can be reduced as much as possible during filtering, thereby enhancing the feature filtering effect of regular expressions, thereby improving the matching efficiency of regular expressions, and improving the filtering effect and matching efficiency of large-scale rule sets that support millions of regular expressions. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 A schematic diagram of a usage scenario of a regular expression provided in an embodiment of the present disclosure;

[0030] Figure 2 A schematic diagram of a regular expression feature extraction method provided in an embodiment of the present disclosure;

[0031] Figure 3 A schematic diagram of a regular expression rule compilation process provided in an embodiment of the present disclosure;

[0032] Figure 4 A schematic diagram of a regular expression matching method provided in an embodiment of the present disclosure;

[0033] Figure 5 A schematic diagram of a regular expression matching process based on feature filtering provided in an embodiment of the present disclosure;

[0034] Figure 6 A schematic diagram of the structure of a feature extraction device for a regular expression provided in an embodiment of the present disclosure;

[0035] Figure 7 A schematic diagram of the structure of a regular expression matching device provided in an embodiment of the present disclosure;

[0036] Figure 8 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] The specific implementation of the present disclosure is described in detail below in conjunction with the accompanying drawings. It should be understood that the specific implementation described herein is only used to illustrate and explain the present disclosure, and is not used to limit the present disclosure.

[0038] As used in this disclosure, the term "and / or" includes any and all combinations of one or more of the associated listed items.

[0039] The terms used in the present disclosure are only used to describe specific embodiments and are not intended to limit the present disclosure.As used in the present disclosure, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0040] When the terms “comprising” and / or “made of…” are used in the present disclosure, it specifies the existence of the stated features, integers, steps, operations, elements and / or components, but does not exclude the existence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof.

[0041] Unless otherwise defined, all terms (including technical and scientific terms) used in this disclosure have the same meaning as those commonly understood by those of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning unless explicitly defined in this disclosure.

[0042] The inventor discovered during the research of the prior art that:

[0043] Common characters of regular expressions are shown in Table 1. Regular expressions can be divided into two major schools. One is the Portable Operating System Interface (POSIX) school, which is the POSIX standard regular expression library, which is divided into two standards: Basic Regular Expression (BRE) and Extended Regular Expression (ERE). Compared with BRE, ERE is more concise, and BRE supports backreferences. The other is the Perl Compatible Regular Expressions (PCRE) school, which is a regular expression library compatible with the Perl language and is also the mainstream regular expression library for modern programming languages. PCRE supports all POSIX expressions, and supports expressions such as non-greedy, forward (backward) lookaround, etc. PCRE is more powerful than POSIX, and most commercial hardware regular expressions support PCRE library.

[0044] Table 1 Common characters in regular expressions

[0045]

[0046]

[0047] There are two main types of regular expression matching engines: Deterministic Finite Automaton (DFA) and Non-Deterministic Finite Automaton (NFA). DFA is a deterministic finite automaton with low time complexity, so the algorithm matching speed is very fast, but the space complexity of DFA is exponentially related to the scale of the regular expression in the worst case, and there is a problem of state explosion, so it cannot support large-scale regular set compilation. NFA is a non-deterministic finite automaton with low space complexity, small memory overhead, and a linear relationship between the number of states and the length of the regular expression. It is suitable for large-scale regular set compilation, but due to the existence of the backtracking mechanism, the matching efficiency is low, so field programmable gate array (FPGA) / application specific integrated circuit (ASIC) is usually used for hardware acceleration. Table 2 shows the comparison of DFA and NFA engines.

[0048] Table 2 Comparison of DFA and NFA engines

[0049]

[0050] Note: m is the size of the regular expression rule set (size = number of expressions × expression length).

[0051] Taking the application of regular expressions in DPU as an example, the application of regular expressions in DPU is integrated into the Next Generation Firewall (NGFW) features, such as Figure 1The following is a schematic diagram of the use of regular expressions. NGFW is developed from traditional firewalls. Traditional firewalls usually use TCAM to detect five-tuples at the network layer. Later, UTM integrated traditional firewalls with IDS / IPS, anti-virus and other threat services to form unified threat management. However, UTM is a file-based serial detection mechanism, which is inefficient when detecting large traffic. NGFW is a flow-based parallel detection, and adds application identification, content filtering, Uniform Resource Locator (URL) filtering and other features on the basis of UTM, which improves the efficiency of regular expression detection. Combined with NGFW features, the application scenarios of regular expressions are as follows: application identification, intrusion detection / intrusion prevention system (IDS / IPS), anti-virus, URL filtering, content filtering, web application firewall (WAF) (for HTTP programs), log analysis, etc. In addition, all of the above application scenarios are based on deep packet inspection (DPI), a technology that matches traffic features by establishing a feature library and performing string or regular matching on traffic features.

[0052] There are currently two IPs that support regular expression hardware acceleration, Regular Expression Processor (RXP) IP and Gregex IP. Table 3 shows a comparison of the two regular expression hard IPs. The single-core processing performance of both IPs can reach 80Gb / s, and the number of rules supported can reach millions (related to the external storage capacity). Taking RXP IP as an example, the working principle of RXP is as follows: After the message enters RXP, the sub-rule of the rule is first matched in the prefix engine. After the sub-rule is matched, one or more regular search threads are called up, and the thread control engine dispatches them to the thread processing engine for the final regular expression match. The sub-rule here refers to: RXP selects a prefix of 1 to 4 from the first 8 bytes or the last 8 bytes of the rule to call up the search thread, and the number of parallel processing threads reaches up to 8000.

[0053] Among the methods for improving the efficiency of regular expressions, the feature filtering used in large-scale regular expression matching mainly includes the following methods:

[0054] 1. Single feature evoked. Taking RXP as an example, a pure character feature evoked rule with a length of 1 to 4 is selected from the first 8 bytes or the last 8 bytes of the rule. For example, the rule / ABC

[12] / extracts ABC as feature filtering, and the rule / A[az][0-9]BCDEFG / extracts BCDE or CDEF or DEFG as feature filtering. The feature bytes taken by RXP are very short, so the feature filtering efficiency will be very high, but the number of evoked regular expressions will also be large, and the filtering effect is not obvious. For example, the parallel threads of RXP have reached 8000. In addition, the longest exact substring method is used to evoke the rule, that is, to extract the longest exact substring of the rule, so that the filtering effect of the feature is enhanced. The longest exact substring is composed of the suffix of the left neighbor, the exact substring, and the prefix of the right neighbor. For example, the regular expression / ABCD+(EC|FC)GHI(JD|JK)L / will extract the substring set {ABCD, CGHIJ, L}. Therefore, the longest exact substring is CGHIJ.

[0055] 2. Prefix and suffix recall, for example, for the regular expression "ac*defgh+xy", assuming that the length of the prefix and suffix are both 2, the prefix and suffix set {acxy, adxy} will be extracted. However, this method does not support intermediate feature recall, and because the fixed-length prefix and suffix set extracted by the regular expression will have more features, it may cause a large number of regular expressions to be recalled;

[0056] 3. Message feature invocation: By establishing a mapping relationship between protocol variables and regular expressions, the number of regular expressions to be invoked is reduced, thereby improving matching efficiency. However, protocol variables and regular expressions are independent and are not extracted from the regular expressions, so accurate filtering is not possible.

[0057] In short, the common problem of these efficiency improvement methods is that the filtering effect of features on rules is not good enough, especially in the scenario of large-scale regular sets. Whether it is a single feature or a prefix-suffix method to evoke regular matching, a large number of unnecessary regular expressions will be evoked, resulting in too long data reading time for rules and reduced matching efficiency.

[0058] In order to solve this problem, an embodiment of the present disclosure provides a regular expression feature extraction method, which can be applied to any electronic device, such as DPU, smart network card, accelerator card and other electronic devices in the field of network data acceleration.

[0059] Figure 2 The figure shows a flow chart of a method for extracting features from a regular expression. The flow chart of the method for extracting features from a regular expression mainly includes the following steps:

[0060] Step 201, parse the regular expression to determine the information of metacharacters in the regular expression.

[0061] Among them, metacharacters are some characters with special meanings in regular expressions, such as the characters other than the 256 ASCII codes in Table 1. These characters contain special symbols and are metacharacters.

[0062] Step 202: extract features from the regular expression according to the metacharacter information to obtain at least one continuous character substring.

[0063] In some embodiments, the feature extraction of the regular expression according to the metacharacter information to obtain at least one continuous character substring includes but does not include the following two situations:

[0064] Case 1: When the metacharacter information indicates that the regular expression does not include the metacharacter, the regular expression is regarded as a continuous character substring.

[0065] For example, if the regular expression has no metacharacters, the continuous character substring extracted by the feature is the regular expression itself. For example, the regular expression "abcdefgh" does not have any metacharacters, so the feature is directly taken as "abcdefgh".

[0066] Case 2: When the metacharacter information is that the regular expression includes the metacharacter, at least one continuous character substring is extracted according to the type and position of the first metacharacter in the regular expression.

[0067] For example, if the regular expression includes metacharacters, feature extraction is performed based on the type and position of the first metacharacter. For example, in the regular expression "abc+def*gh", the first metacharacter is "+". This type of metacharacter indicates that the character c is repeated one or more times. The continuous character substring is determined according to the type of the "+" and its position in the regular expression.

[0068] In some embodiments, when the information of the metacharacter is that the regular expression includes metacharacters, at least one continuous character substring is extracted according to the type and position of the first metacharacter in the regular expression, including: when the information of the metacharacter is that the regular expression includes metacharacters, judging whether there is a prefix and suffix according to the type and position of the first metacharacter in the regular expression, and extracting the longest suffix substring and / or the prefix in the suffix according to the judgment result.

[0069] In some embodiments, when the metacharacter information includes metacharacters in the regular expression, at least one continuous character substring is extracted according to the type and position of the first metacharacter in the regular expression, including but not limited to the following cases:

[0070] Case A: When the first metacharacter in the regular expression represents repetition and the number of repetitions does not include zero, the character substring before the first metacharacter in the regular expression is used as a prefix, and when the character substring after the first metacharacter in the regular expression is not entirely metacharacters, the longest suffix substring is extracted from the character substring after the first metacharacter in the regular expression.

[0071] For example, in the regular expression "abc+def*gh", the first metacharacter is "+", which is repeated at least once and is a metacharacter that is repeated and does not include zero repetitions. The character substring "abc" before the first metacharacter "+" is used as a prefix. The character substring after the first metacharacter "+" is "def*gh", which is not entirely metacharacters. The longest suffix substring is extracted from "def*gh".

[0072] Case B, when the first metacharacter in the regular expression represents repetition and the number of repetitions includes zero, and when there are other characters before the first metacharacter in the regular expression in addition to the repeated characters, the character substring before the first metacharacter in the regular expression that does not contain the repeated characters is used as a prefix, and when the character substring after the first metacharacter in the regular expression is not entirely metacharacters, the longest suffix substring is extracted from the character substring after the first metacharacter in the regular expression;

[0073] For example, in the regular expression "abc*def+gh", the first metacharacter is "*", which contains metacharacters that are repeated 0 times and are repeated including zero times. Before the first metacharacter "*", in addition to the repeated character "c", there are other characters "ab". The character substring "ab" that does not contain the repeated character "c" before the first metacharacter "*" is taken as a prefix; the character substring "def+gh" after the first metacharacter "*" only contains the metacharacter "+", not all metacharacters. The longest suffix substring is extracted from the character substring "def+gh" after the first metacharacter "*".

[0074] Case C, when the first metacharacter in the regular expression represents repetition and the number of repetitions includes zero, and when there are no other characters before the first metacharacter in the regular expression except the repeated character, the regular expression is split into a first expression corresponding to the first metacharacter being repeated zero times and a second expression in which the first metacharacter is repeated excluding zero times, a prefix is ​​extracted based on the first expression and the second expression, and when the character substring after the first metacharacter in the regular expression is not entirely metacharacters, the longest suffix substring is extracted from the character substring after the first metacharacter in the regular expression.

[0075] For example, the regular expression "c*def+gh", the first metacharacter is "*", which contains 0 repetitions. It is a case where the first metacharacter is a metacharacter that is repeated and the number of repetitions includes zero. There is no other character before the first metacharacter "*" except the repeated character "c", so the regular expression is split into the first expression "def+gh" corresponding to the first metacharacter repeated zero times, and the second expression "c+def+gh" where the first metacharacter is repeated excluding zero times. The prefix "def" is extracted from the first expression, and the prefix "c" is extracted from the second expression; the character substring "def+gh" after the first metacharacter "*" only includes one metacharacter "+", which is a case where it is not all metacharacters, and the longest suffix substring is extracted from the character substring after the first metacharacter "*".

[0076] Among them, for the regular expression that can extract the prefix, the prefix is ​​directly extracted. For the regular expression that cannot directly extract the prefix, the regular expression is converted so that the converted regular expression can extract the prefix as much as possible, so that when the regular expression is matched, the matching can start from the end position of the prefix of the string sequence to be matched (such as a message, etc.), thereby shortening the matching time and improving the matching efficiency of the regular expression.

[0077] Compared with the situation that too many rules are evoked and the matching efficiency is reduced by using a single feature of a regular expression for pre-filtering, the embodiment of the present disclosure uses the longest combined feature substring obtained by combining the longest substrings of the prefix and suffix for pre-filtering, so that the number of evoked rules is reduced to the greatest extent, thereby improving the matching efficiency of the regular expression.

[0078] In an exemplary embodiment, the prefix extraction process without prefix rules is described in detail as follows:

[0079] There are two directions for processing the rules without prefixes in the existing technology: ① extracting other features; ② not extracting features and directly performing regular matching. However, there are two advantages of using prefix filtering for regular expression matching: ① prefix pre-filtering, reducing the number of rule calls and improving matching efficiency; ② performing regular matching from the end position of the prefix to speed up the matching speed. For this reason, the idea of ​​prefix extraction in the embodiment of the present disclosure is: ① for regular expressions with prefixes, directly extract the prefix; ② if the prefix cannot be directly extracted in the regular expression, try to convert the rules without prefixes into rules with prefixes to improve the filtering effect of the prefix on the rules.

[0080] Here we define the concepts of prefix and suffix in regular expressions: ① The first metacharacter in the regular expression that does not contain 0 repetitions is used as a marker, the character or string before the metacharacter is the prefix, and the part after the metacharacter is the suffix; ② If the first metacharacter contains 0 repetitions, the prefix does not include the repeated part; the following example illustrates:

[0081] Rule 1: abc+def*gh, the first metacharacter is "+", which is repeated at least once, so the rule prefix is ​​"abc" and the suffix is ​​"def*gh";

[0082] Rule 2: abc*def+gh, the first metacharacter is "*", and it contains 0 repetitions, so the rule prefix is ​​"ab" and the suffix is ​​"def+gh";

[0083] Rule 3: c*def+gh, the first metacharacter is "*", and it is repeated 0 times. Then the rule has no prefix and the suffix is ​​"def+gh".

[0084] There are four non-prefix rules for conversion (assuming P x For a single character, T x is any expression, P x and T x The "x" in each column is a subscript), as shown in Table 3:

[0085] Table 3 Summary of some non-prefix rule conversions

[0086]

[0087] The specific implementation is as follows:

[0088] Specific implementation method 1: "P x *T x " type, converted to pure character "T x ” and “P x +T x ”, “P x +T x "Extractable prefix "P x ". For example, for the prefix-free regular expression "a*b*cd", the compiler will first split it into two regular expressions: "b*cd" and "a+b*cd", and "b*cd" will be further split into two regular expressions "cd" and "b+cd", so a total of three regular expressions are split: "cd", "b+cd" and "a+b*cd", and "b+cd" and "a+b*cd" can extract the prefixes "b" and "a" respectively. If any of the three regular expressions matches, it means that the regular expression "a*b*cd" before the split matches.

[0089] Specific implementation method 2: "P x {0,n}T x " type, converted to "T x ” and “P x P x {0,n-1}T x ”, where “Px P x {0,n-1}T x "Extractable prefix "P x For example, the compiler will split the prefix-free regular expression "a{0,3}bcd" into two regular expressions: "bcd" and "aa{0,2}bcd", where "aa{0,2}bcd" can extract the prefix "a";

[0090] Specific implementation method 3: "P x ? T x " type, can be converted to "T x ” and “P x T x ”, “P x T x "Extractable prefix "P x For example, the compiler will split the prefix-free regular expression "a? b+cd" into two regular expressions: "b+cd" and "ab+cd". These two regular expressions can extract the prefixes "b" and "ab" respectively.

[0091] Specific implementation method 4: "P x |T x " type, can be converted to "P x ” and “T x ”, “P x " itself is a prefix. For example, the compiler will split the unprefixed regular expression "a|(b+cd)" into two regular expressions: "a" and "b+cd", where "b+cd" can extract the prefix "b".

[0092] In some embodiments, extracting the longest suffix substring from the character substring after the first metacharacter in the regular expression includes: extracting the longest character string from the character substring after the first metacharacter in the regular expression; depending on the number of the longest character string extracted, including but not limited to the following situations:

[0093] Case 1: When a longest string is extracted, the extracted longest string is used as the longest suffix substring;

[0094] For example, the regular expression is: "ab+c*de*fgh*ijkl", and the suffix is ​​"c*de*fgh*ijkl", then the set of necessary factors in the suffix is ​​{d, fg, ijkl}, and the longest string extracted is "ijkl", so the longest suffix substring in the suffix is ​​"ijkl".

[0095] Case 2: When at least two longest character strings of the same length are extracted, the longest character string with the least number of repetitions is selected from the at least two longest character strings of the same length according to the number of repetitions of each feature in the rule set as the longest suffix substring;

[0096] For example, the regular expression is "abc+de+fghi+wxyz", and the suffix is ​​"de+fghi+wxyz", then the set of necessary factors in the suffix is ​​{de, fghi, wxyz}, and the two longest strings extracted are "fghi" and "wxyz", and assuming that the feature "fghi" in the rule set is repeated less times than "wxyz", then "fghi" is taken as the longest suffix substring.

[0097] Case 3: When the longest string is not extracted, there is no longest suffix substring.

[0098] For example, in the regular expression "\d.*\w{1,3}", all characters after the first metacharacter are metacharacters, and the longest string cannot be extracted, that is, there is no longest suffix substring.

[0099] Step 203: determine the longest combined feature substring based on the at least one continuous character substring, and use the longest combined feature substring as the feature corresponding to the regular expression.

[0100] In some embodiments, the determining of the longest combined feature substring based on the at least one continuous character substring includes but is not limited to the following cases:

[0101] Case a, when the at least one continuous character substring includes the longest substring of the prefix and the suffix, the combination of the longest substring of the prefix and the suffix is ​​used as the longest combined feature substring;

[0102] For example, in the regular expression "abc.*def", the prefix "abc" is extracted from the string before the first sub-character ".", and the longest suffix substring "def" is extracted from the string after the first sub-character ".". The combination of "abc" and "def" is taken as the longest combined feature substring.

[0103] Case b, when the prefix is ​​included in the at least one continuous character substring and the suffix longest substring is not included, the prefix is ​​used as the longest combined feature substring;

[0104] For example, in the regular expression "abc.*\d{1,3}", the prefix "abc" is extracted from the string before the first metacharacter ".", and the string after the first metacharacter "." is full of metacharacters. The longest suffix substring is not extracted, so "abc" is taken as the longest combined feature substring.

[0105] Case c, when the at least one continuous character substring includes the longest suffix substring and does not include the prefix, the longest suffix substring is used as the longest combined feature substring;

[0106] For example, in the regular expression "\dabc+de+hijk", the longest suffix substring is "hijk", excluding the prefix, and the longest suffix substring is taken as the longest combined feature substring.

[0107] By extracting the longest combined feature substring of the regular expression, as few regular expressions as possible for matching are evoked through the longest combined feature substring, that is, pre-filtering obtains as few regular expressions as possible for matching, so as to improve the matching efficiency of subsequent regular expressions.

[0108] In the exemplary embodiment, the longest combined feature substring of the regular expression is extracted as follows:

[0109] ①After entering the rule set, first determine whether the regular expression in the rule set has a prefix. If so, extract the prefix. If not, try to convert it into a regular expression with a prefix and then extract the prefix;

[0110] ② Regardless of whether the regular expression can extract the prefix, it will try to extract the longest suffix substring in the suffix, so the final extracted feature set is {prefix+suffix longest substring, prefix, suffix longest substring}. The suffix longest substring here refers to selecting the longest necessary factor from the necessary factor set obtained from the suffix. For example, the regular expression is: "ab+c*de*fgh*ijkl", and the suffix is ​​"c*de*fgh*ijkl", then the necessary factor set of the suffix is ​​{d, fg, ijkl}, so the suffix longest substring of the suffix is ​​"ijkl";

[0111] ③ If the regular expression cannot extract any features, for example, all characters in the regular expression are composed of metacharacters, then no features can be extracted and regular expression matching can be performed directly. For example: the regular expression "\d.*\w{1,3}" cannot extract any features and regular expression matching can be performed directly.

[0112] In the exemplary embodiment, specific implementation use cases are exemplified as follows:

[0113] Specific implementation 5: If the regular expression consists of a pure character prefix P x 、Suffix T x If the suffix can extract the longest substring of the suffix, then take P x and T xThe longest suffix substring combination of the suffix is ​​used as the call. If there are two longest suffix substrings of the same length, the longest suffix substring with fewer repetitions in the rule set is selected. For example, the regular expression "abc+de+fghi+wxyz", P x It is "abc", T x is "de+fghi+wxyz", and assuming that "fghi" is repeated less times than "wxyz", then the combination of "abc" and "fghi" evokes the regular expression "abc+de+fghi+wxyz";

[0114] Specific implementation 6: If the regular expression has only the suffix T x , but if there is no prefix and it cannot be converted, then the suffix T is taken x The longest suffix substring of is used as the trigger. If there are two longest suffix substrings of the same length, the longest suffix substring with fewer repetitions in the rule set is selected. For example, the regular expression "\dabc+de+hijk+xyzw", assuming that "hijk" is repeated less times than "xyzw", then "hijk" is used to trigger the regular expression "\dabc+de+hijk+xyzw";

[0115] Specific implementation method 7: If the regular expression consists of the prefix P x 、Suffix T x If the suffix cannot extract characters or character substrings, only the prefix P is taken. x As a trigger. For example, if the regular expression is "abc+\d\w{1,3}", then the "abc" trigger regular expression "\dabc+de+hijk+xyzw" is extracted.

[0116] In an exemplary embodiment, Figure 3 The following is a schematic diagram of the regular expression rule compilation process, which mainly includes:

[0117] Step 301, input a regular expression set;

[0118] Step 302, determine whether the regular expression has a prefix, if not, execute step 303, if yes, execute step 305;

[0119] Step 303, determine whether it can be converted into a regular expression with a prefix, if so, execute step 304, otherwise, execute step 309;

[0120] Step 304, performing regular expression conversion;

[0121] Step 305, extracting the prefix of the regular expression;

[0122] Step 306, determine whether the suffix can extract the longest suffix substring, if so, execute step 307, otherwise, execute step 308;

[0123] Step 307, taking the longest substring combination of the prefix and the suffix as the feature of the regular expression, and executing step 311;

[0124] Step 308, get the prefix, and execute step 311;

[0125] Step 309, determine whether the suffix can extract the longest suffix substring, if so, execute step 311, otherwise, execute step 312;

[0126] Step 310, take the longest suffix substring and execute step 311;

[0127] Step 311, calling the regular expression;

[0128] Step 312, regular expression matching;

[0129] Step 313: Report the matching result and end the process.

[0130] In the disclosed embodiment, the information of metacharacters in the regular expression is obtained by parsing the regular expression, and the feature extraction of the regular expression is performed based on the metacharacter information to obtain at least one continuous character substring; the longest combined feature substring is determined based on the at least one continuous character substring, and the longest combined feature substring is used as the feature corresponding to the regular expression. By using the maximum combined feature substring extracted from the regular expression as the feature of the regular expression, the probability of having the same features corresponding to different regular expressions is reduced in the scenario of large-scale regular sets, and the number of regular expressions evoked by matching features can be reduced as much as possible during filtering, thereby enhancing the feature filtering effect of the regular expression, thereby improving the matching efficiency of the regular expression, and improving the filtering effect and matching efficiency of large-scale rule sets that support millions of regular expressions.

[0131] For example, there are 4 regular expression rules, Rule 1: "abc.*\d{1,3}", Rule 2: "abc.*def", Rule 3: "abc.*de+klm", Rule 4: "abc.*de+kln". If the traditional single feature extraction method is used, the 4 rules all extract the prefix "abc" and evoke 4 rules at the same time. If the method provided by the embodiment of the present disclosure is used, Rule 1 extracts "abc", Rule 2 extracts "abc" and "def", Rule 3 extracts "abc" and "klm", and Rule 4 extracts "abc" and "kln". In this way, each rule can extract a unique feature, and the original 1 prefix evokes 4 rules, which is reduced to evoke 1 rule, which can greatly improve the matching efficiency of the regular expression.

[0132] Based on this example, the rationality of the longest combination feature substring recall rule is analyzed as follows: The purpose of the longest combination feature recall is to reduce the number of rules recalled, that is, to "miss" the recall; the so-called "missed" rules are not really missed, but other features will be selected for recall. In this example, Rule 2 was originally recalled with "abc", but "abc" will recall 4 rules at the same time. If a combination feature is used, the feature of Rule 2 is "abc" plus "def", then the combination feature will only recall Rule 2, and Rules 1, 3, and 4 are not "missed", but other features are selected for recall, but the number of feature recalls is reduced.

[0133] Based on the same concept, a regular expression matching method is provided in an embodiment of the present disclosure, and the method can be applied to any electronic device.

[0134] Figure 4 The figure shows a flow chart of the regular expression matching method, which mainly includes:

[0135] Step 401: Obtain a string sequence to be matched.

[0136] The meaning of the string sequence to be matched may vary according to different application scenarios, including but not limited to application identification, intrusion detection or defense, anti-virus, content filtering, and Uniform Resource Locator (URL) filtering. For example, the string sequence to be matched may be a message to be matched.

[0137] Step 402, using the features corresponding to each regular expression in the rule set to match the string sequence, to obtain features matching the character segments in the string sequence; wherein the features corresponding to the regular expressions are determined using the regular expression feature extraction method provided in the above embodiments.

[0138] In some embodiments, the use of features corresponding to each regular expression in the rule set to match the string sequence to obtain features that match character segments in the string sequence includes: using features corresponding to each regular expression in the rule set to match the string sequence to obtain each character segment in the string sequence that has matching features; in the case where identical character segments exist in each character segment that has matching features, deduplicating the identical character segments to obtain features that match each character segment that has matching features after deduplication.

[0139] In the case where there are multiple character segments in a string sequence that match the features of multiple regular expressions in the rule set, if at least two of the multiple character segments are the same, deduplication processing is performed, for example, only the character segment that successfully matches for the first time in the field segment is retained, and the regular expression corresponding to the feature that matches the character segment is called out to avoid calling the same regular expression multiple times, thereby reducing the number of subsequent regular expression matches and improving matching efficiency.

[0140] In some embodiments, more algorithms can be used to match the string sequence using the features corresponding to each regular expression in the rule set, for example, using the bit splitting (Aho-Corasick Algorithm, AC) algorithm, PAC algorithm, QSV algorithm, etc. Among them, the bit splitting AC algorithm: the AC algorithm is based on a deterministic automaton, and the time complexity of the algorithm is linearly related to the length of the input string. The core of the bit splitting automaton algorithm is to divide the original AC algorithm automaton into multiple micro-automatons, thereby reducing the conversion edges and greatly reducing the compilation memory of the AC algorithm; PAC algorithm: a hardware implementation method of a parallel pipeline multi-string matching AC algorithm, the PAC algorithm eliminates all backward edges and invalid edges in the state transition table, and has high matching efficiency, but the compilation requires high storage capacity; QSV algorithm: a fast sampling and verification algorithm, first quickly samples and compares the fixed-length prefix of the string, and then verifies the bytes after the prefix until it matches or does not match. QSV is based on a hash algorithm, and the corresponding storage capacity requirements are relatively small, but there will be problems such as hash conflicts.

[0141] Step 403: Match the string sequence based on the regular expression corresponding to the matched feature to obtain a matching result.

[0142] In some embodiments, the string sequence is matched based on the regular expression corresponding to the matched feature to obtain a matching result, including: finding a mapping relationship between the regular expression and the feature to obtain a regular expression corresponding to the matched feature; when the feature is the same as the corresponding regular expression, outputting the regular expression as the matching result; when the feature is part of the corresponding regular expression, invoking one or more regular expressions corresponding to the feature to match the string sequence in parallel to obtain a matching result.

[0143] In some embodiments, the matching of the string sequence based on the regular expression corresponding to the matched feature to obtain a matching result includes: based on multiple parallel non-deterministic finite state machines (also called NFA state machines), matching the string sequence using the regular expression corresponding to the matched feature to obtain a matching result.

[0144] Through multiple parallel NFA state machines, the regular expression matching module is flexibly scheduled to perform matching according to the number of evoked regular expressions. The matching rules adopted by the matching module can be stored in an external memory, and of course other storage forms can also be used.

[0145] In some embodiments, after obtaining the matching result, the matching result may be further processed, for example, the matching result may be reported to software for further processing.

[0146] The method flow of regular expression matching provided in the embodiments of the present disclosure may be implemented by hardware acceleration, for example, by using a Field Programmable Gate Array (FPGA) or an Application Specific Integrated Circuit (ASIC).

[0147] In an exemplary embodiment, Figure 5 The figure shows a regular expression matching process diagram based on feature filtering, which mainly includes the following steps:

[0148] Step 501, the message to be matched is input;

[0149] Step 502, feature matching (prefix, suffix longest substring, longest combination feature) is performed on the input message;

[0150] Step 503, determine whether the message has a feature matching result, if so, execute step 504, otherwise, end the process;

[0151] Step 504, performing deduplication processing on the matched features, and retaining only one identical feature;

[0152] Step 505, searching for a mapping relationship between features and rules, where a feature is a rule or a rule invocation, and a rule is a regular expression;

[0153] Step 506, determine whether the matched feature is the rule itself, if so, execute step 508, otherwise, execute step 507;

[0154] Step 507, regular expression matching, that is, matching the regular expression evoked by the matched feature with the message;

[0155] Step 508: If the feature is a rule itself or a regular expression match, the matching result is reported and the process ends.

[0156] In an exemplary embodiment, for example, there are three rules: Rule 1: "abc1234", Rule 2: "abcd*1234", Rule 3: "abcd*4567". Rule 1 consists of pure characters, so the feature is itself, and the matching result is directly output after the feature is matched, while the feature of Rule 2 consists of the prefix "abc" and the suffix "1234", and Rule 3 consists of the prefix "abc" and the suffix "4567". Assuming that the input message to be matched is "Zteabc1234abcd1234End", the message will match 2 "abc" and 2 "1234". After feature deduplication, only the first "abc" and "1234" are retained. Rule 1 directly outputs the matching result after feature matching, and Rule 2 is triggered by the combined features "abc" and "1234" and performs regular expression matching. The longest suffix substring of Rule 3 cannot be matched, so the regular expression is not triggered.

[0157] The step division of the above various methods is only for clear description. When implemented, they can be combined into one step or some steps can be split and decomposed into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this disclosure. Adding insignificant modifications to the algorithm or process or introducing insignificant designs without changing the core design of the algorithm and process are all within the protection scope of this disclosure.

[0158] In an embodiment of the present disclosure, a regular expression feature extraction device is provided. The specific implementation of the device can be found in the relevant description of the method embodiment, which will not be repeated here. Figure 6 The schematic diagram of the structure of the device is shown, which mainly includes:

[0159] The parsing module 601 is used to parse the regular expression and determine the information of the metacharacters in the regular expression;

[0160] An extraction module 602, configured to perform feature extraction on the regular expression according to the information of the metacharacters to obtain at least one continuous character substring;

[0161] The processing module 603 is used to determine the longest combined feature substring based on the at least one continuous character substring, and use the longest combined feature substring as the feature corresponding to the regular expression.

[0162] A regular expression matching device is provided in an embodiment of the present disclosure. The specific implementation of the device can refer to the relevant description of the method embodiment, which will not be repeated here. Figure 7 The schematic diagram of the structure of the device is shown, which mainly includes:

[0163] An acquisition module 701 is used to acquire a string sequence to be matched;

[0164] Processing module 702, used to match the string sequence with features corresponding to each regular expression in the rule set to obtain features matching the character segments in the string sequence; wherein the features corresponding to the regular expressions are determined using the regular expression feature extraction method described above;

[0165] The matching module 703 is used to match the string sequence based on the regular expression corresponding to the matched feature to obtain a matching result.

[0166] The functions or modules included in the device provided in the embodiments of the present disclosure can be used to execute the method described in the method embodiments. The specific implementation and technical effects thereof can be referred to the description of the method embodiments above, and will not be described again here for the sake of brevity.

[0167] It should be noted that all modules involved in this embodiment are logic modules. In practical applications, a logic unit may be a physical unit, or a part of a physical unit, or may be implemented as a combination of multiple physical units. In addition, in order to highlight the innovative part of the present disclosure, this embodiment does not introduce units that are not closely related to solving the technical problems proposed by the present disclosure, but this does not mean that there are no other units in this embodiment.

[0168] Reference Figure 8 , an embodiment of the present disclosure provides an electronic device, comprising:

[0169] at least one processor 801;

[0170] A memory 802 having at least one program stored thereon, and when the at least one program is executed by the at least one processor, the at least one processor implements the above method;

[0171] At least one I / O interface 803 is connected between the processor and the memory and is configured to implement information exchange between the processor and the memory.

[0172] Among them, the processor 801 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 802 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); the I / O interface (read-write interface) 803 is connected between the processor 801 and the memory 802, and can realize information interaction between the processor 801 and the memory 802, including but not limited to a data bus (Bus), etc.

[0173] In some embodiments, the processor 801, the memory 802, and the I / O interface 803 are connected to each other through a bus, and further connected to other components of the computing device.

[0174] This embodiment further provides a computer-readable medium on which a computer program is stored. When the program is executed by a processor, the method provided in this embodiment is implemented. To avoid repeated description, the specific steps of the method are not repeated here.

[0175] It will be appreciated by those skilled in the art that all or some of the steps, systems, and functional modules / units in the methods applied for above may be implemented as software, firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed by several physical components in cooperation. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or non-transitory medium) and a communication medium (or temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0176] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0177] Those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features from different embodiments is meant to be within the scope of the present embodiment and to form different embodiments.

[0178] It is to be understood that the above embodiments are merely exemplary embodiments used to illustrate the principles of the present disclosure, but the present disclosure is not limited thereto. For those of ordinary skill in the art, various modifications and improvements can be made without departing from the spirit and substance of the present disclosure, and these modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A regular expression feature extraction method, characterized in that: include: Parsing the regular expression to determine the information of the metacharacters in the regular expression; Performing feature extraction on the regular expression according to the information of the metacharacter to obtain at least one continuous character substring; The longest combined feature substring is determined based on the at least one continuous character substring, and the longest combined feature substring is used as the feature corresponding to the regular expression.

2. The method according to claim 1, characterized in that The step of extracting features from the regular expression according to the information of the metacharacter to obtain at least one continuous character substring includes: When the information of the metacharacter is that the regular expression does not include the metacharacter, treating the regular expression as a continuous character substring; In the case where the metacharacter information indicates that the regular expression includes the metacharacter, at least one continuous character substring is extracted according to the type and position of the first metacharacter in the regular expression.

3. The method according to claim 2, characterized in that In the case where the information of the metacharacter includes the metacharacter in the regular expression, extracting at least one continuous character substring according to the type and position of the first metacharacter in the regular expression includes: In the case where the metacharacter information includes metacharacters in the regular expression, whether there is a prefix and a suffix is ​​determined according to the type and position of the first metacharacter in the regular expression, and the longest suffix substring and / or the prefix in the suffix is ​​extracted according to the determination result.

4. The method according to claim 2, characterized in that: In the case where the information of the metacharacter includes the metacharacter in the regular expression, extracting at least one continuous character substring according to the type and position of the first metacharacter in the regular expression includes: When the first metacharacter in the regular expression represents repetition and the number of repetitions does not include zero, take the character substring before the first metacharacter in the regular expression as a prefix, and when the character substring after the first metacharacter in the regular expression is not entirely metacharacters, extract the longest suffix substring from the character substring after the first metacharacter in the regular expression; In the case where the first metacharacter in the regular expression represents repetition and the number of repetitions includes zero, and in the case where there are other characters before the first metacharacter in the regular expression in addition to the repeated characters, the character substring before the first metacharacter in the regular expression that does not contain the repeated characters is used as a prefix, and in the case where the character substring after the first metacharacter in the regular expression is not entirely metacharacters, the longest suffix substring is extracted from the character substring after the first metacharacter in the regular expression; In the case where the first metacharacter in the regular expression represents repetition and the number of repetitions includes zero, and in the case where there are no other characters except the repeated character before the first metacharacter in the regular expression, the regular expression is split into a first expression corresponding to the first metacharacter being repeated zero times and a second expression where the first metacharacter is repeated excluding zero times, a prefix is ​​extracted based on the first expression and the second expression, and in the case where the character substring after the first metacharacter in the regular expression is not entirely metacharacters, the longest suffix substring is extracted from the character substring after the first metacharacter in the regular expression.

5. The method according to claim 4, characterized in that The step of extracting the longest suffix substring from the character substring after the first metacharacter in the regular expression includes: Extract the longest character string from the character substring after the first metacharacter of the regular expression; When a longest string is extracted, the extracted longest string is used as the longest suffix substring; When at least two longest character strings of the same length are extracted, according to the number of repetitions of each feature in the rule set, the longest character string with the least number of repetitions is selected from the at least two longest character strings of the same length as the longest suffix substring; If the longest character string is not extracted, there is no longest suffix substring.

6. The method according to claim 3, characterized in that The determining the longest combined feature substring based on the at least one continuous character substring comprises: In the case where the at least one continuous character substring includes the prefix and the suffix longest substring, taking the combination of the prefix and the suffix longest substring as the longest combined characteristic substring; In the case where the prefix is ​​included in the at least one continuous character substring and the suffix longest substring is not included, the prefix is ​​used as the longest combined feature substring; In the case that the at least one continuous character substring includes the longest suffix substring but does not include the prefix, the longest suffix substring is used as the longest combined feature substring.

7. A regular expression matching method, characterized in that: include: Get the string sequence to be matched; Matching the string sequence using features corresponding to each regular expression in the rule set to obtain features that match the character segments in the string sequence; wherein the features corresponding to the regular expressions are determined using the method described in any one of claims 1 to 6; The character string sequence is matched based on the regular expression corresponding to the matched feature to obtain a matching result.

8. The method according to claim 7, characterized in that The matching of the character string sequence with the features corresponding to the regular expressions in the rule set to obtain features matching the character segments in the character string sequence includes: Matching the string sequence using features corresponding to each regular expression in the rule set to obtain each character segment in the string sequence having matching features; In the case that the same character segments exist in the character segments with matching features, deduplication processing is performed on the same character segments to obtain matching features of the character segments with matching features after deduplication processing.

9. The method according to claim 7, characterized in that: The matching of the character string sequence based on the regular expression corresponding to the matched feature to obtain a matching result includes: Based on multiple parallel non-deterministic finite state machines, the character string sequence is matched using regular expressions corresponding to the matching features to obtain a matching result.

10. A feature extraction device for a regular expression, characterized in that: include: A parsing module, used for parsing a regular expression to determine information of metacharacters in the regular expression; An extraction module, used for performing feature extraction on the regular expression according to the information of the metacharacter to obtain at least one continuous character substring; A processing module is used to determine a longest combined feature substring based on the at least one continuous character substring, and use the longest combined feature substring as a feature corresponding to the regular expression.

11. A regular expression matching device, characterized in that: include: An acquisition module is used to obtain a string sequence to be matched; A processing module, used to match the string sequence using features corresponding to each regular expression in a rule set to obtain features matching the character segments in the string sequence; wherein the features corresponding to the regular expressions are determined using the method described in any one of claims 1 to 6; The matching module is used to match the string sequence based on the regular expression corresponding to the matching feature to obtain a matching result.

12. An electronic device, characterized in that: include: at least one processor; A memory having at least one program stored thereon, wherein when the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 6 or implements the method according to any one of claims 7 to 9; At least one I / O interface is connected between the processor and the memory and is configured to implement information interaction between the processor and the memory.

13. A computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the program implements the method according to any one of claims 1 to 6 or the method according to any one of claims 7 to 9.

Images