A WAF rule evaluation method based on regular parse tree

Abstract

The application discloses a WAF rule evaluation method based on a regular analysis tree, relates to the technical field of WAF rule evaluation, and collects regular expression protection rules of an open source WAF as seeds, analyzes and constructs an expression library based on a regular analysis tree, then performs SQL syntax word recognition on a sent test expression, matches all related expressions in the rule expression library through SQL syntax words, traverses the sent expression request, judges whether the target site WAF protection rule is bypassed according to site feedback information, and further evaluates the rule protection capability of the target WAF system. The application can quickly and accurately deduce the target site WAF protection rule, and can reliably and accurately evaluate the protection capability of the target site WAF system.

Description

Technical Field

[0001] This invention relates to the field of WAF rule evaluation technology, and specifically to a WAF rule evaluation method based on regular expression parse trees. Background Technology

[0002] WAF stands for Web Application Firewall. It is the first line of defense in network security and is widely used in many types of websites, including e-commerce sites, corporate portals, blogs, and forums. WAF systems resist external hacker intrusions, attacks, and penetrations, thereby preventing security incidents such as the theft of trade secrets and the leakage of user data.

[0003] A Web Application Firewall (WAF) system can inspect and verify the content of various requests from web application clients, ensuring their security and legitimacy. It blocks illegal requests in real time, providing protection for web applications. Also known as an application firewall, it is a crucial component of a layered network security defense system. WAFs are detection-based and corrective defense control measures. WAFs are categorized into hardware WAFs, software WAFs (ModSecurity), and code-level WAFs. WAFs identify malicious behavior through rule matching and behavioral analysis of request content and execute relevant actions, including blocking, logging, and alerting.

[0004] A Web Application Firewall (WAF) operates before the web server, detecting and identifying HTTP-based communications. In simple terms, a WAF is similar to security checks at a subway station, performing rapid security checks on HTTP requests. By parsing HTTP data, it makes judgments on different fields based on characteristics, rules, and other dimensions, using the results to determine whether to block or allow access.

[0005] Whether it's a hardware WAF, software WAF, cloud WAF, or a website's built-in WAF, they all have a rule base. Generally, the WAF rules in a WAF's rule engine are written using regular expressions, and some are based on semantic rules. This invention targets WAF rules written using regular expressions.

[0006] After setting up the website's WAF rules, you need to use a WAF evaluation tool to assess the protection capabilities of the WAF rules and determine their effectiveness.

[0007] In existing technologies, sqlmap tests the effectiveness of WAF rules from an attacker's perspective. sqlmap is an open-source penetration testing tool that can be used for automated detection, exploiting SQL injection vulnerabilities to gain access to the database server. Using sqlmap to perform SQL injection detection on a WAF system determines the system's protection capabilities.

[0008] WhatWAF and Wafw00f start with security vendor fingerprints, extracting the characteristics of different WAF rule vendors by parsing information in HTTP headers, comparing them with the WAF rules published by the vendor, and further inferring the protection capabilities of the WAF system.

[0009] However, existing technologies have the following drawbacks:

[0010] First, sqlmap is inefficient because it requires opening a new command-line window or ending the previous scan task for each site it checks. Although the -m parameter allows for batch scanning of URLs, the pattern is that a new scan task only starts after the previous one has ended, resulting in slow execution.

[0011] Second, Wafw00f and WhatWAF judge based on response characteristics. First, these characteristics can only detect which vendor's detection rule it belongs to, but cannot accurately identify the details of the rule. Moreover, they rely heavily on the built-in rule libraries of multiple WAF vendors. If a specific rule is not included, there will be false alarms or failure to identify the rule.

[0012] Based on this, we propose a WAF rule evaluation method based on regular parse trees. Summary of the Invention

[0013] The purpose of this invention is to overcome the above-mentioned problems in the prior art and provide a WAF rule evaluation method based on regular expression parsing trees, which can quickly and accurately infer the WAF protection rules of the target site, and at the same time reliably and accurately evaluate the protection capability of the WAF system of the target site.

[0014] To achieve the above-mentioned technical objectives and effects, the present invention is implemented through the following technical solution:

[0015] A WAF rule evaluation method based on regular expression parse trees includes the following steps:

[0016] Step 1: Collect regular expression protection rules from open-source WAFs, use them as seeds, and parse the seed rules based on the regular expression parsing tree to build an expression library;

[0017] Step 2: Then, the sent test expression is subjected to SQL syntax word recognition. By matching the SQL syntax words with all relevant expressions in the rule expression library, the expression requests are sent in a traversal manner and the site feedback information is used to determine whether the target site's WAF protection rules are bypassed, thereby further evaluating the rule protection capabilities of the target WAF system.

[0018] Preferably, the present invention further includes constructing a keyword recognition model to improve efficiency and make WAF rule evaluation more targeted; SQL syntax word recognition is performed before rule evaluation to reduce unnecessary scanning and probing; to identify specific WAF rules, using the actual rule as a seed, all expressions of the rule are detected through a regular expression parsing tree, and the expressions are sent as request parameters to the target site, and the specific WAF rule is inferred from the statistical feedback results; the specific implementation steps are as follows:

[0019] Methods for building a rule expression library:

[0020] Set up an open-source WAF rule collector: Collect as many open-source WAF rules as possible, using methods such as web crawlers, platform interfaces, and downloading from links;

[0021] Set up a rule parser: using open-source WAF rules as seeds, automatically decompose complex regular expressions into single rules through the regular expression parsing tree method;

[0022] Build a dedicated thesaurus: Store the SQL syntax library in a dedicated thesaurus, and also modify the deactivated thesaurus;

[0023] Set up the word segmenter: Import the dedicated dictionary and the stop dictionary, segment the WAF rules, set the segmentation interval to 1, and split the rules into multiple characters to form SQL phrases and regular expression phrases respectively.

[0024] Rule Unit Replacement: Replaces abbreviated rule units in a regular expression phrase with their corresponding characters;

[0025] Generate a rule expression repository: Extract SQL syntax terms and replace rule units from the decomposed individual rules to generate specific rule expressions, while saving the corresponding SQL syntax terms and the original unparsed rules.

[0026] Preferably, the specific steps for rule evaluation are as follows:

[0027] a. Select the test expression: Select the test expression from sqlmap or traffic monitoring device, which can be understood as the payload;

[0028] b. SQL syntax word recognition: Use a dedicated SQL syntax dictionary to match test expressions and extract SQL syntax words from the test expressions;

[0029] c. Expression Query: Use SQL syntax words to query the rule expression repository and match all relevant rule expressions;

[0030] d. Expression selection: Select the matching expression from the rule expression repository;

[0031] e. Send an expression request: Send an expression as a request parameter to the target site;

[0032] f. Feedback Judgment: Obtain the feedback result after sending the expression request, and determine whether there is an injection point based on the site changes in the feedback. If the feedback result is used to query the database, mark it as pass; otherwise, mark it as block.

[0033] g. Iterate through and send expression requests: Iterate through all matched expressions and send them one by one as request parameters to the target site;

[0034] h. Rule Evaluation: Counts the number of pass and block expressions to evaluate the WAF system protection capabilities of the target site. It can also infer the specific WAF rules of the site based on the original unparsed rules associated with the rule expression repository.

[0035] In summary, the present invention has at least one of the following beneficial effects:

[0036] This invention evaluates the protection capabilities of WAF systems. It collects open-source WAF rules, breaks down complex rules into individual rules using regular expression parsing trees, and then constructs a rule expression library using SQL syntax extraction and unit rule replacement methods. Based on this library, precise test expressions are sent to the target site. The site's feedback is used to mark "pass" and "block" expressions. After sending all relevant test expressions, the number of "pass" and "block" expressions is counted, further enabling the evaluation of the target site's WAF system's protection capabilities and the inference of specific rules. This invention can quickly and accurately infer the WAF protection rules of a target site, and reliably and accurately evaluate the protection capabilities of the target site's WAF system. Attached Figure Description

[0037] Figure 1 This is a flowchart illustrating how the method is implemented in this invention;

[0038] Figure 2 A flowchart for constructing the rule expression library of this invention. Detailed Implementation

[0039] The following is in conjunction with the appendix Figure 1-2 The present invention will be described in further detail below.

[0040] One embodiment of the present invention provides a WAF rule evaluation method based on regular expression parse trees, comprising the following steps:

[0041] Step 1: Collect regular expression protection rules from open-source WAFs, use them as seeds, and parse the seed rules based on the regular expression parsing tree to build an expression library;

[0042] Step 2: Then, perform SQL syntax word recognition on the sent test expression. Match all relevant expressions in the rule expression library using SQL syntax words. By iterating through the sent expression requests and judging from the site's feedback information, determine whether the target site's WAF protection rules are bypassed, and further evaluate the target WAF system's rule protection capabilities (the flowchart for implementing this method is shown below). Figure 1 (As shown).

[0043] To achieve more targeted WAF rule evaluation and improve efficiency, this invention constructs a keyword recognition model. Before rule evaluation, SQL syntax word recognition is performed to reduce unnecessary scanning and probing. To identify specific WAF rules, this invention uses actual rules as seeds, probing all expressions of the rule through a regular expression parse tree. These expressions are then sent as request parameters to the target site, and the feedback results are statistically analyzed to infer the specific WAF rule. The specific implementation steps are as follows:

[0044] Methods for building a rule expression library, such as Figure 2 As shown:

[0045] Set up an open-source WAF rule collector: Collect as many open-source WAF rules as possible, using methods such as web crawlers, platform interfaces, and downloading from links;

[0046] Set up a rule parser: using open-source WAF rules as seeds, automatically decompose complex regular expressions into single rules through the regular expression parsing tree method;

[0047] Building a dedicated dictionary: Most WAF protection rules are built based on SQL language keywords, so the SQL syntax library needs to be stored in a dedicated dictionary to improve the accuracy of word segmentation. At the same time, the stop word library also needs to be modified because WAF rules are generated based on regular expressions and contain some special characters: \b, \W, *, ?, etc. These special characters need to be added to the dedicated dictionary and removed from the stop word library.

[0048] Set up the word segmenter: Import the dedicated dictionary and the stop dictionary, segment the WAF rules, and set the segmentation interval to 1. Because the SQL language will intersperse database and field names, it is not continuous, so the segmentation interval is set to 1 to split the rules into multiple characters, which will form SQL phrases and regular expression phrases respectively.

[0049] Rule unit replacement: Replaces abbreviated rule units in a regular expression phrase with the corresponding characters, such as replacing \d with 1, \w with a, and .* with x;

[0050] Generate a rule expression repository: Extract SQL syntax terms and replace rule units from the decomposed individual rules to generate specific rule expressions, while saving the corresponding SQL syntax terms and the original unparsed rules.

[0051] Specific steps for rule evaluation:

[0052] a. Select the test expression: Choose the test expression from sqlmap or the traffic monitoring device. This can be understood as the payload.

[0053] For example, id=0||(updatexml(1,concat(0x7e,(select md5(43943)),0x7e),1));

[0054] b. SQL syntax word recognition: Use a dedicated SQL syntax dictionary to match test expressions and extract SQL syntax words from the test expressions, such as updatexml, concat, select;

[0055] c. Expression Query: Use SQL syntax words to query the regular expression repository and match all relevant regular expressions, such as 0updatexml 1concat a select x;

[0056] d. Expression selection: Select the matching expression from the rule expression repository;

[0057] e. Send an expression request: Send an expression as a request parameter to the target site;

[0058] f. Feedback Judgment: Obtain the feedback result after sending the expression request, and determine whether there is an injection point based on the site changes in the feedback. If the feedback result is used to query the database, mark it as pass; otherwise, mark it as block.

[0059] g. Iterate through and send expression requests: Iterate through all matched expressions and send them one by one as request parameters to the target site;

[0060] h. Rule Evaluation: Counts the number of pass and block expressions to evaluate the WAF system protection capabilities of the target site. It can also infer the specific WAF rules of the site based on the original unparsed rules associated with the rule expression repository.

[0061] This invention evaluates the protection capabilities of WAF systems. It collects open-source WAF rules, breaks down complex rules into individual rules using regular expression parsing trees, and then constructs a rule expression library using SQL syntax extraction and unit rule replacement methods. Based on this library, precise test expressions are sent to the target site. The site's feedback is used to mark "pass" and "block" expressions. After sending all relevant test expressions, the number of "pass" and "block" expressions is counted, further enabling the evaluation of the target site's WAF system's protection capabilities and the inference of specific rules. This invention can quickly and accurately infer the WAF protection rules of a target site, and reliably and accurately evaluate the protection capabilities of the target site's WAF system.

[0062] The WAF rule evaluation method based on regular expression parse trees proposed in this invention is not only applicable to the evaluation of WAF rules, but also applicable to the capability evaluation of other detection systems based on regular expressions.

[0063] The construction of the WAF rule expression library based on SQL syntax word extraction and unit rule replacement can also be applied to other rule-based detection systems.

[0064] The above are all preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Therefore, all equivalent changes made in accordance with the structure, shape and principle of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A WAF rule evaluation method based on regular parse trees, characterized in that: The following steps are included: Step 1: Collect regular expression protection rules from open-source WAFs, use them as seeds, and parse the seed rules based on the regular expression parsing tree to build an expression library. The method for building the rule expression library is as follows: Set up an open-source WAF rule collector: Collect open-source WAF rules through web crawlers, platform interfaces, and links. Set up a rule parser: using open-source WAF rules as seeds, automatically decompose complex regular expressions into single rules through the regular expression parsing tree method; Build a dedicated thesaurus: Store the SQL syntax library in a dedicated thesaurus, and also modify the deactivated thesaurus; Set up the word segmenter: Import the dedicated dictionary and the stop dictionary, segment the WAF rules, set the segmentation interval to 1, and split the rules into multiple characters to form SQL phrases and regular expression phrases respectively. Rule Unit Replacement: Replaces abbreviated rule units in a regular expression phrase with their corresponding characters; Generate a rule expression repository: The decomposed individual rules are processed by extracting SQL syntax terms and replacing rule units to generate specific rule expressions, while saving the corresponding SQL syntax terms and the original unparsed rules; Step 2: Then, the sent test expression is subjected to SQL syntax word recognition. By matching the SQL syntax words with all relevant expressions in the rule expression library, the expression requests are sent in a traversal manner and the site feedback information is used to determine whether the target site's WAF protection rules are bypassed, thereby further evaluating the rule protection capabilities of the target WAF system.

2. The WAF rule evaluation method based on regular parse trees according to claim 1, characterized in that: It also includes building a keyword recognition model, performing SQL syntax word recognition before rule evaluation; and using actual rules as seeds, probing all expressions of the rule through regular expression parsing trees, sending the expressions as request parameters to the target site, and statistically analyzing the feedback results to infer the specific WAF rule.

3. The WAF rule evaluation method based on regular parse trees according to claim 2, characterized in that: The specific steps for rule evaluation are as follows: a. Select the test expression: Select the test expression from sqlmap or traffic monitoring device, which can be understood as the payload; b. SQL syntax word recognition: Use a dedicated SQL syntax dictionary to match test expressions and extract SQL syntax words from the test expressions; c. Expression Query: Use SQL syntax words to query the rule expression repository and match all relevant rule expressions; d. Expression selection: Select the matching expression from the rule expression repository; e. Send an expression request: Send an expression as a request parameter to the target site; f. Feedback Judgment: Obtain the feedback result after sending the expression request, and determine whether there is an injection point based on the site changes in the feedback. If the feedback result is used to query the database, mark it as pass; otherwise, mark it as block. g. Iterate through and send expression requests: Iterate through all matched expressions and send them one by one as request parameters to the target site; h. Rule Evaluation: Counts the number of pass and block expressions to evaluate the WAF system protection capabilities of the target site. It can also infer the specific WAF rules of the site based on the original unparsed rules associated with the rule expression repository.

Images