A Web Application Firewall Rule Matching Method Based on Hyperscan

Abstract

The present invention discloses a Web application firewall rule matching method based on Hyperscan, which mainly includes the following steps: step 1: an OpenResty server adopts a Hyperscan regular matching library to perform rule matching; step 2: a new HTTP API interface is added in the Lua module of OpenResty; step 3: an interface provided by Hyperscan is called to compile a regular expression in the latest configuration, and the interface is continued to be called to serialize the compiled result into a string and then write it into a shared memory; step 4: each worker updates the configuration to the current worker according to a semaphore in the shared memory when each request arrives; step 5: a deserialized database pointer is saved in the shared memory; step 6: a pointer to an allocated scratch memory is saved in the shared memory; step 7: after completing the above steps, each worker takes out a database pointer and a scratch pointer from the shared memory, and then calls a scan interface provided by Hyperscan to scan and obtain a matching result; step 8: each worker determines whether to release or intercept the current request according to the matching result.

Description

Technical Field

[0001] The present invention relates to the fields of information security technology and computer technology, and in particular to a Web application firewall rule matching method based on Hyperscan. Background Art

[0002] A WAF, short for Web Application Firewall, is a specialized application firewall designed to protect web applications from various attacks, such as cross-site scripting (XSS), SQL injection, and cross-site request forgery (CSRF).

[0003] WAF works by establishing a protection layer between the application and the Internet, inspecting and filtering all incoming and outgoing data. It can identify and block malicious data packets that may exploit application vulnerabilities to launch attacks.

[0004] Currently, most WAFs on the market use regular expression matching to match rules, and often use Hyperscan to improve the efficiency of rule matching.

[0005] Hyperscan is an efficient regular expression matching library that can handle multiple regular expressions and large amounts of data. Hyperscan uses a complex finite state machine (FSM) and multi-pattern matching algorithm to match multiple patterns in a single scan.

[0006] A complete Hyperscan matching process includes compilation (compiling regular expressions), allocating scratch space, and runtime (scanning and reporting matches). Traditional solutions store the compiled regular expression results in a Redis database, then retrieve them for each incoming request, allocate scratch space, and perform scanning. Since each request requires reading Redis and allocating scratch space, and scratch memory allocation isn't always efficient, this significantly reduces the WAF's matching efficiency and limits its QPS. Summary of the Invention

[0007] The purpose of this section is to summarize some aspects of the embodiments of the present invention and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of this application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions should not be used to limit the scope of the present invention.

[0008] To solve the above technical problems, the present invention provides the following technical solution: a Web application firewall rule matching method based on Hyperscan, mainly comprising:

[0009] Step 1: The OpenResty server uses the Hyperscan regular expression matching library to perform rule matching;

[0010] Step 2: Add an HTTP API interface to the OpenResty Lua module;

[0011] Step 3: After writing the latest configuration into shared memory, call the interface provided by Hyperscan to compile the regular expression in the latest configuration, and continue to call the interface to serialize the compiled result into a string and write it into shared memory;

[0012] Step 4: Each worker updates the configuration to the current worker based on the semaphore in the shared memory each time a request arrives;

[0013] Step 5: Each worker takes out the serialized string written to the shared memory in step 3, calls the Hyperscan deserialize_db interface to deserialize it, and saves the deserialized databse pointer in the shared memory;

[0014] Step 6: Each worker allocates scratch memory after deserialization and saves the pointer of the allocated scratch memory in shared memory;

[0015] Step 7: After completing the above steps, each worker takes out the database pointer and scrach pointer from the shared memory, and then calls the scan interface provided by Hyperscan to scan and obtain the matching results;

[0016] Step 8: Each worker decides whether to release or intercept the request based on the matching results.

[0017] As a preferred solution of the Hyperscan-based Web Application Firewall rule matching method of the present invention, wherein: the WAF at least includes an OpenResty server, an Agent configuration synchronization component, and a management system platform;

[0018] The Agent configuration synchronization component and management system platform can be any configuration synchronization system;

[0019] The OpenResty server uses the Hyperscan regular expression matching library for rule matching;

[0020] The OpenResty server can be any nginx-based system that supports Lua plugins;

[0021] The Agent component is responsible for calling the OpenResty API interface to synchronize the latest configuration to OpenResty when the configuration changes.

[0022] As a preferred solution of the Hyperscan-based Web application firewall rule matching method of the present invention, the hardware part of the OpenResty server includes a processor, a storage device, input and output interfaces, a communication interface, and a bus;

[0023] The processor adopts a central processing unit;

[0024] The storage device is a random access memory, SSD, or HDD;

[0025] Input and output interfaces are used to connect input / output modules to implement information input and output. Input / output modules can be configured as components in the device or externally connected to the device to provide corresponding functions. Input devices include keyboards, mice, and various sensors, while output devices include displays, speakers, and indicator lights.

[0026] The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices, wherein the communication module can realize communication through wired mode;

[0027] A bus consists of a pathway that carries information between the various components of a device.

[0028] As a preferred solution of the Hyperscan-based Web application firewall rule matching method described in the present invention, the Hyperscan matching process is a process from compiling regular expressions to searching for matching patterns in input data. A complete Hyperscan matching process includes compilation, allocating scratch space, and runtime, and the runtime includes but is not limited to a worker process.

[0029] As a preferred solution of the Hyperscan-based Web application firewall rule matching method described in the present invention, in step 2, a new HTTP API interface is added to the Lua module of OpenResty to facilitate the Agent to synchronize the latest configuration to the WAF. When the API interface determines whether the configuration is added, deleted, or modified, the current configuration is written to the shared memory of OpenResty, and a semaphore is written in the shared memory to facilitate other workers to synchronize the latest configuration.

[0030] As a preferred solution of the Hyperscan-based Web application firewall rule matching method described in the present invention, in step 4, since the address space of each worker process of OpenResty is independent of each other, the solution adopted is to compile on one worker, serialize the compiled rule base into a string and save it in shared memory, and then other workers retrieve the string from the shared memory and deserialize it for use.

[0031] As a preferred solution of the Hyperscan-based Web application firewall rule matching method described in the present invention, in step 4, each worker first obtains a semaphore from the shared memory each time a request arrives to determine whether the configuration needs to be updated. If the configuration needs to be updated, the latest configuration is first obtained from the shared memory and replaces the configuration in the current worker's independent memory.

[0032] As a preferred solution of the Hyperscan-based Web application firewall rule matching method described in the present invention, in step 5, since the memory space of each worker is independent and the database pointer of each worker is different, the key in the shared memory is named according to db_[worker_id].

[0033] As a preferred solution of the Hyperscan-based Web application firewall rule matching method described in the present invention, in step 6, since the memory space of each worker is independent and the scratch pointer of each worker is different, the key in the shared memory is named according to scr_[worker_id].

[0034] As a preferred solution of the Hyperscan-based Web application firewall rule matching method of the present invention, in step six, each worker allocates scratch memory by calling the make_scratch interface of Hyperscan after deserialization.

[0035] Beneficial effects of the present invention:

[0036] 1. Compared with the traditional solution, in each request, the serialized string of the rule base is not obtained from the Redis database. Instead, the serialized string can be obtained from the shared memory with a faster reading speed.

[0037] 2. Compared with traditional solutions, this solution allocates scratch memory in advance after deserialization and saves the database pointer and scratch pointer in shared memory, avoiding deserialization and scratch memory allocation in each request. This improves the WAF regular matching efficiency and effectively enhances the WAF processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0039] Figure 1 The present invention is a flowchart of a Web application firewall rule matching method based on Hyperscan.

[0040] Figure 2 This is a hardware block diagram of the OpenResty server of the present invention.

[0041] Figure 3 This is a structural block diagram of the firewall rule matching system in Example 1. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0043] In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below.

[0044] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive of other embodiments.

[0045] Furthermore, the present invention is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of the present invention, cross-sectional views illustrating device structures may be partially enlarged and not to scale. Furthermore, the schematic diagrams are merely illustrative and should not limit the scope of protection of the present invention. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0046] Example 1

[0047] Reference Figure 1-3 , is an embodiment of the present invention, which provides a Web application firewall rule matching method based on Hyperscan, such as Figure 1 , mainly including:

[0048] Step 1: The OpenResty server uses the Hyperscan regular expression matching library to perform rule matching.

[0049] Step 2: Add an HTTP API interface to the Lua module of OpenResty to facilitate the Agent to synchronize the latest configuration to the WAF. This API interface determines whether the configuration is added, deleted, or modified. It first updates the configuration to the current worker process, then writes the configuration to the shared memory, and then applies for a shared dictionary in the shared memory with the key [worker_id] and the value true to facilitate subsequent synchronization of the latest configuration by other workers. Shared memory is a special memory storage that allows multiple worker processes to share data. Since OpenResty is a multi-process model rather than a multi-threaded model, different worker processes cannot share state or memory by default. However, using shared memory, these processes can share information in this pre-allocated memory area.

[0050] Step 3: After writing the latest configuration to shared memory, call the compile interface provided by Hyperscan to compile the regular expressions in the latest configuration. Then, call the serialize_db interface provided by Hyperscan to serialize the compiled result into a string and write it to shared memory. Because the address space of each worker process in OpenResty is independent, the solution adopted in this paper is to compile on one worker, serialize the compiled rule base into a string, and save it in shared memory. Other workers then retrieve the string from shared memory and deserialize it for use.

[0051] Step 4: Each worker retrieves a semaphore from shared memory upon each incoming request to determine whether a configuration update is required. If so, it retrieves the latest configuration from shared memory and replaces the configuration in the current worker's independent memory. That is, each worker retrieves the corresponding value from shared memory based on the worker_id upon each incoming request. If the value is true, a configuration update is required. The latest configuration is retrieved from shared memory and updated to the current worker process.

[0052] Step 5: Each worker takes out the serialized string written to the shared memory in step 3, calls the Hyperscan deserialize_db interface to deserialize it, and saves the deserialized databse pointer in the shared memory;

[0053] Since each worker's memory space is independent and each worker's database pointer is different, the key in the shared memory is named according to db_[worker_id].

[0054] Step 6: After deserialization, each worker allocates scratch memory by calling Hyperscan's make_scratch interface and saves the pointer to the allocated scratch memory in shared memory. Since each worker's memory space is independent, the scratch pointer of each worker is different, so the key in the shared memory is named according to scr_[worker_id].

[0055] Step 7: After completing the above steps, each worker takes out the database pointer and scrach pointer from the shared memory, and then calls the scan interface provided by Hyperscan to scan and obtain the matching results;

[0056] Step 8: Each worker decides whether to release or intercept the request based on the matching results.

[0057] Specifically, the WAF includes at least an OpenResty server, an Agent configuration synchronization component, and a management system platform, wherein the Agent configuration synchronization component and the management system platform can be any configuration synchronization system;

[0058] The Agent configuration synchronization component and management system platform can be any configuration synchronization system;

[0059] The OpenResty server uses the Hyperscan regular expression matching library for rule matching. The Hyperscan matching process is a process from compiling regular expressions to searching for matching patterns in input data. A complete Hyperscan matching process includes compilation (compiling regular expressions), allocating scratch space, and runtime (scanning and reporting matches). The runtime includes but is not limited to a worker process.

[0060] The OpenResty server can be any nginx-based system that supports Lua plug-ins, and the present invention does not limit this;

[0061] The Agent component is responsible for calling the OpenResty API interface to synchronize the latest configuration to OpenResty when the configuration changes.

[0062] Furthermore, Figure 2 , the hardware part of the OpenResty server may include a processor, a storage device, an input and output interface, a communication interface, and a bus;

[0063] The processor may be a general-purpose central processing unit for executing relevant programs to implement the technical solutions provided in the embodiments of this specification;

[0064] The storage device can be implemented in the form of RAM (Random Access Memory), SSD (Solid State Drive), HDD (Hard Disk Drive), etc.

[0065] Input and output interfaces are used to connect input / output modules to implement information input and output. Input / output modules can be configured as components in the device or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, various sensors, etc. Output devices may include displays, speakers, indicator lights, etc.

[0066] The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices, where the communication module can realize communication through wired means (such as USB, network cable, etc.);

[0067] The bus comprises a pathway that transmits information between various components of a device, such as processors, memory devices, input / output interfaces, and communication interfaces.

[0068] It should be noted that although the above device only shows a processor, a storage device, an input / output interface, a communication interface, and a bus, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all components.

[0069] like Figure 3 The present invention also provides a Hyperscan-based Web application firewall rule matching system, which is applied to the above method and includes:

[0070] OpenResty module: Inspects and filters all incoming and outgoing data. It can identify and block malicious packets that may exploit application vulnerabilities.

[0071] Agent module: responsible for synchronizing the latest configuration to the OpenResty module through the API interface in real time;

[0072] Management platform module: responsible for sending the latest configuration items on the web page to the Agent module;

[0073] Hyperscan regular expression matching module: responsible for regular expression matching of WAF protection rules and returning matching results.

[0074] In summary, compared to traditional solutions, the serialized string of the rule base is not retrieved from the Redis database in each request. Instead, the serialized string is retrieved from shared memory, which has a faster read speed. This is because shared memory in Openresty is designed as a high-speed data sharing mechanism between worker processes. It allows data to be stored and retrieved between worker processes without the need for inter-process communication (IPC). Compared to traditional solutions, scratch memory is allocated in advance after deserialization and the database pointer and scratch pointer are saved in shared memory, avoiding deserialization and scratch memory allocation for each request. This significantly improves WAF processing speed. Memory allocation and deallocation are relatively expensive operations in themselves. If performed for every request, the overhead can become very large. In addition, frequent memory allocation and deallocation can lead to memory fragmentation, further degrading performance.

[0075] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A Web application firewall rule matching method based on Hyperscan, characterized in that: The following steps are involved: Step 1: The OpenResty server uses the Hyperscan regular expression matching library to perform rule matching; Step 2: Add an HTTP API interface to the OpenResty Lua module to facilitate the Agent to synchronize the latest configuration to the WAF; The WAF includes at least an OpenResty server, an Agent configuration synchronization component, and a management system platform; The Agent configuration synchronization component and management system platform can be any configuration synchronization system; The OpenResty server uses the Hyperscan regular expression matching library for rule matching; The OpenResty server can be any nginx-based system that supports Lua plugins; The Agent component is responsible for calling the OpenResty API interface to synchronize the latest configuration to OpenResty when the configuration changes; Step 3: After writing the latest configuration into shared memory, call the interface provided by Hyperscan to compile the regular expression in the latest configuration, and continue to call the interface to serialize the compiled result into a string and write it into shared memory; Step 4: Each worker updates the configuration to the current worker based on the semaphore in the shared memory each time a request arrives. Specifically, each worker first obtains the semaphore from the shared memory to determine whether a configuration update is required. If a configuration update is required, the worker first obtains the latest configuration from the shared memory and replaces the configuration in the current worker's independent memory. Step 5: Each worker takes out the serialized string written to the shared memory in step 3, then calls Hyperscan's deserialize_db interface to deserialize it and saves the deserialized database pointer in the shared memory. Since each worker's memory space is independent and the database pointer of each worker is different, the key in the shared memory is named according to db_[worker_id]; Step 6: After deserialization, each worker allocates scratch memory by calling Hyperscan's make_scratch interface and saves the pointer of the allocated scratch memory in the shared memory. Since the memory space of each worker is independent, the scratch pointer of each worker is different, so the key in the shared memory is named according to scr_[worker_id]; Step 7: After completing the above steps, each worker takes out the database pointer and scrach pointer from the shared memory, and then calls the scan interface provided by Hyperscan to scan and obtain the matching results; Step 8: Each worker decides whether to release or intercept the request based on the matching results.

2. The Hyperscan-based Web application firewall rule matching method according to claim 1, wherein: The hardware of the OpenResty server includes a processor, storage devices, input and output interfaces, communication interfaces, and a bus; The processor adopts a central processing unit; The storage device is a random access memory, SSD, or HDD; Input and output interfaces are used to connect input / output modules to implement information input and output. Input / output modules can be configured as components in the device or externally connected to the device to provide corresponding functions. Input devices include keyboards, mice, and various sensors, while output devices include displays, speakers, and indicator lights. The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices, wherein the communication module can realize communication through wired mode; A bus consists of a pathway that carries information between the various components of a device.

3. The Hyperscan-based Web application firewall rule matching method according to claim 1, wherein: The Hyperscan matching process is a process from compiling regular expressions to searching for matching patterns in input data. A complete Hyperscan matching process includes compilation, allocating scratch space, and runtime. The runtime includes but is not limited to a worker process.

4. The Hyperscan-based Web application firewall rule matching method according to claim 1, wherein: In step 2, if the configuration is added, deleted, or modified in the API interface, the current configuration is written to the shared memory of OpenResty, and a semaphore is written to the shared memory to facilitate other workers to synchronize the latest configuration.

5. The Hyperscan-based Web application firewall rule matching method according to claim 1, wherein: In step 4, since the address space of each worker process of OpenResty is independent of each other, the solution adopted is to compile on one worker, serialize the compiled rule base into a string and save it in shared memory, and then other workers take out the string from the shared memory and deserialize it for use.

Images