Hyperscan-based web application firewall rule matching method

Abstract

Disclosed in the present application is a Hyperscan-based Web application firewall rule matching method, mainly comprising the following steps: step 1, an OpenResty server using a Hyperscan regular expression match library for rule matching; step 2, adding an HTTP API interface into a Lua module of the OpenResty; step 3, invoking an interface provided by the Hyperscan to compile regular expressions in the latest configuration, and continuing to invoke the interface to serialize a compiling result into a character string and then write same into a shared memory; step 4, each time when a request arrives, each worker updating the configuration to the current worker on the basis of a semaphore in the shared memory; step 5, storing a deserialized database pointer in the shared memory; step 6, storing in the shared memory a pointer of an allocated scratch memory; step 7, after completing the above steps, each worker retrieving the database pointer and the scratch pointer from the shared memory, and then invoking a scan interface provided by the Hyperscan for scanning and acquiring a matching result; and step 8, on the basis of the matching result, each worker determining whether to release or intercept the current request.

Description

Web application firewall rule matching method based on Hyperscan

[0001] Related applications

[0002] This application claims priority to Chinese patent application number 202311707378.0, filed on December 13, 2023, entitled “A Web Application Firewall Rule Matching Method Based on Hyperscan,” the entire text of which is hereby incorporated by reference. Technical Field

[0003] The present application relates to the fields of information security technology and computer technology, and in particular to a Web application firewall rule matching method based on Hyperscan. Background Art

[0004] A WAF, short for Web Application Firewall, is a specialized application firewall designed to protect web applications from various attacks, such as cross-site scripting (XSS), SQL injection, and cross-site request forgery (CSRF).

[0005] WAF works by establishing a protection layer between the application and the Internet, inspecting and filtering all incoming and outgoing data. It can identify and block malicious data packets that may exploit application vulnerabilities to launch attacks.

[0006] Currently, most WAFs on the market use regular expression matching to match rules, and often use Hyperscan to improve the efficiency of rule matching.

[0007] Hyperscan is an efficient regular expression matching library that can handle multiple regular expressions and large amounts of data. Hyperscan uses a complex finite state machine (FSM) and multi-pattern matching algorithm to match multiple patterns in a single scan.

[0008] A complete Hyperscan matching process includes compilation (compiling regular expressions), allocating scratch space, and runtime (scanning and reporting matches). Traditional solutions store the compiled regular expression results in a Redis database, then retrieve them for each incoming request, allocate scratch space, and perform scanning. Since each request requires reading Redis and allocating scratch space, and scratch memory allocation isn't always efficient, this significantly reduces the WAF's matching efficiency and limits its QPS. Summary of the Invention

[0009] The purpose of this section is to summarize some aspects of the embodiments of the present application and briefly introduce some preferred embodiments. Some simplifications or omissions may be made in this section and the abstract and title of the present application to avoid obscuring the purpose of this section, the abstract and the title of the invention, and such simplifications or omissions shall not be used to limit the scope of the present application.

[0010] To solve the above technical problems, this application provides the following technical solutions: a Hyperscan-based Web application firewall rule matching method, which mainly includes:

[0011] Step 1: The OpenResty server uses the Hyperscan regular expression matching library to perform rule matching;

[0012] Step 2: Add an HTTP API interface to the OpenResty Lua module;

[0013] Step 3: After writing the latest configuration into shared memory, call the interface provided by Hyperscan to compile the regular expression in the latest configuration, and continue to call the interface to serialize the compiled result into a string and write it into shared memory;

[0014] Step 4: Each worker updates the configuration to the current worker based on the semaphore in the shared memory each time a request arrives;

[0015] Step 5: Each worker takes out the serialized string written to the shared memory in step 3, calls the Hyperscan deserialize_db interface to deserialize it, and saves the deserialized databse pointer in the shared memory;

[0016] Step 6: Each worker allocates scratch memory after deserialization and saves the pointer of the allocated scratch memory in shared memory;

[0017] Step 7: After completing the above steps, each worker takes out the database pointer and scrach pointer from the shared memory, and then calls the scan interface provided by Hyperscan to scan and obtain the matching results;

[0018] Step 8: Each worker decides whether to release or intercept the request based on the matching results.

[0019] In some embodiments, the WAF includes at least an OpenResty server, an Agent configuration synchronization component, and a management system platform;

[0020] The Agent configuration synchronization component and management system platform can be any configuration synchronization system;

[0021] The OpenResty server uses the Hyperscan regular expression matching library for rule matching;

[0022] The OpenResty server can be any nginx-based system that supports Lua plugins;

[0023] The Agent component is responsible for calling the OpenResty API interface to synchronize the latest configuration to OpenResty when the configuration changes.

[0024] In some embodiments, the hardware components of the OpenResty server include a processor, a storage device, input and output interfaces, a communication interface, and a bus;

[0025] The processor adopts a central processing unit;

[0026] The storage device is a random access memory, SSD, or HDD;

[0027] Input and output interfaces are used to connect input / output modules to implement information input and output. Input / output modules can be configured as components in the device or externally connected to the device to provide corresponding functions. Input devices include keyboards, mice, and various sensors, while output devices include displays, speakers, and indicator lights.

[0028] The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices, wherein the communication module can realize communication through wired mode;

[0029] A bus consists of a pathway that carries information between the various components of a device.

[0030] In some embodiments, the Hyperscan matching process is a process from compiling a regular expression to searching for matching patterns in input data. A complete Hyperscan matching process includes compilation, allocating scratch space, and runtime, and the runtime includes but is not limited to a worker process.

[0031] In some embodiments, in step 2, an HTTP API interface is added to the Lua module of OpenResty to facilitate the Agent to synchronize the latest configuration to the WAF. When the API interface determines whether the configuration is added, deleted, or modified, the current configuration is written to the shared memory of OpenResty, and a semaphore is written in the shared memory to facilitate other workers to synchronize the latest configuration.

[0032] In some embodiments, in step 4, since the address space of each worker process of OpenResty is independent of each other, the solution adopted is to compile on one worker, serialize the compiled rule base into a string and save it in the shared memory, and then other workers retrieve the string from the shared memory and deserialize it for use.

[0033] In some embodiments, in step 4, each worker first obtains a semaphore from the shared memory each time a request arrives to determine whether the configuration needs to be updated. If the configuration needs to be updated, the latest configuration is first obtained from the shared memory and replaces the configuration in the current worker's independent memory.

[0034] In some embodiments, in step 5, since the memory space of each worker is independent and the database pointer of each worker is different, the key in the shared memory is named according to db_[worker_id].

[0035] In some embodiments, in step 6, since the memory space of each worker is independent and the scratch pointer of each worker is different, the key in the shared memory is named according to scr_[worker_id].

[0036] In some embodiments, in step six, each worker allocates scratch memory by calling the make_scratch interface of Hyperscan after deserialization.

[0037] Beneficial effects of this application:

[0038] 1. Compared with the traditional solution, in each request, the serialized string of the rule base is not obtained from the Redis database. Instead, the serialized string can be obtained from the shared memory with a faster reading speed.

[0039] 2. Compared with traditional solutions, this solution allocates scratch memory in advance after deserialization and saves the database pointer and scratch pointer in shared memory, avoiding deserialization and scratch memory allocation in each request. This improves the WAF regular matching efficiency and effectively enhances the WAF processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] To more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. Those skilled in the art can also derive other drawings based on these drawings without inventive effort. Among them:

[0041] FIG1 is a flowchart of a Hyperscan-based Web application firewall rule matching method according to an embodiment of the present application.

[0042] FIG2 is a hardware block diagram of an OpenResty server according to an embodiment of the present application.

[0043] FIG3 is a structural block diagram of the firewall rule matching system in Example 1 of the present application. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the drawings in the specification.

[0045] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0046] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present application. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0047] Furthermore, this application is described in detail with reference to schematic diagrams. For ease of illustration, when describing the embodiments of this application, cross-sectional views of device structures may be partially enlarged and not to scale. Furthermore, these schematic diagrams are merely illustrative and should not limit the scope of protection of this application. Furthermore, in actual production, the three-dimensional dimensions of length, width, and depth should be included.

[0048] Example 1

[0049] 1 to 3 , the present application provides, in one embodiment, a Hyperscan-based Web application firewall rule matching method, as shown in FIG1 , which mainly includes the following steps 1 to 8.

[0050] Step 1: The OpenResty server uses the Hyperscan regular expression matching library to perform rule matching.

[0051] Step 2: Add an HTTP API interface to the Lua module of OpenResty to facilitate the Agent to synchronize the latest configuration to the WAF. This API interface determines whether the configuration is added, deleted, or modified. It first updates the configuration to the current worker process, then writes the configuration to the shared memory, and then applies for a shared dictionary in the shared memory with the key [worker_id] and the value true to facilitate subsequent synchronization of the latest configuration by other workers. Shared memory is a special memory storage that allows multiple worker processes to share data. Since OpenResty is a multi-process model rather than a multi-threaded model, different worker processes cannot share state or memory by default. However, using shared memory, these processes can share information in this pre-allocated memory area.

[0052] Step 3: After writing the latest configuration to shared memory, call the compile interface provided by Hyperscan to compile the regular expressions in the latest configuration. Then, call the serialize_db interface provided by Hyperscan to serialize the compiled result into a string and write it to shared memory. Because the address space of each worker process in OpenResty is independent, the solution adopted in this application is to compile on one worker, serialize the compiled rule base into a string, and save it in shared memory. Other workers then retrieve the string from shared memory and deserialize it for use.

[0053] Step 4: Each worker retrieves a semaphore from shared memory upon each incoming request to determine whether a configuration update is required. If so, it retrieves the latest configuration from shared memory and replaces the configuration in the current worker's independent memory. That is, each worker retrieves the corresponding value from shared memory based on the worker_id upon each incoming request. If the value is true, a configuration update is required. The latest configuration is retrieved from shared memory and updated to the current worker process.

[0054] Step 5: Each worker takes out the serialized string written to the shared memory in step 3, calls the Hyperscan deserialize_db interface to deserialize it, and saves the deserialized databse pointer in the shared memory;

[0055] Since each worker's memory space is independent and each worker's database pointer is different, the key in the shared memory is named according to db_[worker_id].

[0056] Step 6: After deserialization, each worker allocates scratch memory by calling Hyperscan's make_scratch interface and saves the pointer to the allocated scratch memory in shared memory. Since each worker's memory space is independent, the scratch pointer of each worker is different, so the key in the shared memory is named according to scr_[worker_id].

[0057] Step 7: After completing the above steps, each worker takes out the database pointer and scrach pointer from the shared memory, and then calls the scan interface provided by Hyperscan to scan and obtain the matching results;

[0058] Step 8: Each worker decides whether to release or intercept the request based on the matching results.

[0059] Specifically, the WAF includes at least an OpenResty server, an Agent configuration synchronization component, and a management system platform, wherein the Agent configuration synchronization component and the management system platform can be any configuration synchronization system;

[0060] The Agent configuration synchronization component and management system platform can be any configuration synchronization system;

[0061] The OpenResty server uses the Hyperscan regular expression matching library for rule matching. The Hyperscan matching process is a process from compiling regular expressions to searching for matching patterns in input data. A complete Hyperscan matching process includes compilation (compiling regular expressions), allocating scratch space, and runtime (scanning and reporting matches). The runtime includes but is not limited to a worker process.

[0062] The OpenResty server can be any nginx-based system that supports Lua plug-ins, and this application does not limit this;

[0063] The Agent component is responsible for calling the OpenResty API interface to synchronize the latest configuration to OpenResty when the configuration changes.

[0064] Furthermore, as shown in FIG2 , the hardware portion of the OpenResty server may include a processor, a storage device, input and output interfaces, a communication interface, and a bus;

[0065] The processor may be a general-purpose central processing unit for executing relevant programs to implement the technical solutions provided in the embodiments of this specification;

[0066] The storage device can be implemented in the form of random access memory (RAM), SSD (Solid State Drive), HDD (Hard Disk Drive), etc.

[0067] Input and output interfaces are used to connect input / output modules to implement information input and output. Input / output modules can be configured as components in the device or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, various sensors, etc. Output devices may include displays, speakers, indicator lights, etc.

[0068] The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices, where the communication module can realize communication through wired means (such as USB, network cable, etc.);

[0069] The bus comprises a pathway that transmits information between various components of a device, such as processors, memory devices, input / output interfaces, and communication interfaces.

[0070] It should be noted that although the above device only shows a processor, a storage device, an input / output interface, a communication interface, and a bus, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all components.

[0071] As shown in FIG3 , the present application also provides a Hyperscan-based Web application firewall rule matching system, which is applied to the above method and includes:

[0072] OpenResty module: Inspects and filters all incoming and outgoing data. It can identify and block malicious packets that may exploit application vulnerabilities.

[0073] Agent module: responsible for synchronizing the latest configuration to the OpenResty module through the API interface in real time;

[0074] Management platform module: responsible for sending the latest configuration items on the web page to the Agent module;

[0075] Hyperscan regular expression matching module: responsible for regular expression matching of WAF protection rules and returning matching results.

[0076] In summary, compared to traditional solutions, the serialized string of the rule base is not retrieved from the Redis database in each request. Instead, the serialized string is retrieved from shared memory, which has a faster read speed. This is because shared memory in Openresty is designed as a high-speed data sharing mechanism between worker processes. It allows data to be stored and retrieved between worker processes without the need for inter-process communication (IPC). Compared to traditional solutions, scratch memory is allocated in advance after deserialization and the database pointer and scratch pointer are saved in shared memory, avoiding deserialization and scratch memory allocation for each request. This significantly improves WAF processing speed. Memory allocation and deallocation are relatively expensive operations in themselves. If performed for every request, the overhead can become very large. In addition, frequent memory allocation and deallocation can lead to memory fragmentation, further degrading performance.

[0077] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include at least one of non-volatile and volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0078] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A Web application firewall rule matching method based on Hyperscan, comprising: Step 1: The OpenResty server uses the Hyperscan regular expression matching library to perform rule matching; Step 2: Add a new HTTP API interface in the Lua module of OpenResty; Step 3: After writing the latest configuration into the shared memory, call the interface provided by Hyperscan to compile the regular expression in the latest configuration, and continue to call the interface to serialize the compilation result into a string and write it into the shared memory; Step 4: Each worker updates the configuration to the current worker according to the semaphore in the shared memory each time a request arrives; Step 5: Each worker takes out the serialized string written to the shared memory in step 3, then calls the deserialize_db interface of Hyperscan to deserialize it, and saves the deserialized databse pointer in the shared memory; Step 6: Each worker allocates scratch memory after deserialization and saves the pointer of the allocated scratch memory in the shared memory; Step 7: After completing the above steps, each worker takes out the database pointer and the scrach pointer from the shared memory, and then calls the scan interface provided by Hyperscan to scan and obtain the matching results; Step 8: Each worker decides whether to release or intercept the request based on the matching results.

2. The method according to claim 1, wherein the WAF comprises at least an OpenResty server, an Agent configuration synchronization component, and a management system platform; The Agent configuration synchronization component and management system platform can be any configuration synchronization system; The OpenResty server uses the Hyperscan regular matching library to perform rule matching; The OpenResty server can be any nginx-based system that supports Lua plug-ins; The Agent component is responsible for calling the OpenResty API interface to synchronize the latest configuration to OpenResty when the configuration changes.

3. The method according to claim 2, wherein the hardware part of the OpenResty server includes a processor, a storage device, an input and output interface, a communication interface, and a bus; The processor adopts a central processing unit; The storage device is one of a random access memory, an SSD, and a HDD; The input and output interfaces are used to connect input / output modules to realize information input and output. The input / output modules can be configured in the device as components or externally connected to the device to provide corresponding functions. Input devices include keyboards, mice and various sensors, and output devices include displays, speakers and indicator lights. The communication interface is used to connect the communication module to realize the communication interaction between the device and other devices, wherein the communication module can realize communication through wired mode; A bus consists of a pathway that transfers information between the various components of a device.

4. The method according to claim 2, wherein the matching process of Hyperscan is a process from compiling a regular expression to searching for a matching pattern in input data, and a complete matching process of Hyperscan includes a compile period, allocating scratch space, and a run period, and the run period includes but is not limited to a worker process.

5. According to the method of claim 1, in the step 2, a new HTTP API interface is added in the Lua module of OpenResty to facilitate the Agent to synchronize the latest configuration to the WAF, and if it is determined in the API interface that the configuration is added, deleted, or modified, the current configuration is written into the shared memory of OpenResty, and a semaphore is written in the shared memory to facilitate other workers to synchronize the latest configuration.

6. According to the method described in claim 1, in step 4, since the address space of each worker process of OpenResty is independent of each other, the solution adopted is to compile on one worker, and serialize the compiled rule base into a string and save it in the shared memory, and then other workers take out the string from the shared memory and deserialize it for use.

7. According to the method described in claim 1, in step 4, each worker first obtains the semaphore from the shared memory to determine whether the configuration needs to be updated each time a request arrives. If the configuration needs to be updated, the latest configuration is first obtained from the shared memory and replaces the configuration in the current worker's independent memory.

8. The method according to claim 1, wherein in step 5, since the memory space of each worker is independent and the database pointer of each worker is different, the key in the shared memory is named according to db_[worker_id].

9. The method according to claim 1, wherein in step 6, since the memory space of each worker is independent and the scratch pointer of each worker is different, the key in the shared memory is named according to scr_[worker_id].

10. The method according to claim 1, wherein in step 6, each worker allocates scratch memory by calling the make_scratch interface of Hyperscan after deserialization.

Images