Hyperscan-based read-write processing method and device
By managing the rules set, compiling status machine database and data matching operations in a separate thread, the Hyperscan's writing operation and data matching operation cannot be collaborated, and the efficient read and writing operation process is achieved.
Patent Information
- Application Number
- CN202411355684.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-26
- Publication Date
- 2025-01-21
Smart Images
Figure CN119336780A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a read-write processing method and device based on Hyperscan. Background Art
[0002] As an advanced network monitoring technology, Deep Packet Inspection (DPI) can deeply analyze the contents of network data packets in order to accurately identify, classify and control network traffic. As a high-performance regular expression matching library, Hyperscan is applied to pattern matching during DPI operation to accurately identify specific patterns or signatures in massive data packets, significantly improving the efficiency of pattern matching.
[0003] However, due to the limitations of Hyperscan's own characteristics, when writing Hyperscan (including adding, deleting or modifying pattern strings), the rule set is usually reconstructed first, then the database is recompiled, and the Scratch objects required for matching are rebuilt. However, during the operation of DPI, many pattern strings that need to be added dynamically may be encountered. For static rules, these rules can be loaded all at once during the system initialization phase, which will not affect the access of real-time readers; for real-time rules that need to be added dynamically during operation, they cannot be directly integrated into the existing database and Scratch objects. Instead, the rule set needs to be rebuilt together with the original rule set, and it can only take effect after compilation, which means that it is not possible to support the dynamic addition of new rules to the existing database and Scratch.
[0004] Therefore, when implementing high-speed read and write operations in a multi-threaded environment, how to ensure that Hyperscan's write operations (including the addition, deletion and modification of pattern strings) and the real-time reader's data matching operations can be seamlessly coordinated to avoid blocking between read and write operations is a major problem that needs to be solved urgently.
[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention
[0006] The embodiments of the present application provide a read / write processing method and device based on Hyperscan to at least solve the technical problem that the write operation of Hyperscan and the data matching operation performed by the user calling Hyperscan cannot be seamlessly coordinated.
[0007] According to one aspect of an embodiment of the present application, a read-write processing method based on Hyperscan is provided, including: in response to a write operation instruction for a first rule set, calling a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules; calling a second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set; obtaining data to be matched corresponding to the target regular expression rule in the second rule set; calling a third thread to perform a matching operation on the data to be matched in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0008] Optionally, before responding to the write operation instruction of the target object to the first rule set, the method also includes: obtaining at least one regular expression rule; performing compliance check on at least one regular expression rule respectively to obtain corresponding check results, and forming a first rule set from all regular expression rules with passed check results, wherein the compliance check includes at least one of the following: syntax check, logic check, security check; compiling the first rule set through a preset compilation interface and using a preset compilation mode to obtain a state machine database corresponding to the first rule set, wherein the compilation mode includes: block mode, stream mode or vector mode.
[0009] Optionally, in response to a write operation instruction for the first rule set, calling the first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, including: receiving a write operation instruction for the first rule set, wherein the write operation instruction carries at least a regular expression rule to be processed and a corresponding write operation type, and the write operation type includes at least one of the following: a new operation, a delete operation, and a modify operation; when the write operation type is a new operation, calling the first thread to add a new regular expression rule to the first rule set to obtain a second rule set; when the write operation type is a delete operation, calling the first thread to delete the regular expression rule in the first rule set to obtain the second rule set; when the write operation type is a modify operation, calling the first thread to modify the regular expression rule in the first rule set to obtain the second rule set.
[0010] Optionally, calling the first thread to add a regular expression rule to the first rule set to obtain a second rule set includes: determining whether the first rule set includes the regular expression rule; if the first rule set does not include the regular expression, calling the first thread to perform a compliance check on the regular expression rule to obtain a corresponding check result; when the check result is that the check passes, calling the first thread to combine the first rule set and the regular expression rule to obtain the second rule set.
[0011] Optionally, calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set includes: calling the second thread, compiling the second rule set through a preset compilation interface and using a preset compilation mode to obtain a state machine database corresponding to the second rule set.
[0012] Optionally, before calling at least one third thread to perform a matching operation on the data to be matched in the state machine database corresponding to the second rule set, the method also includes: replacing the state machine database pointer of Hyperscan from pointing to the state machine database corresponding to the first rule set to pointing to the state machine database corresponding to the second rule set, and releasing the state machine database corresponding to the first rule set using a preset destruction interface.
[0013] Optionally, after obtaining the corresponding matching result, the method further includes: calling a preset callback function to feed back the matching result.
[0014] According to another aspect of an embodiment of the present application, a read-write processing device based on Hyperscan is also provided, including: a management module, which is used to respond to a write operation instruction on a first rule set, call a first thread to perform a corresponding write operation on the first rule set, and obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules; a compilation module, which is used to call a second thread to compile the second rule set, and obtain a state machine database corresponding to the second rule set; an acquisition module, which is used to obtain data to be matched corresponding to the target regular expression rule in the second rule set; a data matching module, which is used to call a third thread to perform a matching operation on the data to be matched in the state machine database corresponding to the second rule set, and obtain a corresponding matching result.
[0015] According to another aspect of an embodiment of the present application, a non-volatile storage medium is further provided, which includes a stored computer program, wherein the device where the non-volatile storage medium is located executes the above-mentioned Hyperscan-based read and write processing method by running the computer program.
[0016] According to another aspect of an embodiment of the present application, a computer program product is further provided. The computer program product includes a stored computer program, wherein when the computer program is executed by a processor, the above-mentioned Hyperscan-based read and write processing method is implemented.
[0017] In an embodiment of the present application, in response to a write operation instruction for a first rule set, a first thread is called to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules; a second thread is called to compile the second rule set to obtain a state machine database corresponding to the second rule set; data to be matched corresponding to the target regular expression rule in the second rule set is obtained; a third thread is called to perform a matching operation on the data to be matched in the state machine database corresponding to the second rule set to obtain a corresponding matching result. By executing the management of the rule set, the compilation of the state machine database, and the data matching process in separate threads respectively, the influence of the switching between the new and old state machine databases on the data matching can be avoided. It can be seen that the technical solution of the embodiment of the present application can solve the technical problem that the write operation of Hyperscan and the user calling Hyperscan for data matching operations cannot achieve seamless coordination. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0019] Figure 1 It is a hardware structure block diagram of an optional computer terminal (or mobile device) for implementing a read-write processing method based on Hyperscan according to an embodiment of the present application;
[0020] Figure 2 is a flowchart of an optional Hyperscan-based read / write processing method according to an embodiment of the present application;
[0021] Figure 3 is a schematic diagram of pointer exchange between positive and secondary samples of an optional Hyperscan state machine according to an embodiment of the present application;
[0022] Figure 4 is a schematic structural diagram of an optional Hyperscan-based read / write processing device according to an embodiment of the present application;
[0023] Figure 5 It is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION
[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.
[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0026] In addition, the relevant information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for display, data for analysis, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. For example, an interface is set up between this system and the relevant user or organization. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.
[0027] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:
[0028] Deep Packet Inspection (DPI) technology is based on traditional IP packet inspection technology (mainly focusing on the analysis of packet header information from Layer 2 to Layer 4 of the OSI model), and further enhances the ability to deeply analyze application layer data. This includes the identification of application layer protocols, the detection of packet contents, and the deep decoding of packet payloads, thereby achieving more comprehensive and sophisticated monitoring and management of network traffic.
[0029] Hyperscan is a high-performance regular expression matching library developed by Intel, which is specially designed to efficiently search for patterns in large-scale data sets. It cleverly utilizes the parallel computing characteristics of the CPU (Central Processing Unit) and precompiles regular expressions into highly optimized machine code, thereby greatly accelerating the text search process and achieving high-speed text search.
[0030] Example 1
[0031] According to an embodiment of the present application, an embodiment of a read-write processing method based on Hyperscan is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0032] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 FIG. 1 shows a hardware structure block diagram of a computer terminal (or mobile device) for implementing a read-write processing method based on Hyperscan. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more (102a, 102b, ..., 102n are used to illustrate) processors 102 (the processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.
[0033] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).
[0034] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the read-write processing method based on Hyperscan in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the read-write processing method based on Hyperscan of the above-mentioned application is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0035] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.
[0036] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).
[0037] In the above operating environment, Figure 2 is a flowchart of an optional Hyperscan-based read and write processing method according to an embodiment of the present application, such as Figure 2 As shown, the method at least includes steps S202-S206, wherein:
[0038] Step S202: In response to a write operation instruction for the first rule set, calling the first thread to perform a corresponding write operation on the first rule set to obtain a second rule set.
[0039] In the technical solution provided in step S202, the above-mentioned first rule set includes at least one static regular expression rule, and the write operation generally includes: add, delete, and modify operations. Therefore, in response to the write operation instruction of the first rule set, the first thread is separately scheduled to add new regular expression rules, delete regular expression rules, or modify regular expression rules to the first rule set to obtain a second rule set. Therefore, the second rule set also includes at least one group of regular expression rules.
[0040] Step S204: calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set.
[0041] In the technical solution provided in step S204, since managing and compiling the rule set both require a lot of time, in order to improve efficiency and not affect the state machine database currently in use, the embodiment of the present application proposes to separately schedule a second thread to compile the second rule set to convert all regular expression rules in the second rule set into a state machine in a binary representation form, so that Hyperscan can use the pre-compiled state machine to process multiple data matches in parallel. At the same time, the state machine can also enable Hyperscan to support multiple scanning methods to adapt to different data streams and application scenarios.
[0042] Step S206: Acquire the to-be-matched data corresponding to the target regular expression rule in the second rule set.
[0043] In the technical solution provided in step S206, since the deep packet inspection process involves matching data packets with rules, the data to be matched in the embodiment of the present application can be obtained by extracting and analyzing the network traffic in the deep packet inspection process, and the data to be matched corresponds to at least one target regular expression rule in the second rule set.
[0044] Step S208: calling the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0045] In the technical solution provided in step S208, in order to make the management of the rule set, the compilation of the state machine data block, and the data matching process independent of each other and not affect each other, the third thread can be separately scheduled to perform matching operations on the data to be matched in the state machine database corresponding to the second rule set to obtain the corresponding matching results.
[0046] The above method of this embodiment is further introduced below.
[0047] As an optional implementation, in the technical solution provided in the above step S202, preloading may be performed according to the following method, including:
[0048] Step 1: Get at least one regular expression rule;
[0049] Step 2: Perform compliance check on at least one regular expression rule to obtain corresponding check results, and form a first rule set from all regular expression rules with check results of passing the check.
[0050] Among them, the above-mentioned compliance check includes at least one of the following: syntax check, logic check, security check, special character processing, etc., among which, syntax check is to check whether the syntax of the regular expression rule is correct, such as whether there are unclosed brackets, incorrect use of quantifiers, etc. in the regular expression rule; logic check is to check whether the regular expression rule can match the expected string pattern; security check is to check whether the regular expression rule has potential security risks; special character processing is to check whether the regular expression rule uses special characters correctly, such as ".", "*", "?", etc.
[0051] Step 3: Compile the first rule set through a preset compilation interface and using a preset compilation mode to obtain a state machine database corresponding to the first rule set.
[0052] The compilation interface is hs_compile, and the first rule set is compiled according to a preset compilation mode to obtain a state machine database corresponding to the first rule set, wherein the compilation mode includes at least one of the following: block mode (Streaming Mode), flow mode (Block Mode) and vector mode (Vectored Mode). Block mode (Block Mode) is the most basic mode, which is suitable for processing independent data blocks, that is, each time it is called, a complete data block will be matched, and the match is limited to the data block; while the streaming mode (Streaming Mode) is specifically related to network data flow, and is suitable for the situation where the data is split into multiple messages for transmission; and when the user has prepared all the input data, but the data is not continuous in the memory, but is scattered into multiple blocks, the vector mode (VectoredMode) is usually used, which treats the string group as several data streams in internal implementation, and calls the streaming mode in sequence to complete the matching work.
[0053] Therefore, the above steps can be understood as, in the preloading phase after the deep packet inspection program is started, all built-in static regular expression rules can be read in advance, and compliance checks can be performed on these regular expression rules, and the regular expression rules that meet the specifications can be stored in the corresponding data structure in sequence to obtain the corresponding state machine database.
[0054] As an optional implementation, in the technical solution provided in the above step S202, the second rule set may be obtained according to the following method, including:
[0055] First, a write operation instruction for a first rule set is received.
[0056] The write operation instruction at least carries the regular expression rule to be processed and the corresponding write operation type, and the write operation type includes at least one of the following: a new operation, a delete operation, and a modify operation.
[0057] Case 1: The write operation type is a new operation.
[0058] In the above case, the first thread may be called to add a regular expression rule to the first rule set to obtain a second rule set.
[0059] Specifically, in order to ensure the validity of the newly added regular expression rules, it is possible to first determine whether the first rule set includes regular expression rules; if the first rule set includes regular expressions, there is no need to add the regular expression rules to the first rule set; if the first rule set does not include regular expressions, the first thread can be called to perform compliance check on the regular expression rules to obtain the corresponding check results, and when the check result is that the check passes, the first thread is called to combine the first rule set and the regular expression rules to obtain the second rule set.
[0060] For example, the first rule set includes n regular expression rules, namely: Pattern 0 ( / foo.*bar / s), Pattern 1 ( / [af]{6,12} / i), ..., Pattern n ( / ^GET\s.*HTTP / m), and the regular expression rule to be added is Pattern n+1 ( / ^POST\s.*HTTP / n). Since the first rule set does not include Pattern n+1, and Pattern n+1 meets the requirements of the specification, the second rule set can be composed of Pattern 0 ( / foo.*bar / s), Pattern 1 ( / [af]{6,12} / i), ..., Pattern n ( / ^GET\s.*HTTP / m) and Pattern n+1 ( / ^POST\s.*HTTP / n).
[0061] The second case: the write operation type is a delete operation.
[0062] In the above case, the first thread may be called to delete the regular expression rules in the first rule set to obtain the second rule set.
[0063] The third case: the write operation type is a modification operation.
[0064] In the above case, the first thread may be called to modify the regular expression rules in the first rule set to obtain the second rule set.
[0065] As an optional implementation, in the technical solution provided in the above step S204, the method may include: calling a second thread, compiling the second rule set through a preset compilation interface and using a preset compilation mode, and obtaining a state machine database corresponding to the second rule set.
[0066] In this embodiment, the compilation interface is also hs_compile, which calls the second thread and compiles the second rule set according to a preset compilation mode to obtain a state machine database corresponding to the second rule set, wherein the compilation mode includes: block mode, stream mode or vector mode.
[0067] As an optional implementation, in the technical solution provided in the above step S206, after the compilation of the state machine database corresponding to the second rule set is completed, the state machine database pointer of Hyperscan can be replaced from pointing to the state machine database corresponding to the first rule set to pointing to the state machine database corresponding to the second rule set, and the state machine database corresponding to the first rule set can be released using the preset destruction interface.
[0068] In the above embodiment, if Figure 3 As shown, if the state machine database corresponding to the first rule set is recorded as the Hyperscan state machine positive sample (database), and the state machine database corresponding to the second rule set is recorded as the Hyperscan state machine secondary sample (database_new). The above steps can be understood as: exchanging the pointers of the Hyperscan state machine positive sample (database) and the Hyperscan state machine secondary sample (database_new), so that when the Hyperscan state machine database is accessed later, the Hyperscan state machine secondary sample (database_new) is directly accessed. Among them, since the pointer assignment is an atomic operation, it can be guaranteed that the data read by the read operation before and after this operation is one of the positive sample or the secondary sample, and no abnormal data will be read.
[0069] In addition, after the exchange is completed, the Hyperscan state machine database accessed by the data matching process will point to the state machine database corresponding to the second rule set. At this time, the destruction structure provided by Hyperscan can be used to release the state machine database corresponding to the first rule set.
[0070] As an optional implementation, after obtaining the corresponding matching result of the data to be matched, a preset callback function (Match Callback) can be called to feedback the matching result, wherein the callback function is set by the user when calling Hyperscan for data matching.
[0071] Furthermore, the Hyperscan-based read and write processing method provided in this application will be briefly described below through a specific example.
[0072] First, obtain the original string to be matched (i.e., the data to be matched) "47 45 54 20 73 61 6d 70 6c65 4669 6c 65 2e 68 74 6d 6c 20 48 54 54 50 2f 31 2e 31 0d 0a", and the target regular expression rule corresponding to the original string is "sampleFile.html HTTP / 1.1". The original N regular expression rules in the first rule set of Hyperscan and the target regular expression rule "sampleFile.html HTTP / 1.1" are not repeated.
[0073] Next, in response to the write operation instruction of the newly added target regular expression rule "sampleFile.html HTTP / 1.1" of the first rule set, and verifying the legality of the target regular expression rule within the grammar range supported by Hyperscan, if the target regular expression rule is legal, call an independent thread to combine the target regular expression rule "sampleFile.html HTTP / 1.1" and the first rule set of Hyperscan into a second rule set.
[0074] Then, the independent thread is called and the second rule set is compiled through the interface hs_compile_multi() provided by Hyperscan to obtain the corresponding state machine database_new. It should be noted that the compilation process is completed in an independent thread and will not affect the state machine database (compiled by the first rule set) currently running in Hyperscan.
[0075] Finally, the pointers of the state machine database_new and the state machine database are exchanged to call the independent thread to match the original string "47 45 54 20 73 61 6d 70 6c 65 46 69 6c 65 2e 68 74 6d 6c 20 48 54 54 50 2f31 2e 31 0d0a" with the target regular expression rule "sampleFile.html HTTP / 1.1" in the state machine database_new, obtain the corresponding matching result, and call the preset callback function to feed back the matching result to the target user.
[0076] Based on the scheme defined by the above steps S202 to S208, it can be known that, in an embodiment, the management of the rule set, the compilation of the state machine database, and the data matching process are executed in separate threads respectively to avoid the impact of the switching between the new and old state machine databases on the data matching. It can be seen that the technical solution of the embodiment of the present application can solve the technical problem that the write operation of Hyperscan and the user calling Hyperscan for data matching operations cannot achieve seamless coordination.
[0077] Example 2
[0078] Based on Embodiment 1 of the present application, an embodiment of a read / write processing device based on Hyperscan is also provided, and when the device is running, the read / write processing method based on Hyperscan of the above embodiment is executed. Figure 4 is a schematic diagram of the structure of an optional read-write processing device based on Hyperscan according to an embodiment of the present application, such as Figure 4 As shown, the read-write processing device based on Hyperscan at least includes a management module 42, a compilation module 44, an acquisition module 46 and a data matching module 48, wherein:
[0079] The management module 42 is configured to, in response to a write operation instruction on the first rule set, call the first thread to perform a corresponding write operation on the first rule set to obtain a second rule set.
[0080] Among them, the above-mentioned first rule set includes at least one static regular expression rule, and the write operation generally includes: add, delete, and modify operations. Therefore, in response to the write operation instruction of the first rule set, the first thread is separately scheduled to add new regular expression rules, delete regular expression rules or modify regular expression rules to the first rule set to obtain the second rule set. Therefore, the second rule set also includes at least one group of regular expression rules.
[0081] Optionally, the management module 42 may preload the first rule set and the corresponding state machine data block according to the following method, including:
[0082] Step 1: Get at least one regular expression rule;
[0083] Step 2: Perform compliance check on at least one regular expression rule to obtain corresponding check results, and form a first rule set from all regular expression rules with check results of passing the check.
[0084] Among them, the above-mentioned compliance check includes at least one of the following: syntax check, logic check, security check, special character processing, etc., among which, syntax check is to check whether the syntax of the regular expression rule is correct, such as whether there are unclosed brackets, incorrect use of quantifiers, etc. in the regular expression rule; logic check is to check whether the regular expression rule can match the expected string pattern; security check is to check whether the regular expression rule has potential security risks; special character processing is to check whether the regular expression rule uses special characters correctly, such as ".", "*", "?", etc.
[0085] Step 3: Compile the first rule set through a preset compilation interface and using a preset compilation mode to obtain a state machine database corresponding to the first rule set.
[0086] The compilation interface is hs_compile, and the first rule set is compiled according to a preset compilation mode to obtain a state machine database corresponding to the first rule set, wherein the compilation mode includes at least one of the following: block mode (Streaming Mode), flow mode (Block Mode) and vector mode (Vectored Mode). Block mode (Block Mode) is the most basic mode, which is suitable for processing independent data blocks, that is, each time it is called, a complete data block will be matched, and the match is limited to the data block; while the streaming mode (Streaming Mode) is specifically related to network data flow, and is suitable for the situation where the data is split into multiple messages for transmission; and when the user has prepared all the input data, but the data is not continuous in the memory, but is scattered into multiple blocks, the vector mode (VectoredMode) is usually used, which treats the string group as several data streams in internal implementation, and calls the streaming mode in sequence to complete the matching work.
[0087] As an optional implementation, the management module 42 may obtain the second rule set according to the following method, including:
[0088] First, a write operation instruction for a first rule set is received.
[0089] The write operation instruction at least carries the regular expression rule to be processed and the corresponding write operation type, and the write operation type includes at least one of the following: a new operation, a delete operation, and a modify operation.
[0090] Case 1: The write operation type is a new operation.
[0091] In the above case, the first thread may be called to add a regular expression rule to the first rule set to obtain a second rule set.
[0092] Specifically, in order to ensure the validity of the newly added regular expression rules, it is possible to first determine whether the first rule set includes regular expression rules; if the first rule set includes regular expressions, there is no need to add the regular expression rules to the first rule set; if the first rule set does not include regular expressions, the first thread can be called to perform compliance check on the regular expression rules to obtain the corresponding check results, and when the check result is that the check passes, the first thread is called to combine the first rule set and the regular expression rules to obtain the second rule set.
[0093] The second case: the write operation type is a delete operation.
[0094] In the above case, the first thread may be called to delete the regular expression rules in the first rule set to obtain the second rule set.
[0095] The third case: the write operation type is a modification operation.
[0096] In the above case, the first thread may be called to modify the regular expression rules in the first rule set to obtain the second rule set.
[0097] The compiling module 44 is used to call the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set.
[0098] Since managing and compiling rule sets both require a lot of time, in order to improve efficiency without affecting the state machine database currently in use, an embodiment of the present application proposes to separately schedule a second thread to compile the second rule set to convert all regular expression rules in the second rule set into a state machine in a binary representation, so that Hyperscan can use the pre-compiled state machine to process multiple data matches in parallel. At the same time, the state machine can also enable Hyperscan to support multiple scanning methods to adapt to different data streams and application scenarios.
[0099] Optionally, the compiling module 44 may call the second thread to compile the second rule set through a preset compiling interface and using a preset compiling mode to obtain a state machine database corresponding to the second rule set.
[0100] The acquisition module 46 is used to acquire the to-be-matched data corresponding to the target regular expression rule in the second rule set.
[0101] Since the deep packet inspection process involves matching data packets with rules, the data to be matched in the embodiment of the present application can be obtained by extracting and analyzing the network traffic during the deep packet inspection process, and the data to be matched corresponds to at least one target regular expression rule in the second rule set.
[0102] The data matching module 48 is used to call the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0103] Furthermore, the device also includes a feedback module, and the feedback module is used to call a preset callback function (MatchCallback) to feed back the matching result, wherein the callback function is set by the user when calling Hyperscan for data matching.
[0104] It should be noted that the various modules in the above-mentioned Hyperscan-based read-write processing device can be program modules (for example, a set of program instructions that implement a certain specific function) or hardware modules. For the latter, it can be expressed in the following forms, but is not limited to this: the expression form of each of the above-mentioned modules is a processor, or the functions of each of the above-mentioned modules are implemented by a processor.
[0105] Example 3
[0106] According to an embodiment of the present application, a non-volatile storage medium is further provided, in which a program is stored, wherein when the program is running, the device where the non-volatile storage medium is located is controlled to execute the Hyperscan-based read and write processing method in Example 1.
[0107] Optionally, the device where the non-volatile storage medium is located implements the following steps by running the program:
[0108] Step S1, in response to a write operation instruction for a first rule set, calling a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules;
[0109] Step S2, calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set;
[0110] Step S3, obtaining the to-be-matched data corresponding to the target regular expression rule in the second rule set;
[0111] Step S4, calling the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0112] According to an embodiment of the present application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, the Hyperscan-based read and write processing method in Example 1 is implemented.
[0113] Optionally, the computer program executes the following steps:
[0114] Step S1, in response to a write operation instruction for a first rule set, calling a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules;
[0115] Step S2, calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set;
[0116] Step S3, obtaining the to-be-matched data corresponding to the target regular expression rule in the second rule set;
[0117] Step S4, calling the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0118] According to an embodiment of the present application, a processor is also provided, which is used to run a program, wherein the Hyperscan-based read and write processing method in Example 1 is executed when the program is running.
[0119] Optionally, the following steps are performed when the program is running:
[0120] Step S1, in response to a write operation instruction for a first rule set, calling a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules;
[0121] Step S2, calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set;
[0122] Step S3, obtaining the to-be-matched data corresponding to the target regular expression rule in the second rule set;
[0123] Step S4, calling the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0124] According to an embodiment of the present application, an electronic device is also provided, wherein: Figure 5 is a schematic diagram of the structure of an optional electronic device according to an embodiment of the present application, such as Figure 5 As shown, the electronic device includes one or more processors; a memory for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors are implemented to run the programs, wherein the programs are configured to execute the Hyperscan-based read and write processing method in the above-mentioned embodiment 1 when running.
[0125] Optionally, the processor is configured to implement the following steps by executing a computer program:
[0126] Step S1, in response to a write operation instruction for a first rule set, calling a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules;
[0127] Step S2, calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set;
[0128] Step S3, obtaining the to-be-matched data corresponding to the target regular expression rule in the second rule set;
[0129] Step S4, calling the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
[0130] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.
[0131] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.
[0132] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.
[0133] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed over multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.
[0136] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.
Claims
1. A read-write processing method based on Hyperscan, characterized in that: include: In response to a write operation instruction for a first rule set, calling a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules; Invoking a second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set; Acquire the to-be-matched data corresponding to the target regular expression rule in the second rule set; The third thread is called to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
2. The method according to claim 1, characterized in that Before responding to the write operation instruction of the target object to the first rule set, the method further includes: Obtaining the at least one regular expression rule; Performing compliance check on the at least one regular expression rule respectively to obtain corresponding check results, and forming a first rule set from all the regular expression rules whose check results are passed, wherein the compliance check includes at least one of the following: syntax check, logic check, and security check; The first rule set is compiled through a preset compilation interface and using a preset compilation mode to obtain a state machine database corresponding to the first rule set, wherein the compilation mode includes: block mode, stream mode or vector mode.
3. The method according to claim 1, characterized in that In response to a write operation instruction for the first rule set, Calling the first thread to perform a corresponding write operation on the first rule set to obtain a second rule set includes: Receiving a write operation instruction for the first rule set, wherein the write operation instruction carries at least a regular expression rule to be processed and a corresponding write operation type, and the write operation type includes at least one of the following: a new operation, a delete operation, and a modify operation; When the write operation type is the new addition operation, calling the first thread to add a regular expression rule to the first rule set to obtain the second rule set; In a case where the write operation type is the delete operation, calling the first thread to delete the regular expression rules in the first rule set to obtain the second rule set; When the write operation type is the modification operation, the first thread is called to modify the regular expression rules in the first rule set to obtain the second rule set.
4. The method according to claim 3, characterized in that Calling the first thread to add a regular expression rule to the first rule set to obtain the second rule set includes: Determining whether the first rule set includes the regular expression rule; In the case that the first rule set does not include the regular expression, calling the first thread to perform compliance check on the regular expression rule to obtain a corresponding check result; When the verification result is verification passed, the first thread is called to combine the first rule set and the regular expression rule to obtain the second rule set.
5. The method according to claim 1, characterized in that Calling the second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set includes: The second thread is called, and the second rule set is compiled through a preset compilation interface and using a preset compilation mode to obtain a state machine database corresponding to the second rule set.
6. The method according to claim 1, characterized in that Before calling at least one third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set, the method further includes: The state machine database pointer of Hyperscan is replaced from pointing to the state machine database corresponding to the first rule set to pointing to the state machine database corresponding to the second rule set, and the state machine database corresponding to the first rule set is released using a preset destruction interface.
7. The method according to claim 1, characterized in that After obtaining the corresponding matching result, the method further includes: The preset callback function is called to feedback the matching result.
8. A read-write processing device based on Hyperscan, characterized in that: include: A management module, configured to, in response to a write operation instruction for a first rule set, call a first thread to perform a corresponding write operation on the first rule set to obtain a second rule set, wherein the first rule set and the second rule set respectively include at least one group of regular expression rules; A compiling module, configured to call a second thread to compile the second rule set to obtain a state machine database corresponding to the second rule set; An acquisition module, used for acquiring to-be-matched data corresponding to a target regular expression rule in the second rule set; The data matching module is used to call the third thread to perform a matching operation on the to-be-matched data in the state machine database corresponding to the second rule set to obtain a corresponding matching result.
9. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, wherein the device where the non-volatile storage medium is located executes the Hyperscan-based read and write processing method according to any one of claims 1 to 7 by running the computer program.
10. A computer program product, characterized in that include: A computer program, wherein when the computer program is executed by a processor, the Hyperscan-based read and write processing method described in any one of claims 1 to 7 is implemented.