A method and device for optimizing configuration of dual regularization engines

Abstract

The present invention relates to the field of network security technology, and discloses a method and device for optimizing the configuration of dual regular engines. The method provided by the present invention comprises: obtaining a task to be processed, wherein the task to be processed comprises at least one regular expression; calculating the total number of rules and the total number of characters in the task to be processed, wherein the total number of characters is the sum of the number of characters contained in the total number of rules; when the total number of characters meets a first preset condition, starting the first engine; when a single expression in the total rule meets a second preset condition, segmenting the expression, and starting the second engine for the segmented expression. The method provided by the present invention gives full play to the respective advantages of the first engine and the second engine, and achieves the purpose of taking into account both processing speed and comprehensiveness without increasing the occupation of system resources.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a method and device for optimizing configuration of a dual regularization engine. Background Art

[0002] Regular expressions describe a string matching pattern, which is used to check whether a string contains a certain substring, replace the matched substring, or extract the substring that meets a certain condition from a string. Regular expressions have been widely used and developed in various computer languages ​​or various application fields, such as various editing software, search engines, database management systems, etc. Regular engines (also called regular expression engines or regular matching engines) are the basis of regular expression matching algorithms. Mainstream operating systems and development languages ​​have regular engines. Regular engines play an important role in the security field, such as sensitive word recognition in data security and deep packet parsing in network security. Developing the most efficient regular engine possible is a hot topic in academia and industry. One development direction is to allow the regular engine to process multiple regular expressions at the same time, that is, multi-regular engine technology. At present, the optimization solutions for multi-regular engines mainly include:

[0003] (1) Multi-process / thread processing technology: With the development of computers, multi-threading technology has significantly improved the computing efficiency of programs. It is necessary to ensure that multiple regular engines fully utilize hardware capabilities to improve program efficiency. Assigning different parts of the task (different regular expressions) to multiple instances of the same regular engine for parallel processing is a relatively direct method to improve the processing speed of the regular engine. The disadvantage of this multi-engine technology combined with multi-threading technology is that parallel processing takes up more system resources because each engine has to scan all texts. This method is not applicable in scenarios where resource usage is limited. For example, on a general office terminal, the regular expression engine that scans sensitive information cannot take up too many resources, thereby affecting the user's normal office application.

[0004] (2) Deterministic finite automaton (DFA) method: This method can match multiple regular expressions by scanning the text once, collectively referred to as a high-speed engine. This engine is dominated by the scanned text and uses the text to match in multiple regular engines, thereby achieving the purpose of completing all matches by scanning the text only once. Compared with the traditional regular engine dominated by regular expressions (hereinafter referred to as a low-speed engine), which requires multiple scans of the text, the DFA method is faster. The disadvantage of this multi-engine technology of scanning multiple times in one scan is that it cannot support certain types of regular expressions, such as regular expressions using backreferences. Therefore, if only a high-speed engine is used, the types of expressions supported by the program will be reduced, thereby reducing the coverage of the business.

[0005] In summary, how to improve the processing speed of the regular expression engine while maintaining its compatibility as much as possible and reducing the occupation of system resources is a technical problem that needs to be solved urgently. Summary of the invention

[0006] In view of this, the present invention provides a method and device for optimizing the configuration of dual regular engines to solve the problem of how to improve the processing speed of the regular engine while maintaining its compatibility as much as possible and reducing the occupation of system resources.

[0007] In a first aspect, the present invention provides a method for optimizing the configuration of a dual regularization engine, the method comprising:

[0008] Acquire a task to be processed, wherein the task to be processed includes at least one regular expression;

[0009] Calculate the total number of rules and the total number of characters in the task to be processed, where the total number of characters is the sum of the number of characters contained in the total rules;

[0010] When the total number of characters meets a first preset condition, starting the first engine;

[0011] When a single expression in the general rule meets the second preset condition, the expression is segmented, and the second engine is started for the segmented expression.

[0012] The method for optimizing the configuration of dual regular engines provided by the present invention fully utilizes the respective advantages of the first engine and the second engine, and achieves the purpose of balancing processing speed and comprehensiveness without increasing system resource usage.

[0013] In an optional implementation, the first engine is used to scan the character string once to complete the matching of each keyword.

[0014] The method for optimizing the configuration of dual regular engines provided by the present invention achieves the purpose of taking into account both processing speed and comprehensiveness without increasing the occupation of system resources through the first engine.

[0015] In an optional implementation, the second engine is used to match the entire string using each keyword in the regular expression.

[0016] The method for optimizing the configuration of dual regular engines provided by the present invention achieves the purpose of taking into account both processing speed and comprehensiveness without increasing the occupation of system resources through the second engine.

[0017] In an optional implementation, segmenting the expression includes:

[0018] When the expression contains a logical relationship or, separate the sub-expressions in the logical relationship or.

[0019] The method for optimizing the configuration of dual regular engines provided by the present invention improves the processing speed of the regular engines through segmentation while maintaining their compatibility as much as possible and reducing the occupation of system resources.

[0020] In an optional implementation, when the sub-expression satisfies the second preset condition, the sub-expression is segmented, and the second engine is started for the segmented sub-expressions.

[0021] In a second aspect, the present invention provides a device for optimizing the configuration of a dual regularization engine, the device comprising:

[0022] An acquisition module, used for acquiring a task to be processed, wherein the task to be processed includes at least one regular expression;

[0023] A calculation module, used to calculate the total number of rules and the total number of characters in the task to be processed, wherein the total number of characters is the sum of the number of characters contained in the total rules;

[0024] A first configuration module, configured to start the first engine when the total number of characters meets a first preset condition;

[0025] The second configuration module is used to segment a single expression in the general rule when the single expression meets the second preset condition, and start the second engine for the segmented expression.

[0026] In an optional implementation, in the first configuration module, the first engine is used to scan the character string once to complete the matching of each keyword.

[0027] In an optional implementation, in the second configuration module, the second engine is used to match the entire string through each keyword in the regular expression.

[0028] In an optional implementation, in the second configuration module, when the expression is segmented, it includes:

[0029] When the expression contains a logical relationship or, separate the sub-expressions in the logical relationship or.

[0030] In an optional implementation, when the sub-expression satisfies the second preset condition, the sub-expression is segmented, and the second engine is started for the segmented sub-expressions.

[0031] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the method for optimizing the configuration of a dual regular engine according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0032] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the method for optimizing configuration of a dual regularization engine according to the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is a flow chart of a method for optimizing configuration of a dual regular engine according to an embodiment of the present invention;

[0035] Figure 2 is a flow chart of another method for optimizing configuration of a dual regular engine according to an embodiment of the present invention;

[0036] Figure 3 is a flowchart of another method for optimizing configuration of a dual regularization engine according to an embodiment of the present invention;

[0037] Figure 4 is a structural block diagram of a device for optimizing configuration of dual regularization engines according to an embodiment of the present invention;

[0038] Figure 5 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0040] According to an embodiment of the present invention, a method embodiment for optimizing the configuration of a dual regular engine is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] In this embodiment, a method for optimizing the configuration of a dual regularization engine is provided, which can be used in the above-mentioned mobile terminals, such as mobile phones, tablet computers, etc. Figure 1 is a flow chart of a method for optimizing configuration of a dual regular engine according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0042] Step S101, obtaining a task to be processed, where the task to be processed includes at least one regular expression.

[0043] In a specific embodiment, the tasks to be processed are not limited here, and a corresponding number of regular expressions are obtained according to actual conditions.

[0044] Step S102, calculating the total number of rules and the total number of characters in the task to be processed, wherein the total number of characters is the sum of the number of characters included in the total rules.

[0045] In an embodiment of the present invention, the total number of rules RuleNum in the task to be processed and the total number of characters CharNum contained in all the rules are calculated. The total number of rules RuleNum is the number of regular expressions contained in the task. Usually, a task will contain multiple regular expressions. For example, if a task is to find sensitive information, it may search for information such as name, age, and gender, and each type of such information will correspond to one or more regular expressions for identification. The total number of characters CharNum refers to the result obtained by adding the number of characters contained in all regular expressions in the task. For example, a regular expression \b\d{9}\b includes 9 characters.

[0046] Step S103: when the total number of characters meets a first preset condition, start the first engine.

[0047] In an embodiment of the present invention, the first engine is used to scan the character string once to complete the matching of various keywords. The scanning task is completed by a one-time scanning and multiple-matching method. For example, to scan the two keywords "school" and "class", the first engine (high-speed engine) will send each character of the character string to be processed "My graduation school is Tsinghua University" to the two keywords "school" and "class" in sequence for matching. The entire character string only needs to be scanned once to complete the matching task.

[0048] The first preset condition is not limited here and is selected accordingly according to actual conditions. For example, when the high-speed engine can process rules greater than Thd_RuleNum in task T, or the total number of characters in the rules that can be processed in task T exceeds Thd_CharNum, the high-speed engine is enabled.

[0049] Step S104: when a single expression in the general rule satisfies the second preset condition, the expression is segmented, and the second engine is started for the segmented expression.

[0050] In the embodiment of the present invention, the second engine is used to match the entire string through each keyword in the regular expression, which is equivalent to using each keyword in the regular expression to match the entire string, so the string will be scanned repeatedly. In the embodiment of the present invention, the second engine is also called a low-speed engine.

[0051] The second preset condition is not limited here and is selected accordingly according to the actual situation. For example, 2) the method of adaptively distributing workloads for segmentation and reorganization of regular expressions, the way of distributing tasks in different regular engines is to use the entire regular expression as the segmentation granularity. In actual work, some regular expressions are very long, which may be several times or even dozens of times longer than other regular expressions, and the processing overhead is too large. Therefore, an embodiment of the present invention proposes a method for segmenting regular expressions and distributing them to different regular engines. The regular expression is segmented according to its length, and the total number of characters to be cut, Cut_Thd_CharNum, is determined. If the number of characters contained in a regular expression exceeds Cut_Thd_CharNum, it is segmented.

[0052] In the embodiment of the present invention, when the expression is segmented, it includes: when the expression contains a logical relationship or, each sub-expression in the logical relationship or is segmented. For example, if the regular expression itself contains a logical relationship like "condition 1 or condition 2", the entire regular expression can be segmented into new regular expressions containing condition 1 and condition 2, and these newly generated regular expressions are completely treated as independent regular expressions.

[0053] In the embodiment of the present invention, when the sub-expression satisfies the second preset condition, the sub-expression is segmented, and the second engine is started for the segmented sub-expressions.

[0054] like Figure 2 , Figure 3 As shown, it is a flowchart of another method for optimizing the configuration of a dual regularization engine provided by an embodiment of the present invention.

[0055] The method for optimizing the configuration of dual regular engines provided in this embodiment selects various characteristics of regular expressions themselves as indicators for measuring the difficulty of regular expression matching tasks, adaptively cuts and reorganizes regular expressions, flexibly distributes workloads to multiple regular engines, and speeds up the overall calculation speed of the regular engines.

[0056] In the present embodiment, a device for optimizing the configuration of dual regular engines is also provided, and the device is used to implement the above-mentioned embodiments and preferred embodiments, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0057] This embodiment provides a device for optimizing the configuration of dual regular engines, such as Figure 4 As shown, including:

[0058] An acquisition module 401 is used to acquire a task to be processed, where the task to be processed includes at least one regular expression;

[0059] A calculation module 402 is used to calculate the total number of rules and the total number of characters in the task to be processed, wherein the total number of characters is the sum of the number of characters included in the total rules;

[0060] A first configuration module 403, configured to start the first engine when the total number of characters meets a first preset condition;

[0061] The second configuration module 404 is used to segment a single expression in the general rule when the single expression meets the second preset condition, and start the second engine for the segmented expression.

[0062] In some optional implementations, in the first configuration module, the first engine is used to scan the character string once to complete the matching of each keyword.

[0063] In some optional implementations, in the second configuration module, the second engine is used to match the entire string through each keyword in the regular expression.

[0064] In some optional implementations, in the second configuration module, when segmenting the expression, it includes:

[0065] When the expression contains a logical relationship or, separate the sub-expressions in the logical relationship or.

[0066] In some optional implementations, when a sub-expression satisfies a second preset condition, the sub-expression is segmented, and a second engine is started for the segmented sub-expressions.

[0067] The device for optimizing the configuration of the dual regularization engine in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0068] The embodiment of the present invention also provides a computer device having the above Figure 4 The device with the optimally configured dual regularization engine is shown.

[0069] See also Figure 5 , Figure 5 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 5 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 5 A processor 10 is taken as an example.

[0070] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0071] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0072] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0073] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0074] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0075] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0076] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for optimizing the configuration of a dual regularization engine, characterized in that: The method comprises: Acquire a task to be processed, wherein the task to be processed includes at least one regular expression; Calculate the total number of rules and the total number of characters in the task to be processed, where the total number of characters is the sum of the number of characters contained in the total rules; When the total number of characters meets the first preset condition, the first engine is started, and the first engine is used to scan the character string once to complete the matching of each keyword. The first preset condition is that the number of tasks that can be processed is greater than the total number of preset rules, or the total number of characters of the rules of the tasks that can be processed is greater than the total number of preset characters; When a single expression in the total rule meets the second preset condition, the expression is segmented, and the second engine is started for the segmented expression, the second engine is used to match the entire string through each keyword in the regular expression, and the second preset condition is that the characters contained in the regular expression exceed the preset total number of segmentation characters; Among them, when splitting the expression, it includes: When the expression contains a logical relationship or, separate the sub-expressions in the logical relationship or.

2. The method according to claim 1, characterized in that When the sub-expression satisfies the second preset condition, the sub-expression is segmented, and the second engine is started for the segmented sub-expressions.

3. A device for optimizing the configuration of a dual regularization engine, characterized in that: The device comprises: An acquisition module, used for acquiring a task to be processed, wherein the task to be processed includes at least one regular expression; A calculation module, used to calculate the total number of rules and the total number of characters in the task to be processed, wherein the total number of characters is the sum of the number of characters contained in the total rules; A first configuration module is used to start a first engine when the total number of characters meets a first preset condition, wherein the first engine is used to scan the character string once to complete the matching of each keyword, and the first preset condition is that the number of tasks that can be processed is greater than the total number of preset rules, or the total number of characters of the rules of the tasks that can be processed is greater than the total number of preset characters; A second configuration module is used to segment a single expression in the general rule when the single expression meets the second preset condition, and start a second engine for the segmented expression, wherein the second engine is used to match the entire string through each keyword in the regular expression, and the second preset condition is that the characters contained in the regular expression exceed the preset total number of segmentation characters; The segmentation of the expression includes: when the expression contains a logical relationship or, segmenting each sub-expression in the logical relationship or.

4. The device according to claim 3, characterized in that When the sub-expression satisfies the second preset condition, the sub-expression is segmented, and the second engine is started for the segmented sub-expressions.

5. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the method for optimizing the configuration of a dual regular engine according to any one of claims 1 to 2 by executing the computer instructions.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the method for optimizing configuration of a dual regularization engine according to any one of claims 1 to 2.

Images