Method, apparatus, medium and electronic device for implementing traffic collection analysis

Abstract

A method, device, medium and electronic equipment for implementing traffic collection analysis are disclosed, wherein the method comprises: performing common subexpression extraction processing on each regular rule in a regular rule set to obtain one or more common subexpressions; obtaining to-be-processed data packet information that needs to be subjected to traffic collection analysis; in a process of performing regular rule matching calculation on the to-be-processed data packet information based on each regular rule in the regular rule set, judging whether a current to-be-matched subexpression is a common subexpression; and in a case where it is judged that the current to-be-matched subexpression is a common subexpression and a matching calculation result of the common subexpression is currently stored, obtaining the matching calculation result of the current to-be-matched subexpression from the stored matching calculation result. The present disclosure is conducive to guaranteeing the integrity of traffic collection analysis, the real-time performance of attack interception and the robustness of the system in a high-concurrency scenario.

Description

Technical Field

[0001] This disclosure relates to the field of network security technology, and in particular to a method, apparatus, storage medium, and electronic device for implementing traffic collection and analysis. Background Technology

[0002] With the acceleration of digital transformation, network traffic is growing explosively. For example, at nodes in large enterprise intranets, Internet backbones, and data centers, the number of data packets processed per second has reached hundreds of thousands or even millions.

[0003] As the core entry point for security protection, traffic collection and analysis systems need to perform multi-dimensional regular expression rule matching on each data packet. This includes matching and calculating protocol identifiers (such as HTTP / 1.1 and HTTPS) and illegal content (such as SQL injection statements and malicious scripts) to achieve core functions such as traffic feature identification, sensitive information filtering, and attack interception. To ensure comprehensive security protection, the regular expression rules used by traffic collection and analysis systems typically need to cover multiple scenarios and types, leading to a continuous increase in the number of regular expression rules.

[0004] Currently, traffic acquisition and analysis systems typically employ a "flat storage + serial call" logic to achieve security protection. Specifically, for each received data packet, each regular rule in the regular rule set / regular rule library is used to sequentially execute the complete syntax parsing, state construction, and matching calculation of data packet information.

[0005] However, in high-concurrency scenarios, existing security measures not only cause a significant spike in processor utilization (such as CPUs, or central processing units), but also a substantial increase in latency for regular expression matching of data packets. For example, in a high-concurrency scenario of 100,000 data packets per second, on the one hand, the utilization of a single-core CPU often spikes to over 90%, far exceeding the CPU's hardware capacity; on the other hand, the latency for regular expression matching often exceeds 30ms. This not only causes some data packets to be dropped due to timeouts, affecting the integrity of traffic analysis, but also fails to meet the real-time requirements for attack interception in fields such as financial transactions and industrial control. This can potentially create security vulnerabilities and easily lead to system crashes.

[0006] How to reduce processor utilization and the latency of regular expression matching for data packets in high-concurrency scenarios, thereby improving the completeness of traffic analysis, the real-time performance of attack interception, and the robustness of the system, is a technical issue that deserves attention. Summary of the Invention

[0007] To address the aforementioned technical problems, this disclosure is proposed. Embodiments of this disclosure provide a method for implementing traffic acquisition and analysis, an apparatus for implementing traffic acquisition and analysis, a storage medium, and an electronic device.

[0008] According to a first aspect of the present disclosure, a method for implementing traffic collection and analysis is provided, comprising: extracting common subexpressions from each regular expression rule in a set of regular expressions to obtain one or more common subexpressions; wherein the common subexpression is a subexpression contained in multiple regular expression rules in the set of regular expressions; acquiring data packet information to be processed for traffic collection and analysis; determining whether the subexpression to be matched and calculated is a common subexpression during the process of performing regular expression rule matching calculation on the data packet information to be processed based on each regular expression rule in the set of regular expressions; if it is determined that the subexpression to be matched and calculated is a common subexpression and the matching calculation result of the common subexpression is not currently stored, performing matching calculation on the subexpression to be matched and calculating, and storing the matching calculation result as the matching calculation result of the common subexpression; if it is determined that the subexpression to be matched and calculated is a common subexpression and the matching calculation result of the common subexpression is currently stored, obtaining the matching calculation result of the subexpression to be matched and calculated from the stored matching calculation result.

[0009] According to a second aspect of the present disclosure, an apparatus for implementing traffic collection and analysis is provided, comprising: an extraction processing module, configured to perform common subexpression extraction processing on each regular expression rule in a set of regular expression rules to obtain one or more common subexpressions; wherein the common subexpression is a subexpression contained in multiple regular expression rules in the set of regular expression rules; a packet information acquisition module, configured to acquire data packet information to be processed for traffic collection and analysis; and a judgment module, configured to determine whether the subexpression to be matched and calculated is a common subexpression in the process of performing regular expression rule matching calculation on the data packet information to be processed based on each regular expression rule in the set of regular expression rules. A common subexpression; an execution storage module, used to perform a matching calculation on the current subexpression to be matched when the judgment module determines that the subexpression to be matched is a common subexpression and there is no matching calculation result of the common subexpression currently stored, and store the matching calculation result as the matching calculation result of the common subexpression; a result reuse module, used to obtain the matching calculation result of the current subexpression to be matched from the stored matching calculation results when the judgment module determines that the current subexpression to be matched is a common subexpression and there is a matching calculation result of the common subexpression currently stored.

[0010] According to a third aspect of the present disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program for implementing any of the methods described above.

[0011] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement any of the methods described above.

[0012] Based on the methods, apparatus, storage media, and electronic devices for traffic acquisition and analysis provided in the above embodiments of this disclosure, by extracting common sub-expressions from regular rules in a set of regular rules, the sub-expressions contained in a regular rule can be divided into common sub-expressions and non-common sub-expressions. Since a common sub-expression may appear in multiple regular rules, if a regular rule contains a common sub-expression, when performing matching calculations on the data packet information to be processed based on that regular rule, it may not be necessary to use the common sub-expression to perform matching calculations on the data packet information to be processed. Instead, only the non-common sub-expressions may be used to perform matching calculations on the data packet information to be processed. This is because the common sub-expression may have already been executed during the matching calculations on the data packet information to be processed based on other regular rules. The matching calculation process is performed, and the results are stored. These stored results can be reused by subsequent regular rules containing the common subexpression; that is, the matching result of the common subexpression can be directly retrieved from the stored information. This allows the matching calculation operation performed on the data packet information using the common subexpression to be executed only once, thus avoiding a certain number of matching calculation operations. Especially in high-concurrency scenarios with a large number of regular rules containing the common subexpression, this significantly reduces the amount of matching calculations. This not only helps avoid excessive processor load but also shortens the rule matching calculation time. For example, in some experimental scenarios, CPU utilization can be reduced by 10%-20%, while the rule matching calculation time for a single data packet can be reduced by 20%-30%. Therefore, the technical solution provided in this disclosure improves rule matching efficiency, enhances security response speed, and reduces processor utilization. It is particularly suitable for high-concurrency scenarios, ensuring the integrity of traffic collection and analysis, the real-time nature of attack interception, and the robustness of the system in high-concurrency scenarios.

[0013] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0014] The above and other objects, features, and advantages of this disclosure will become more apparent from the more detailed description of the embodiments thereof in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this disclosure and form part of the specification. They are used together with the embodiments of this disclosure to explain the disclosure and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0015] Figure 1 This is a flowchart of an embodiment of the method for implementing traffic acquisition and analysis disclosed herein;

[0016] Figure 2 This is a flowchart of another embodiment of the method for implementing traffic acquisition and analysis disclosed herein;

[0017] Figure 3 This is a schematic diagram of an embodiment of the apparatus for implementing traffic acquisition and analysis disclosed herein;

[0018] Figure 4 This is a structural diagram of an electronic device provided in an exemplary embodiment of this disclosure. Detailed Implementation

[0019] Example embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure, and it should be understood that this disclosure is not limited to the example embodiments described herein.

[0020] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0021] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0022] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0023] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0024] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0025] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0026] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0027] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0028] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0029] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0030] The embodiments of this disclosure can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, or servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems, etc.

[0031] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment. In a distributed cloud computing environment, tasks can be performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can reside on local or remote computing system storage media, including storage devices.

[0032] This disclosure outlines

[0033] In developing this disclosure, the inventors discovered that different regular expression rules may contain the same matching calculations. For example, multiple regular expression rules may require IP address matching and / or port matching calculations. If each regular expression rule in the set is processed independently based on packet information, the same matching calculation will often be performed multiple times for a single packet. This not only wastes the computing resources of the CPU and other processors (the following explanation uses the CPU as an example) but also hinders the reduction of the time consumed by regular expression rule matching calculations. Especially when the set of regular expression rules contains a large number of rules and the number of identical matching calculations involved in different rules is large, the waste of computing resources and the time consumed by matching calculations are considerable. Furthermore, high-concurrency scenarios will greatly exacerbate the waste of computing resources and the time consumed by matching calculations.

[0034] If, for a single data packet, the same matching calculation in different regular expression rules is controlled to be performed only once, and the result of this matching calculation can be reused by multiple regular expression rules, it will not only help reduce the amount of computation for matching calculation and reduce the CPU's computational load, but also help reduce the time for matching calculation of data packet information. This will help improve the efficiency and robustness of the traffic collection and analysis system, and make the traffic collection and analysis system more suitable for high-concurrency scenarios.

[0035] Exemplary Overview

[0036] The technical solution for traffic collection and analysis provided in this disclosure can be applied to a variety of application scenarios, such as traffic collection and analysis for enterprise intranet security and traffic collection and analysis for internet e-commerce platforms.

[0037] A concrete example of setting up traffic collection and analysis requirements for enterprise intranet security includes: real-time monitoring of abnormal behavior in intranet traffic and interception of data leakage behaviors, etc.

[0038] First, to address the actual security needs of traffic collection and analysis, multiple regular expression rules are set up, and all these rules form a set of regular expression rules (also known as a regular expression rule library). For example, regular expression rules can be set for IP (Internet Protocol) addresses (such as source IP address and destination IP address), ports (such as source port and destination port), packet protocol type, packet size, session duration, HTTP / HTTPS (Hypertext Transfer Protocol / Hypertext Transfer Protocol Secure) requests, FTP (File Transfer Protocol) file transfer records, database query statements (such as SQL (Structured Query Language)), and email content (such as sender, recipient, and attachment name), forming a set of regular expression rules.

[0039] Secondly, during the startup of the traffic collection and analysis system, a regular expression parser can be used to parse each regular expression rule in the set of regular expression rules, thereby obtaining the Abstract Syntax Tree (AST) of each regular expression rule. By extracting common subexpressions from all ASTs, one or more common subexpressions can be obtained. Afterwards, rule loading operations such as regular expression rule parsing and state construction can be performed to form a regular expression rule state machine, such as an NFA (Non-deterministic Finite Automaton) or a DFA (Deterministic Finite Automaton). It should be noted that the rule loading operation here can be based on the principle of avoiding the repetition of common subexpressions.

[0040] Furthermore, for a currently received data packet, during the process of performing regular expression-based matching calculations using the data packet information (such as the packet header information and the packet payload information), for common subexpressions in the regular expression rules, only one matching calculation is needed, and the matching calculation result corresponding to the common subexpression is stored. This stored matching calculation result corresponding to the common subexpression can be directly read when matching calculations are needed for the common subexpression next time. For each non-common subexpression, the corresponding matching calculation should be performed using NFA / DFA.

[0041] The same matching calculation process is then performed on the next data packet to achieve security protection for the enterprise's internal network.

[0042] Exemplary methods

[0043] Figure 1 This is a flowchart of an embodiment of the method for implementing traffic acquisition and analysis disclosed herein. Figure 1 The method shown includes steps S100, S110, S120, S130, and S140, and may further include step S150. The following describes... Figure 1 Each step in the process will be explained separately.

[0044] S100. Extract common subexpressions from each regular expression rule in the set of regular expressions to obtain one or more common subexpressions.

[0045] The regular expression (Regex) set (also known as the regular expression rule library) disclosed herein contains regular expression rules configured according to the actual security protection needs of traffic collection and analysis. For example, the regular expression rule set / library disclosed herein may include: regular expression rules configured for packet header information (i.e., packet header information) and regular expression rules configured for packet payload information; the regular expression rules configured for packet header information may include: regular expression rules configured for header fields such as IP address (e.g., source IP address and destination IP address), port (e.g., source port and destination port), packet protocol type, IP version, and packet size; the regular expression rules configured for packet payload information may include: regular expression rules configured for payload content such as application layer protocol content, application layer malicious feature detection, and sensitive data identification. In one example, the number of regular expression rules contained in the regular expression rule set is usually large, for example, dozens or hundreds of regular expression rules. This disclosure does not limit the specific content of the regular expression rules or the number of regular expression rules contained in the regular expression rule set.

[0046] In this disclosure, a common subexpression refers to a subexpression that is included in multiple regular expression rules within a set of regular expression rules. That is, if a subexpression appears in multiple regular expression rules, it may be considered a common subexpression. If a subexpression is not considered a common subexpression, it can be considered a specific or differential subexpression of the regular expression rule. Therefore, any regular expression rule in this disclosure includes both common and / or specific subexpressions.

[0047] In this disclosure, both subexpressions and common subexpressions can include one or more atomic matching segments. Furthermore, both subexpressions and common subexpressions should possess independent integrity; that is, each subexpression and common subexpression should be a complete functional unit within the regular expression rule. In other words, accurate matching calculations can be performed on subexpressions and common subexpressions without relying on external context or the regular expression rule itself. In one example, a regular expression rule is set up to detect common sensitive data such as mobile phone numbers, ID card numbers, and bank card numbers from the HTTP request body and URL (Uniform Resource Locator) parameters to prevent data leakage. This regular expression rule can be expressed as:

[0048] (?i)\b(?:1[3-9]\d{9}|\d{17}[\dXx]|\d{16,19})\b

[0049] The above regular expression contains a subexpression, namely (?:1[3-9]\d{9}|\d{17}[\dXx]|\d{16,19}), where 1, [3-9], \d, and [\dXx] are all atomic matching fragments of this regular expression.

[0050] This disclosure primarily determines whether a subexpression in a regular expression rule will be considered a common subexpression by assessing the CPU resource consumption of each subexpression's matching calculation within the regular expression rule set. Since the number of times a subexpression appears in all regular expressions reflects the number of times its matching calculation is performed, this frequency becomes a factor in measuring its CPU resource consumption. Therefore, this disclosure utilizes the frequency of a subexpression's appearance in multiple regular expressions to determine which subexpressions can be considered common subexpressions.

[0051] In one example, for any given subexpression, the number of times it appears in all regular expression rules in the set of regular expression rules (including regular expression rules set for packet headers and packet payloads) is counted. This count is then used to determine whether the subexpression should be considered a common subexpression. Such a common subexpression can be considered a globally common subexpression. For example, let's define one regular expression rule for violations of IP addresses containing \d{1,3}\.\d{1,3} in the packet payload (hereinafter referred to as the violation IP rule), and another regular expression rule for extracting IP addresses containing \d{1,3}\.\d{1,3} in the packet header's additional fields (hereinafter referred to as the IP extraction rule). Then, the subexpression in these two regular expression rules can be considered a repeating subexpression. If the count of the subexpression's repetitions meets the requirements, then the subexpression will be considered a common subexpression, such as a common subexpression for "IP fragment sharing units".

[0052] In one example, for any subexpression, if it is a subexpression in a regular expression rule set for the packet header, the number of times the subexpression appears in all regular expression rules set for the packet header in the regular expression rule set can be counted, and the number of times it appears can be used to determine whether the subexpression should be considered a common subexpression. Such a common subexpression can be considered a header-local common subexpression. If the subexpression is a subexpression in a regular expression rule set for the packet payload, the number of times the subexpression appears in all regular expression rules set for the packet payload in the regular expression rule set can be counted, and the number of times it appears can be used to determine whether the subexpression should be considered a common subexpression. Such a common subexpression can be considered a payload-local common subexpression.

[0053] In one example, this disclosure may extract only the global common subexpression. In another example, this disclosure may also extract both the header local common subexpression and the payload local common subexpression.

[0054] By forming global or local common subexpressions during the extraction process, it is beneficial to adopt different strategies to define common subexpressions according to actual needs, thereby improving the flexibility of system implementation. The following mainly uses global common subexpressions as an example to illustrate the technical solution of this disclosure.

[0055] In one example, this disclosure can utilize the abstract syntax tree of regular expression rules to count the recurrence of subexpressions within the regular expression rules and identify common subexpressions. A more concrete example is as follows:

[0056] First, each regular expression rule in the set is parsed individually to obtain its AST. For example, this disclosure can utilize parsing logic / interfaces in a regular expression rule library, RE2 (Regular Expression Engine), or PCRE2 (Perl Compatible Regular Expressions 2) to parse each regular expression rule in the set individually, thereby obtaining its AST.

[0057] After obtaining the AST of each regular expression rule, this disclosure can perform node normalization on each AST to eliminate irrelevant differences between ASTs, thereby ensuring that subexpressions with the same structure correspond to completely identical normalization results. In one example, node normalization may include: removing context-dependent nodes (such as deleting nodes like ^ (beginning of line) and $ (end of line), standardizing grouping attributes (such as uniformly retaining capture=Fals for non-capturing groups), standardizing quantifier representation (such as converting ? to {0,1}), uniformly sorting logical or branching (such as unifying a|b and b|a to a|b), and ignoring irrelevant flags. This disclosure does not limit the specific implementation process of parsing to obtain the AST of each regular expression rule, nor the specific implementation process of node normalization.

[0058] Secondly, based on the AST of each regular expression rule, the subexpression repetition count is performed. Specifically, this disclosure can extract all subexpression nodes from each AST. For example, by recursively traversing the normalized AST and extracting all group type nodes from each AST, the subexpressions contained in each regular expression rule in the regular expression rule set can be obtained. Then, the repetition count of subexpressions can be counted by serializing the nodes. For example, all group type nodes can be converted to JSON (JavaScript Object Notation) strings or tuples, and the repetition count of the JSON strings or tuples can be counted. This disclosure can record the repetition count of each subexpression in key-value pairs (e.g., storing JSON strings or tuples as keys and repetition counts as values). Additionally, this disclosure can convert the nodes in the AST representing the subexpressions into regular expression representations and store the subexpressions and their repetition counts based on the regular expression representations for easy display to the user. For example, show the user: Subexpression: (?:\d{1,3}), repeated 4 times; Subexpression: (?:\d{1,3}\.), repeated 2 times.

[0059] Finally, based on the number of repetitions of each subexpression obtained from statistics, the common subexpression of the regular expression rules in the regular expression rule set is determined. That is, this disclosure will set certain requirements for the number of repetitions. If the number of repetitions of a subexpression meets the pre-set requirements, then the subexpression can be used as a common subexpression.

[0060] This disclosure utilizes the AST of each regular expression rule to count the number of times a subexpression in a regular expression rule appears repeatedly in the set of regular expression rules. This not only accurately identifies semantically equivalent subexpressions and superficially similar but not equivalent subexpressions in different regular expression rules, but also accurately identifies subexpressions with nested or hidden structures. This avoids the problem of not accurately identifying actually identical or different subexpressions when directly counting the number of recurrences of subexpressions in regular expression rules. This improves the accuracy of filtering common subexpressions and, in turn, helps to minimize the CPU resource consumption and latency of matching calculations while ensuring the accuracy of regular expression matching calculations, thus optimizing the matching efficiency of regular expression rules.

[0061] In one example, considering factors such as the management cost of common subexpressions, if a subexpression appears only a few times in multiple regular expression rules (e.g., 2-5 times, i.e., low frequency of repetition), it may not be considered a common subexpression even though it appears repeatedly in multiple regular expression rules. In other words, this disclosure can set the repetition condition for a subexpression to be considered a common subexpression based on factors such as the management cost of common subexpressions. For example, this disclosure sets a first predetermined requirement for the number of repetitions, and this first predetermined requirement is for high-frequency repetitions. If the number of repetitions of a subexpression meets the first predetermined requirement, then the subexpression can be considered a common subexpression; otherwise, it cannot be considered a common subexpression. The high-frequency repetition count is usually not a single digit; for example, it can be 10, 15, or 20, and the first predetermined requirement can be: the number of repetitions of the subexpression reaches or exceeds this high-frequency repetition count, etc.

[0062] This disclosure uses the frequency of repetition to set the first preset requirement, which helps to avoid the phenomenon of consuming more resources (such as computing resources and storage resources) in order to reduce the CPU computing resources consumed by the matching calculation of regular rules. This helps to ensure that the technical solution of this disclosure can produce significant technical effects. For example, in high-concurrency scenarios, it can significantly reduce the CPU computing resources occupied by the matching calculation of regular rules and significantly reduce the latency of the matching calculation of regular rules.

[0063] Since the complexity of a subexpression is also a factor in measuring its CPU resource consumption (i.e., higher complexity means higher CPU resource consumption), this disclosure can use the number of times a subexpression appears in multiple regular expression rules and its complexity to determine which subexpressions can be used as common subexpressions. The complexity of a subexpression can be considered the difficulty of performing matching calculations on it, and can be reflected by one or more indicators such as nesting depth and quantifier usage type. For example, this disclosure can quantify the nesting depth and quantifier usage type of subexpressions separately, and then set the complexity of each subexpression based on the quantification results and the indicators contained in each subexpression. In other words, for any subexpression, after obtaining the number of times the subexpression appears in all regular expression rules in the set based on the AST statistics of each regular expression rule, a comprehensive judgment can be made based on the number of times the subexpression appears and its complexity to determine whether to use the subexpression as a common subexpression. By combining the repetition count and complexity of subexpressions, the CPU computational resources required for repeated matching calculations of subexpressions can be more accurately evaluated. This helps to avoid the phenomenon that subexpressions with higher complexity cannot be used as common subexpressions simply because their repetition count is slightly lower (e.g., the repetition count does not meet the first predetermined requirement). This helps to filter out subexpressions that actually consume more CPU computational resources as common subexpressions, thereby ensuring that the technical solution disclosed in this invention can produce better technical effects. For example, in high-concurrency scenarios, it can significantly reduce the CPU computational resource consumption of regular expression rule matching calculations and significantly reduce the latency of regular expression rule matching calculations.

[0064] In one example, this disclosure may set a second predetermined requirement for the product of the repetition count and complexity. That is, for any subexpression, the repetition count of the subexpression is multiplied by the complexity corresponding to the subexpression, and the subexpressions whose product satisfies the second predetermined requirement are taken as common subexpressions. The second predetermined requirement may be that the product of the repetition count and complexity of the subexpression belongs to the top N (N is an integer greater than 1, for example, N can be determined based on the top 30% of all product rankings) of all products, or it may be that the product of the repetition count and complexity of the subexpression reaches or exceeds a predetermined value, etc. Since the product of the number of repetitions and complexity of a subexpression can more accurately reflect the total computing resources and total matching computation time occupied by repeated matching calculations of the subexpression, determining the common subexpression by judging whether the product of the number of repetitions and complexity of the subexpression meets the second predetermined requirement is beneficial to filtering out subexpressions that occupy more total computing resources and have a longer total matching computation time as common subexpressions, thereby further optimizing the accuracy of the selection of common subexpressions. This helps to ensure that the technical solution disclosed in this paper can produce better technical effects. For example, in high-concurrency scenarios, it can reduce the CPU computing resources occupied by the matching calculation of regular rules to a greater extent, and can reduce the latency of the matching calculation of regular rules to a greater extent.

[0065] In one example, this disclosure can cluster regular rules based on common subexpressions, that is, cluster regular rules with the same common subexpression in a set of regular rules to obtain multiple regular rule classes. Each regular rule class typically corresponds to one common subexpression, and each regular rule class should consist of multiple regular rules. A regular rule class can have a class identifier, and this identifier can be the identifier of the common subexpression. Furthermore, when a regular rule contains multiple common subexpressions, the regular rule should belong to multiple regular rule classes simultaneously, that is, the regular rule corresponds to multiple class identifiers. This disclosure can store the correspondence information between the subexpression identifier, the regular rule identifier to which the subexpression belongs, and the cluster identifier to which the regular rule belongs. For example, the subexpression identifier and the regular rule identifier to which the subexpression belongs can be stored as key-value pairs, and the regular rule identifier to which the subexpression belongs and the cluster identifier to which the regular rule belongs can also be stored as key-value pairs. By clustering regular rules based on common subexpressions, this disclosure makes subsequent judgment operations and other processes more convenient.

[0066] It should be noted that this disclosure can perform the common subexpression extraction operation on all regular expression rules in the regular expression rule set during the startup process of the traffic acquisition and analysis system. If the regular expression rules in the regular expression rule set change, such as adding new regular expression rules, deleting existing regular expression rules, or modifying the content of existing regular expression rules, the common subexpression extraction operation on the regular expression rules in the regular expression rule set should be performed again in a timely manner. For example, this can be performed by restarting the traffic acquisition and analysis system. In addition, during the startup / restart process of the traffic acquisition and analysis system, this disclosure can perform the regular expression rule loading operation after successfully extracting the common subexpression. Typically, the regular expression rule loading operation includes: parsing the syntax of each regular expression rule to obtain the AST of each regular expression rule, and then using the AST of each regular expression rule to form a state machine (such as NFA / DFA, etc.), thereby using the state machine to complete the subsequent matching calculation operation. After extracting the common subexpression, this disclosure simplifies the regular expression rules in the regular expression rule set that contain the common subexpression. In this way, during the loading process of regular expression rules, the loading operation can be performed using the simplified regular expression rule set.

[0067] In a concrete example, this disclosure first performs a process to remove duplicate common subexpressions from the regular expression rules in the regular expression rule set based on the regular expression rule class. That is, for any given regular expression rule class, one regular expression rule can contain the common subexpression corresponding to that regular expression rule class, while the common subexpression corresponding to that regular expression rule class is removed from the other regular expression rules. This ensures that only one regular expression rule within the same regular expression rule class contains the common subexpression. After successfully performing this removal operation for each regular expression rule class, the regular expression rules in all regular expression rule classes can form a new set of regular expression rules, and this new set of regular expression rules does not contain duplicate common subexpressions. Then, tools such as RE2 / PCRE2 are used to perform syntax parsing on each regular expression rule in the new set of regular expression rules to obtain the AST of each regular expression rule. Finally, based on the obtained ASTs of each regular expression rule, a state machine (such as NFA / DFA) is constructed to obtain the state machine of the regular expression rules, thus realizing the loading process of the regular expression rules.

[0068] This disclosure simplifies regular expression rules by removing recurring common subexpressions from the regular expression rule set. This reduces syntax parsing processing and the size of the state machine during the loading of regular expression rules, thereby reducing the state machine's cache space usage and the complexity of controlling and managing the state machine.

[0069] S110. For a data packet, obtain the information of the data packet to be processed that needs to be collected and analyzed for traffic.

[0070] The data packet information to be processed in this disclosure may include: packet header information and / or packet payload information. The packet header information includes one or more of the following: source IP address, destination IP address, source port, destination port, and transport layer protocol; the packet payload information may be the transport layer payload of the data packet, such as a URL and HTTP version.

[0071] This disclosure allows for the parsing of captured network data packets to obtain information about the data packets to be processed, such as packet header information and / or payload information. It should be noted that this disclosure allows for simultaneous parsing and subsequent judgment and matching calculations, rather than parsing to obtain complete data packet information before performing these steps. For example, after obtaining the source IP address and / or destination IP address of the data packet through parsing, this source IP address and / or destination IP address can be used as the data packet information to be processed, and subsequent steps such as S120 can be executed. This disclosure does not limit the specific implementation process for obtaining the data packet information to be processed for traffic collection and analysis.

[0072] S120. In the process of performing regular rule matching calculation on the data packet information to be processed based on each regular rule in the set of regular rules, determine whether the subexpression to be matched and calculated is a common subexpression.

[0073] In one example, when performing clustering processing according to this disclosure, this disclosure can use class identifiers to determine whether the subexpression to be matched and evaluated is a common subexpression. That is, first determine the class to which the regular expression belongs, and then determine whether the class to which the regular expression belongs corresponds to a common subexpression. Specifically, a regular expression rule has a rule identifier, a cluster has a class identifier, and a class identifier can correspond to multiple rule identifiers. The class identifier can also be considered as the identifier of the common subexpression corresponding to the cluster. This disclosure can obtain the rule identifier of the regular expression rule to which the subexpression belongs when determining the regular expression rule to which the subexpression belongs (e.g., by searching in the corresponding key-value pairs using the identifier of the subexpression to be matched and evaluated, and obtaining the rule identifier of the regular expression rule from the matched key-value pairs), and determine whether the rule identifier corresponds to a common subexpression. The system first identifies the corresponding class identifier (e.g., using the rule identifier to find matching key-value pairs in the corresponding key-value pairs). If a corresponding class identifier exists (i.e., a matching key-value pair exists, and the class is not empty; since a regular expression rule may be divided into multiple clusters due to containing multiple common subexpressions, a rule identifier may correspond to multiple class identifiers), then it continues to determine whether the subexpression to be matched and evaluated is a common subexpression corresponding to that class. That is, it determines whether the identifier of the subexpression to be matched and evaluated is the class identifier of that class. For example, the identifier of the subexpression to be matched and evaluated is compared with the class identifiers of the multiple clusters to which the regular expression rule belongs. If it matches the class identifier of one of the clusters, then the subexpression to be matched and evaluated is determined to be a common subexpression; otherwise, the subexpression to be matched and evaluated is determined to be a common subexpression. Additionally, in the above determination process, if a corresponding class identifier does not exist (i.e., no matching key-value pair exists, and the class is empty), then the subexpression to be matched and evaluated is determined not to be a common subexpression. This disclosure uses the class of the regular expression to which the subexpression belongs to determine whether the subexpression is a common subexpression, which helps to obtain the judgment result quickly and accurately.

[0074] S130. If it is determined that the subexpression to be matched and calculated is a common subexpression and the matching calculation result of the common subexpression is not currently stored, perform the matching calculation on the subexpression to be matched and calculate, and store the matching calculation result as the matching calculation result of the common subexpression.

[0075] In one example, this disclosure can first use the common subexpression identifier to search in the currently stored matching calculation results. If the search is unsuccessful, it can be determined that based on the data packet information to be processed, no matching calculation has been performed on the common subexpression. Therefore, the matching calculation result of the common subexpression is not currently stored. Based on the data packet information to be processed obtained above, a matching calculation operation should be performed on the common subexpression (for example, the data packet information to be processed is provided to the state machine, and the matching calculation operation is completed by controlling the state machine). The correspondence information between the common subexpression and the matching calculation result should be stored in a timely manner. For example, the identifier of the common subexpression can be used as the key, and the matching calculation result of the common subexpression can be used as the value to store the key-value pair. Another example is to use the data packet identifier and the identifier of the common subexpression together as the key, and the matching calculation result of the common subexpression as the value to store the key-value pair. This helps to avoid the phenomenon of misuse of the matching calculation results of different data packets.

[0076] S140. If it is determined that the subexpression to be matched and calculated is a common subexpression, and the matching and calculation results of the common subexpression are currently stored, the matching and calculation results of the subexpression to be matched and calculated are obtained from the currently stored matching and calculation results.

[0077] In one example, if the identifier of the subexpression to be matched and calculated is used as the key (or the packet identifier + the identifier of the subexpression can be used as the key), and a key-value pair matching the key is successfully found, the value of the matching key-value pair can be directly used as the result of the matching calculation of the subexpression to be matched and calculated, without performing a matching calculation on the subexpression to be matched and calculated again, thus avoiding repeated matching calculation operations on the subexpression to be matched and calculated. For example, continuing the previous example, assuming that this disclosure has stored the matching calculation results for the common subexpression of "IP fragment sharing unit", if the subexpression to be matched and calculated is the common subexpression in the violation IP rule, the corresponding matching calculation result can be directly obtained from the stored information, and subsequent operations can be performed using the matching calculation result, such as comparing the matching calculation result with the malicious database; if the subexpression to be matched and calculated is the common subexpression in the IP extraction rule, the corresponding matching calculation result can be directly obtained from the stored information, and subsequent operations can be performed using the matching calculation result, such as directly formatting and outputting the matching calculation result.

[0078] In one example, when this disclosure completes the traffic collection and analysis of all pending data packet information for the currently captured data packet, it can delete the matching calculation results of the common subexpressions stored for all pending data packet information for that data packet. For example, it can delete all key-value pairs stored for the currently captured data packet, formed by the common subexpression identifier and the matching calculation results. That is, for a data packet, when the traffic collection and analysis of that data packet is completed, the lifecycle of the matching calculation results of the common subexpression corresponding to that data packet ends.

[0079] S150. When it is determined that the subexpression to be matched and evaluated is not a common subexpression, the matching and evaluation are performed on the subexpression to be matched and evaluated, but the result of the matching and evaluation is not stored.

[0080] Figure 2 This is a flowchart of another embodiment of the method for implementing traffic acquisition and analysis disclosed herein. Figure 2 The method shown includes steps S200, S210, S220, S230, and S240. The following describes... Figure 2 Each step in the process will be explained separately.

[0081] S200. During the startup process of the traffic acquisition and analysis system, extract common sub-expressions.

[0082] In one example, the set of regular expression rules is set to contain 100 regular expression rules, and this disclosure can use the parsing tool in the regex library to parse these 100 regular expression rules one by one (i.e., decompose the regular expression rules) to form 100 ASTs, and perform subexpression identification and statistical processing of the number of recurrences based on the 100 ASTs.

[0083] In one example, we define a method to filter out four frequently occurring subexpressions or subexpressions that are frequently occurring and consume a lot of computational resources, based on the number of repetitions or the number of repetitions multiplied by the complexity, as follows:

[0084] 1\d{10} (appears 35 times, associated with mobile phone number and virtual operator number segment rules);

[0085] \d{17}[\dXx] (appears 30 times, associated with ID card number and Hong Kong, Macao and Taiwan residence permit rules);

[0086] * (Appeared 28 times, associated with prohibited text and nested sensitive identifier rules, restricted) (Boundary avoids matching redundancy)

[0087] [A-Za-z0-9]{6,20} (occurs 22 times, associated with user ID and device encoding rules).

[0088] Then all four sub-expressions mentioned above can be regarded as common sub-expressions, which can be called shared unit 1, shared unit 2, shared unit 3 and shared unit 4.

[0089] Other subexpressions are treated as non-common subexpressions because their repetition count or repetition count multiplied by complexity does not meet the corresponding requirements. For example, the remaining 5 subexpressions are treated as non-common subexpressions.

[0090] S210. Cluster the regular rules based on common subexpressions.

[0091] Continuing the previous example, based on the above four common sub-expressions, the 100 regular expression rules are clustered to obtain four classes, as follows:

[0092] Class 1 corresponds to Shared Unit 1 (related to mobile phone number). Class 1 contains 35 regular expression rules (i.e., Shared Unit 1 is associated with 35 regular expression rules), and all 35 regular expression rules contain the subexpression 1\d{10}.

[0093] Class 2 corresponds to Shared Unit 2 (related to ID card number). Class 2 contains 30 regular expression rules (i.e., Shared Unit 2 is associated with 30 regular expression rules), and all 30 regular expression rules contain the subexpression \d{17}[\dXx].

[0094] Class 3 corresponds to Shared Unit 3 (fuzzy matching related). Class 3 includes 28 regular expression rules (i.e., Shared Unit 3 is associated with 28 regular expression rules), and all 28 regular expression rules contain the .* regular expression rule.

[0095] Class 4 corresponds to Shared Unit 4 (account identifier related). Class 4 includes 22 regular expression rules (i.e., Shared Unit 4 is associated with 22 regular expression rules), and all 22 regular expression rules contain the subexpression [A-Za-z0-9]{6,20}.

[0096] The remaining 5 non-common subexpressions each belong to a separate category.

[0097] The above clustering results are mainly used in step S230.

[0098] S220, Regular expression rule loading and processing.

[0099] Specifically, loading processing can be performed on the regular rules contained in the current regular rule set, or the four shared units can be used to strip the regular rules contained in the current regular rule set to form a new regular rule set, and loading processing can be performed on the regular rules contained in the new regular rule set.

[0100] S230. Perform traffic collection and analysis on the current data packets to be processed.

[0101] Continuing the previous example, for any of the above-mentioned shared units, if the storage mechanism of the shared unit is triggered (i.e., the first time a matching calculation is performed for the shared unit; the determination process for whether the storage mechanism is triggered uses the above-mentioned clustering results, which will not be described in detail here), then after the matching calculation is performed and the matching calculation result is obtained, the matching calculation result is stored. For example, the identifier of the current data packet to be processed and the identifier of the shared unit are used as keys, and the matching calculation result is used as the value, and the key-value pair is stored. If the reuse mechanism of the shared unit is triggered (i.e., a matching calculation has been performed for the shared unit; the determination process for whether the reuse mechanism is triggered uses the above-mentioned clustering results, which will not be described in detail here), then the identifier of the current data packet to be processed and the identifier of the shared unit can be used as keys to find key-value pairs that match the keys, thereby obtaining the matching calculation result of the shared unit.

[0102] For the aforementioned shared unit 1, this disclosure can reduce the number of matching calculations for 1\d{10} by 34;

[0103] Regarding the aforementioned shared unit 2, this disclosure can reduce the number of matching calculations for \d{17}[\dXx] by 29;

[0104] Regarding the aforementioned shared unit 3, this disclosure can reduce the number of matching calculations for .* by 27;

[0105] For the aforementioned shared unit 4, this disclosure can reduce the number of matching calculations for [A-Za-z0-9]{6,20} by 21.

[0106] After completing traffic collection and analysis for the current data packet to be processed (such as outputting the corresponding information to the security analysis module), the key-value pairs in the cache should be cleared in a timely manner to avoid invalid occupation of cache resources and to allow for the mixed use of matching calculation results across data packets.

[0107] S240: Is there a next data packet that needs to be collected and analyzed? If so, take the next data packet that needs to be collected and analyzed as the current data packet to be processed and return to S230. If not, continue to periodically check whether there is a next data packet that needs to be collected and analyzed.

[0108] The following is a real experimental data point disclosed in this paper:

[0109] The operating environment for traffic collection and analysis is as follows:

[0110] CPU: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz

[0111] MEM (Memory): 8GB DDR3 (Double Data Rate 3 SDRAM, third-generation double data rate synchronous dynamic random access memory)

[0112] The experimental results are shown in Table 1 below:

[0113] Table 1

[0114]

[0115] As can be seen from Table 1 above, compared with the existing "flat storage + serial call" technical solution, the technical solution of storing and reusing the matching calculation results of common subexpressions disclosed in this invention has significant improvements in both CPU utilization and the time consumption of matching calculation for a single data packet.

[0116] Exemplary device

[0117] The structure of one embodiment of the apparatus for implementing flow acquisition and analysis disclosed herein is as follows: Figure 3 As shown. The apparatus of this embodiment can be used to implement the corresponding method embodiments of this disclosure. Figure 3 The apparatus shown includes: an extraction and processing module 300, a packet information acquisition module 310, a judgment module 320, an execution storage module 330, and a result reuse module 340. Optionally, the apparatus may further include: a clustering module 350, a rule loading module 360, and a lifecycle management module 370.

[0118] The extraction and processing module 300 is mainly used to extract common subexpressions from each regular expression rule in the regular expression rule set, obtaining one or more common subexpressions. These common subexpressions are subexpressions that are included in multiple regular expression rules within the regular expression rule set.

[0119] In one example, the extraction processing module 300 may include a parsing submodule 301, a statistics submodule 302, and a filtering submodule 303. The parsing submodule 301 is primarily used to perform syntax parsing on the regular expression rules in the regular expression rule set to obtain the abstract syntax tree of each regular expression rule. The statistics submodule 302 is primarily used to perform a subexpression repetition count operation based on the abstract syntax tree of each regular expression rule obtained by the parsing submodule 301. The filtering submodule 303 is primarily used to determine the common subexpressions of the regular expression rules in the regular expression rule set based on the repetition count of each subexpression obtained by the statistics submodule 302.

[0120] In one example, the filtering submodule 303 can use a subexpression whose repetition frequency meets a first predetermined requirement as a common subexpression; where the first predetermined requirement may be a requirement for the frequency of high-frequency repetitions.

[0121] In another example, the filtering submodule 303 can determine the common subexpressions of the regular rules in the regular rule set based on the number of repetitions of each subexpression obtained statistically and the complexity of the subexpressions. For example, for any subexpression, the number of repetitions of the subexpression can be multiplied by the preset complexity of the subexpression, and the subexpression whose multiplication result satisfies the second predetermined requirement can be taken as the common subexpression. The complexity of the subexpression represents the resources occupied by performing a matching calculation for the subexpression once. The second predetermined requirement may include the requirement for the total resources occupied by performing the matching calculation of the number of repetitions of the subexpression.

[0122] The packet information acquisition module 310 is mainly used to acquire information about data packets that need to be processed for traffic collection and analysis.

[0123] The judgment module 320 is mainly used to determine whether the subexpression to be matched and calculated is a common subexpression during the process of performing regular expression rule matching calculation on the data packet information to be processed based on the regular expression rules in the set of regular expression rules.

[0124] The execution storage module 330 is mainly used to perform a matching calculation on the subexpression to be matched when the judgment result of the judgment module 320 is that the subexpression to be matched and calculated is a common subexpression, and the matching calculation result of the common subexpression is not currently stored. The execution storage module 330 stores the identifier of the common subexpression (which can also be the identifier of the data packet and the identifier of the common subexpression) and the matching calculation result of the common subexpression as key-value pairs.

[0125] The result reuse module 340 is mainly used to retrieve the matching result of the current subexpression to be matched from the stored matching results when the judgment result of the judgment module 320 is that the subexpression to be matched and calculated is a common subexpression, and the matching result of the common subexpression is currently stored. For example, using the identifier of the current subexpression to be matched and calculated as the key (or the identifier of the data packet and the identifier of the current subexpression to be matched and calculated as the key), it searches for key-value pairs that match the key, and uses the value in the found matching key-value pairs as the matching result of the current subexpression to be matched and calculated.

[0126] The clustering module 350 is mainly used to cluster regular rules with the same common subexpression in the regular rule set, forming one or more regular rule classes, with one common subexpression corresponding to one regular rule class. The information of the one or more regular rule classes formed by the clustering module 350 can be provided to the judgment module 320. For example, the judgment module 320 can first determine the class of the regular rule to which the current subexpression to be matched belongs. If the class is not empty, the judgment module 320 continues to determine whether the current subexpression to be matched is a common subexpression corresponding to the class. If the class is empty, the judgment module 320 determines that the current subexpression to be matched is not a common subexpression.

[0127] The rule loading module 360 ​​is mainly used to first remove duplicate common subexpressions from each regular expression rule in the regular expression rule set according to the regular expression rule class, forming a new regular expression rule set; in the new regular expression rule set, only one regular expression rule in all regular expression rules belonging to the same regular expression rule class contains a common subexpression; then, the rule loading module 360 ​​performs syntax parsing on each regular expression rule in the new regular expression rule set to obtain the abstract syntax tree of each regular expression rule, and constructs the state based on the abstract syntax tree of each regular expression rule to obtain the state machine of the regular expression rule. This state machine is used for subsequent matching calculation. For example, the execution storage module 330 uses the state machine to perform matching calculation on the subexpression to be matched.

[0128] The lifecycle management module 370 is mainly used to delete the matching calculation results of the common sub-expressions stored for the data packet information to be processed after the traffic collection and analysis of the data packet information to be processed is completed.

[0129] The specific operations performed by each of the above modules and sub-modules can be found in the relevant descriptions in the above method embodiments, and will not be described in detail here.

[0130] Exemplary electronic devices

[0131] The following is for reference. Figure 4 To describe an electronic device according to embodiments of the present disclosure. Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (As follows) Figure 4 As shown, the electronic device 41 includes one or more processors 411 and memory 412.

[0132] The processor 411 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 41 to perform desired functions.

[0133] The memory 412 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, and flash memory. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 411 may execute the program instructions to implement the methods for implementing traffic acquisition and analysis and / or other desired functions described in the various embodiments of this disclosure above.

[0134] In one example, electronic device 41 may further include input device 413 and output device 414, etc., these components being interconnected via a bus system and / or other forms of connection mechanism (not shown). Furthermore, the input device 413 may also include, for example, a keyboard, mouse, etc. The output device 414 can output various information to the outside. The output device 414 may include, for example, a display, speaker, printer, and communication networks and their connected remote output devices, etc.

[0135] Of course, for the sake of simplicity, Figure 4 Only some of the components of the electronic device 41 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 41 may include any other suitable components depending on the specific application.

[0136] Exemplary computer program products and computer-readable storage media

[0137] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the methods for implementing traffic acquisition and analysis according to various embodiments of this disclosure as described in the "Exemplary Methods" section of this specification.

[0138] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0139] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the methods for implementing traffic acquisition and analysis according to various embodiments of this disclosure as described in the "Exemplary Methods" section above.

[0140] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (not an exhaustive list) may include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0141] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0142] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0143] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0144] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0145] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0146] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0147] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A method for implementing traffic collection analysis, characterized by, The method comprises: performing common sub-expression extraction processing on each regular rule in the regular rule set to obtain one or more common sub-expressions; wherein the common sub-expression is a sub-expression contained in multiple regular rules in the regular rule set; obtaining to-be-processed data packet information that needs to be subjected to traffic collection analysis; in a process of performing regular rule matching calculation on the to-be-processed data packet information based on each regular rule in the regular rule set, judging whether a current to-be-matched sub-expression is a common sub-expression; in a case where it is judged that the current to-be-matched sub-expression is a common sub-expression and the matching calculation result of the common sub-expression is not currently stored, performing matching calculation on the current to-be-matched sub-expression and storing the matching calculation result as the matching calculation result of the common sub-expression; in a case where it is judged that the current to-be-matched sub-expression is a common sub-expression and the matching calculation result of the common sub-expression is currently stored, obtaining the matching calculation result of the current to-be-matched sub-expression from the stored matching calculation result.

2. The method of claim 1, wherein, The common sub-expression extraction processing on the regular rules in the regular rule set comprises: performing syntax analysis processing on the regular rules in the regular rule set to obtain an abstract syntax tree of each regular rule; performing a sub-expression repetition number counting operation according to the abstract syntax tree of each regular rule; determining the common sub-expressions of the regular rules in the regular rule set according to the repetition numbers of each sub-expression obtained by counting.

3. The method of claim 2, wherein, The determination of the common sub-expressions of the regular rules in the regular rule set according to the repetition numbers of each sub-expression obtained by counting comprises: taking a sub-expression whose repetition number meets a first predetermined requirement as a common sub-expression; or determining the common sub-expressions of the regular rules in the regular rule set according to the repetition numbers of each sub-expression obtained by counting and the complexity of the sub-expressions; The determination of the common sub-expressions of the regular rules in the regular rule set according to the repetition numbers of each sub-expression obtained by counting and the complexity of the sub-expressions comprises: for any sub-expression, multiplying the repetition number of the sub-expression by the complexity of the sub-expression preset for the sub-expression, and taking a sub-expression whose multiplication result meets a second predetermined requirement as a common sub-expression; wherein the first predetermined requirement comprises a requirement for a high-frequency repetition number; the complexity of the sub-expression represents a resource condition occupied by a single execution of matching calculation on the sub-expression; and the second predetermined requirement comprises a requirement for a total resource condition occupied by repeated execution of matching calculation on the sub-expression.

4. The method of claim 1, wherein, The method further comprises: clustering the regular rules in the regular rule set that have the same common sub-expressions to form one or more regular rule classes; wherein one common sub-expression corresponds to one regular rule class; The judgment of whether the current to-be-matched sub-expression is a common sub-expression comprises: determining a class to which a regular rule to which the current sub-expression to be matched and calculated belongs; in a case where the class to which the regular rule belongs is not empty, determining whether the current sub-expression to be matched and calculated is a common sub-expression corresponding to the class to which the regular rule belongs; in a case where the class to which the regular rule belongs is empty, determining that the current sub-expression to be matched and calculated is not a common sub-expression.

5. The method of claim 4, wherein, The method further comprises, before the step of obtaining the to-be-processed data packet information requiring flow collection analysis, a regular rule loading step, and the regular rule loading step comprises: performing repeated common sub-expression stripping processing on each regular rule in the regular rule set according to the regular rule class, to form a new regular rule set; wherein only one regular rule in all regular rules belonging to the same regular rule class in the new regular rule set contains a common sub-expression; performing syntax analysis processing on each regular rule in the new regular rule set respectively, to obtain an abstract syntax tree of each regular rule; performing state construction according to the abstract syntax tree of each regular rule, to obtain a state machine of the regular rule; the matching calculation performed on the current sub-expression to be matched and calculated comprises: performing matching calculation on the current sub-expression to be matched and calculated by using the state machine.

6. The method according to any one of claims 1 to 5, characterized in that, the matching calculation result of the common sub-expression is stored as the matching calculation result of the common sub-expression. the identification of the common sub-expression and the matching calculation result of the common sub-expression are stored in the form of a key-value pair; and the matching calculation result of the current sub-expression to be matched and calculated is obtained from the stored matching calculation result, comprising: taking the identification of the current sub-expression to be matched and calculated as a key, searching for a key-value pair matching the key, and taking the value in the matching key-value pair as the matching calculation result of the current sub-expression to be matched and calculated.

7. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: in a case where the flow collection analysis of the to-be-processed data packet information is completed, deleting the matching calculation result of the common sub-expression stored for the to-be-processed data packet information.

8. An apparatus for enabling traffic collection analysis, characterized by The device comprises: an extraction processing module configured to perform common sub-expression extraction processing on each regular rule in the regular rule set, to obtain one or more common sub-expressions; wherein the common sub-expression is a sub-expression contained in multiple regular rules in the regular rule set; a packet information obtaining module configured to obtain to-be-processed data packet information requiring flow collection analysis; a judgment module configured to, in a process of performing regular rule matching calculation on the to-be-processed data packet information based on each regular rule in the regular rule set, determine whether a current sub-expression to be matched and calculated is a common sub-expression; an execution and storage module configured to, in a case where the judgment module determines that the current sub-expression to be matched and calculated is a common sub-expression, and in a case where no matching calculation result of the common sub-expression is currently stored, perform matching calculation on the current sub-expression to be matched and calculated, and store the matching calculation result as the matching calculation result of the common sub-expression. A result multiplexing module is configured to, in a case where the judging module judges that the current sub-expression to be matched and calculated is a common sub-expression and the matching and calculating result of the common sub-expression is stored, acquire the matching and calculating result of the current sub-expression to be matched and calculated from the stored matching and calculating result. 9.A computer readable storage medium, the storage medium storing a computer program, the computer program being configured to execute the method of any one of claims 1-7. 10.An electronic device, the electronic device comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method of any one of claims 1-7.

Images