A Smart Selection Method and System for a Multi-Pattern Matching Engine

Abstract

This invention relates to an intelligent selection method and system for multi-pattern matching engines. The method includes: acquiring the features of currently loaded detection rules and statistical features of network traffic; predicting the expected performance indicators of each engine based on the detection rule features and statistical features; constructing a multi-objective optimization function; using the expected performance indicators to solve for the single engine or engine combination that maximizes the multi-objective optimization function as the objective; and obtaining the decision result. This invention solves the problem that a single engine cannot adapt to all scenarios by intelligently selecting the optimal engine or using multiple engines in combination, significantly improving the throughput and resource utilization efficiency of intrusion detection / prevention systems.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and in particular to an intelligent selection method and system for a multi-pattern matching engine. Background Technology

[0002] As cyberattack methods become increasingly sophisticated, the massive amounts of network traffic and complex detection rules that intrusion detection / prevention systems (IDS / IPS) need to handle are growing exponentially. Pattern matching, as a core technology of IDS / IPS, directly impacts the throughput and detection latency of the entire system.

[0003] Currently, the mainstream pattern matching engines include: 1. AC-BM (Aho-Corasick + Boyer-Moore) engine: The classic algorithm used by open-source IDS such as Snort. It is suitable for small-scale rule sets and has low memory consumption, but its performance drops significantly when dealing with large-scale rules.

[0004] 2. Hyperscan Engine: A high-performance regular expression matching library developed by Intel. It is optimized based on SIMD instructions and is suitable for large-scale rule sets, but it has high memory consumption and long compilation time.

[0005] 3. HFA (Hybrid Finite Automaton) engine: Hybrid finite automaton combines the advantages of DFA and NFA, but it is complex to implement and difficult to optimize.

[0006] The shortcomings of the engine selection methods disclosed in existing technical solutions are: 1. A single engine cannot adapt to all scenarios.

[0007] 2. When multiple engines coexist, there is a lack of intelligent selection mechanism, and a single engine is often simply and fixedly used.

[0008] 3. Unable to dynamically adjust based on real-time traffic characteristics and rule characteristics.

[0009] 4. It can lead to resource waste or performance bottlenecks. Summary of the Invention

[0010] The purpose of this invention is to provide an intelligent selection method and system for a multi-pattern matching engine, aiming to solve the technical problem that existing multi-pattern matching engines cannot adaptively select based on the scenario.

[0011] To achieve the above objectives, the present invention provides the following solution: A smart selection method for a multi-pattern matching engine includes: Obtain the characteristics of the currently loaded detection rules and the statistical characteristics of network traffic, and predict the expected performance indicators of each engine based on the detection rule characteristics and the statistical characteristics; A multi-objective optimization function is constructed, with the goal of maximizing a single engine or a combination of engines. The expected performance index is used to solve the problem and obtain the decision result.

[0012] Optionally, the detection rule features include: rule quantity features, rule complexity features, and rule update frequency; The rule quantity characteristics include: the total number of rules and the rule distribution for each protocol type; the rule complexity characteristics include: the proportion of regular expressions, the nesting depth, and the backtracking risk index; the rule update frequency includes: the temporal characteristics of adding / deleting rules.

[0013] Optionally, the statistical features include: packet-level features, stream-level features, and content features; The packet-level features include: average packet size, packet size distribution, and entropy value of packet arrival interval; the stream-level features include: number of concurrent connections, rate of new connection establishment, and proportion of long connections; and the content features include: proportion of encrypted traffic, proportion of compressed content, and distribution of protocol types.

[0014] Optionally, predicting the expected performance metrics of each engine based on the detection rule features and the statistical features includes: The detection rule features and statistical features are input into the pre-trained engine performance model to obtain the expected performance indicators of each engine. The engine performance model uses a regression model based on historical data. ; in, To predict processing time, For the number of rules, This represents the percentage of regular expression rules. For data packet size distribution, This is a load-balanced state.

[0015] Optionally, constructing the multi-objective optimization function includes: ; in, For a multi-objective optimization function, , , These are three weighting coefficients, corresponding to the relative importance of the three optimization objectives: throughput, memory usage, and latency. For the target throughput, For actual throughput, Due to memory limitations, This represents the actual memory usage. To handle delays.

[0016] Optionally, during the solution process using the expected performance indicators, constraints are set, namely, throughput is not less than the system line speed requirement, memory usage is not greater than the system available memory threshold, and rule compilation latency is not greater than the business tolerance threshold.

[0017] Optionally, obtaining the decision result includes: When the decision result is an engine combination, the rule set is divided into multiple subsets, and different engines are used for different subsets. That is, when the proportion of regular rules in the rule set is greater than the target proportion, the regular rules are given the Hyperscan engine and the literal rules are given the AC-BM engine. When the total number of rules in the rule set is greater than the target value, the high-priority rules are given the HFA engine and the other rules are given the AC-BM engine. When the proportion of regular rules in the rule set is less than the target proportion or the total number is less than the target value, all rules are given the AC-BM engine.

[0018] Optionally, the method further includes: comparing the expected performance index with a preset predicted value to update the performance prediction model parameters.

[0019] To achieve the above objectives, the present invention also provides an intelligent selection system for a multi-mode matching engine, comprising: The feature acquisition module is used to acquire the features of the currently loaded detection rules and the statistical features of network traffic; A performance prediction engine is used to predict the expected performance metrics of each engine based on the detection rule features and the statistical features. An intelligent decision engine is used to construct a multi-objective optimization function, with the goal of maximizing a single engine or a combination of engines within the multi-objective optimization function, and to solve the problem using the expected performance indicators to obtain the decision result.

[0020] The beneficial effects of this invention are as follows: 1. Performance improvement: Compared with using a single engine, this invention can select the optimal engine according to the scenario, improving throughput by 30%-200%.

[0021] 2. Resource optimization: This invention avoids memory waste in the Hyperscan engine in simple scenarios, or performance bottlenecks in AC-BM in complex scenarios.

[0022] 3. Adaptability: This invention requires no manual tuning; the system automatically adapts to rule changes and traffic fluctuations.

[0023] 4. Smooth Evolution: This invention supports hot-swapping of new engines, protecting technological investments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart of an intelligent selection method for a multi-pattern matching engine according to an embodiment of the present invention. Detailed Implementation

[0026] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] like Figure 1 As shown in the figure, this embodiment discloses an intelligent selection method for a multi-pattern matching engine, including: obtaining the characteristics of the currently loaded detection rules and the statistical characteristics of network traffic; predicting the expected performance indicators of each engine based on the detection rule characteristics and statistical characteristics; constructing a multi-objective optimization function, taking the single engine or engine combination that maximizes the multi-objective optimization function as the objective, solving it using the expected performance indicators, and obtaining the decision result.

[0029] Specifically, this embodiment discloses an intelligent selection method for a multi-pattern matching engine, including: extracting multi-dimensional features from the detection rule set and calculating a complexity score; collecting network traffic statistical features in real time; predicting the performance indicators of each engine based on a pre-trained model; constructing a multi-objective optimization function to select the optimal engine or engine combination; performing pattern matching and continuously providing feedback optimization. This invention solves the problem that a single engine cannot adapt to all scenarios by intelligently selecting the optimal engine or using multiple engines in combination, significantly improving the throughput and resource utilization efficiency of intrusion detection / prevention systems.

[0030] Furthermore, the detection rule features include: rule quantity features, rule complexity features, and rule update frequency; among which, rule quantity features include: total number of rules and rule distribution for each protocol type; rule complexity features include: regular expression ratio, nesting depth, and backtracking risk index; rule update frequency includes: temporal characteristics of added / deleted rules.

[0031] Specifically, the rule feature analysis stage: Multi-dimensional feature extraction is performed on the currently loaded detection rule set, including: Rule quantity characteristics: total number of rules, rule distribution for each protocol type; Rule complexity characteristics: proportion of regular expressions, nesting depth, and backtracking risk index; Rule update frequency: the temporal characteristics of adding / deleting rules.

[0032] Computation of rule set complexity score based on extracted features: ; in, , , The weighting coefficient is determined based on historical optimal configuration experience. Nrules represents the total number of rules, in units of rules. Regex_Ratio represents the proportion of regular expression rules, with a value range of 0 to 1. UpdateFreq represents the rule update frequency, in units of times per day.

[0033] The purpose of the rule set complexity score is to quantitatively evaluate the complexity of the rule set, providing a basis for engine selection and resource scheduling decisions. A lower score indicates a simpler rule set, while a higher score indicates a more complex rule set.

[0034] Furthermore, the statistical features include: packet-level features, stream-level features, and content features; among which, packet-level features include: average packet size, packet size distribution, and entropy value of packet arrival interval; stream-level features include: number of concurrent connections, rate of new connection establishment, and proportion of long connections; and content features include: proportion of encrypted traffic, proportion of compressed content, and distribution of protocol types.

[0035] Specifically, the traffic characteristic collection stage: Real-time collection of statistical characteristics of network traffic, including: Packet-level features: average packet size, packet size distribution, and entropy value of packet arrival interval; Stream-level characteristics: number of concurrent connections, rate of new connection establishment, proportion of long-lived connections; Content characteristics: proportion of encrypted traffic, proportion of compressed content, and distribution of protocol types.

[0036] Furthermore, predicting the expected performance metrics of each engine based on detection rule features and statistical features includes: inputting the detection rule features and statistical features into a pre-trained engine performance model to obtain the expected performance metrics of each engine.

[0037] Specifically, based on the pre-trained engine performance model, the expected performance indicators of each engine under the current rule set and traffic characteristics are predicted, and the predicted expected performance indicators are used as the basis for solving the optimization problem below: 1. AC-BM: Predicted throughput T_ac, memory usage M_ac, CPU usage C_ac; 2. Hyperscan: Predicts throughput T_hs, memory usage M_hs, and compilation time Compile_hs; 3. HFA: Predicted throughput T_hfa, memory usage M_hfa; The performance prediction model uses a regression model based on historical data: ; in, To predict processing time, For the number of rules, This represents the percentage of regular expression rules. For data packet size distribution, This is a load-balanced state.

[0038] Parameter Details: `Tpredict` represents the predicted packet processing time, typically in microseconds or milliseconds. This metric directly reflects the time required for the engine to process a single packet. `Nrules` represents the total number of currently loaded detection rules. The number of rules directly affects the matching computation; the more rules, the more patterns the engine needs to traverse and match, resulting in longer processing time. `Regex_Ratio` represents the proportion of regular expression rules in the rule base, ranging from 0 to 1. `Pkt_Size_Dist` represents the size distribution characteristics of the input packet. Packet size has a significant impact on processing time; small packets are processed quickly, while large packets are processed slowly. This parameter is not a single value but a distribution characteristic, including statistical indicators such as average packet length, maximum packet length, and packet length variance. In real-world network environments, HTTP request packet sizes vary greatly; a simple GET request may only be a few hundred bytes, while a file upload request may reach several megabytes. `Pkt_Size_Dist` helps the model predict processing time under different traffic characteristics. `Load_Balancing` represents the current system load balancing status, reflecting the task allocation among multi-core CPUs. The value range is usually between 0 and 1, where 0 indicates that the load is completely unbalanced (one core is fully loaded while other cores are idle), and 1 indicates that the load is completely balanced (all cores have the same load).

[0039] Furthermore, during the solution process using expected performance indicators, constraints are set, namely, throughput is not less than the system line speed requirement, memory usage is not greater than the system available memory threshold, and rule compilation latency is not greater than the business tolerance threshold.

[0040] Specifically, the intelligent decision-making stage: Construct a multi-objective optimization function that comprehensively considers performance, resources, and latency: ; in, For a multi-objective optimization function, , , These are three weighting coefficients, corresponding to the relative importance of the three optimization objectives: throughput, memory usage, and latency. For the target throughput, For actual throughput, Due to memory limitations, This represents the actual memory usage. To handle delays.

[0041] Detailed parameter description: , , These are three weighting coefficients, corresponding to the relative importance of the three optimization objectives: throughput, memory usage, and latency. The values ​​of all three coefficients range from 0 to 1 and satisfy the following conditions: add add =1. The larger the weight coefficient, the higher the priority of the target in optimization. For example, if the business scenario is latency-sensitive (such as a financial transaction system), w3 can be increased; if the business traffic is high (such as a video website), w1 can be increased; if hardware resources are limited (such as edge devices), w2 can be increased. By adjusting the weight coefficient, different business needs and hardware environments can be flexibly adapted. Ttarget represents the target throughput that the system expects to achieve, usually in Mbps or Gbps. Tactual represents the actual throughput that the system can achieve under the current configuration. This value is obtained through performance testing or online monitoring and is affected by factors such as the number of rules, engine type, and hardware performance. Mlimit represents the maximum available memory capacity of the system, in MB or GB. This value is determined by the hardware configuration. Mactual represents the actual memory size occupied by the current engine configuration. The memory usage of different engines varies greatly. For example, the AC automaton engine needs to pre-compute the state transition table, resulting in high memory usage; the regular expression engine compiles on demand, resulting in low memory usage. The ratio of Mlimit to M_actual reflects memory usage efficiency. The larger the ratio, the lower the memory pressure and the more stable the system. Latency represents the processing delay of data packets, measured in milliseconds. This is a penalty term in the optimization function; the higher the latency, the smaller the objective function value, and the optimization tends to choose the lowest latency solution.

[0042] Constraints: Throughput ≥ System line speed requirement; Memory usage ≤ System available memory threshold; Rule compilation latency ≤ Business tolerance threshold; Solve the optimization problem and select the engine or engine combination that maximizes Objective.

[0043] Furthermore, obtaining the decision results includes: when the decision result is an engine combination, the rule set is divided into multiple subsets, and different engines are used for different subsets. That is, when the proportion of regular rules in the rule set is greater than the target proportion, the regular rules are used with the Hyperscan engine, and the literal rules are used with the AC-BM engine. When the total number of rules in the rule set is greater than the target value, the high-priority rules are used with the HFA engine, and other rules are used with the AC-BM engine. When the proportion of regular rules in the rule set is less than the target proportion or the total number is less than the target value, all rules are used with the AC-BM engine.

[0044] Specifically, it will be implemented based on the decision outcome: Single-engine mode: Selects the optimal single engine to execute all rules; Hybrid mode: Divide the rule set into multiple subsets, use different engines for different subsets, and then merge the results; The rule-based division strategy for hybrid modes is as follows: Condition 1: If the proportion of rules containing regular expressions is greater than 60%, use Hyperscan for regular expression rules and AC-BM for literal rules. Under condition 2, if the total number of rules is greater than 10,000, high-priority rules use HFA and other rules use AC-BM. Under condition 3, and without condition 1 and condition 2, all rules use AC-BM; The method also includes comparing the expected performance indicators with preset predicted values ​​to update the performance prediction model parameters.

[0045] Specifically, in the feedback optimization phase: continuously monitor actual performance indicators, compare them with predicted values, update the parameters of the performance prediction model, and achieve closed-loop optimization.

[0046] In one embodiment, the engine selection is rule-feature driven: Scenario: A company network deploys 10,000 IPS rules, 30% of which contain regular expressions. Steps: 1. Rule feature analysis: N_rules = 10,000, Regex_Ratio = 30%, Complexity_Score = 0.5×10000 + 0.3×30 = 5009.

[0047] 2. The complexity score of 5009 exceeds the threshold of 3000, and is therefore classified as a high-complexity rule set.

[0048] 3. Traffic characteristics collection: average packet size = 512 bytes, concurrent connections = 50,000, traffic load = 10Gbps.

[0049] 4. Performance prediction: AC-BM predicted throughput = 5Gbps (not meeting line speed), Hyperscan predicted throughput = 15Gbps (meets requirements).

[0050] 5. Decision result: Hyperscan engine selected.

[0051] 6. Actual operation monitoring: The throughput reached 12Gbps, with a deviation of 20% from the prediction, and the model parameters were updated.

[0052] In one embodiment, dynamic switching of traffic burst scenarios: Scenario: AC-BM engine is used during normal office hours; traffic characteristics change drastically during a sudden DDoS attack; Steps: 1. Normal period: 500 rules, 10% regular expression ratio, AC-BM selected, low memory usage.

[0053] 2. Attack Detection: The traffic feature acquisition module detected that the new connection rate suddenly increased from 100 / s to 10,000 / s, the proportion of small packets increased from 30% to 90%, and the entropy value decreased abnormally.

[0054] 3. Re-decision: Current AC-BM forecasts throughput will drop to 2Gbps, which does not meet demand.

[0055] 4. Engine switching: Hot-switch to the HFA engine to take advantage of its high throughput for small packets.

[0056] 5. After the attack ends: Traffic returns to normal, switch back to AC-BM to release memory.

[0057] In one embodiment, the hybrid mode is executed as follows: Scenario: A financial data center, with a rule set containing 20,000 rules, using a combination of multiple detection technologies; Steps: 1. Automatic rule classification: Category A: Pure string matching rules (12,000 rules) → AC-BM, Category B: Complex regular expression rules (5,000 rules) → Hyperscan, Category C: Protocol rules requiring state tracking (3,000 rules) → HFA.

[0058] 2. Parallel compilation: The three engines compile their respective rule subsets simultaneously.

[0059] 3. Runtime execution: Each data packet enters three engines simultaneously, and each engine only matches a subset of the assigned rules. Results are merged: an alarm is triggered if any engine hits the target.

[0060] 4. Performance comparison: Single Hyperscan: 8Gbps throughput, 8GB memory; Mixed mode: 12Gbps throughput, 5GB memory.

[0061] This embodiment also provides an intelligent selection system for a multi-mode matching engine, including: The feature acquisition module is used to acquire the features of the currently loaded detection rules and the statistical features of network traffic; The performance prediction engine is used to predict the expected performance metrics of each engine based on detection rule features and statistical features. The intelligent decision engine is used to construct a multi-objective optimization function. It aims to maximize the multi-objective optimization function using a single engine or a combination of engines, and solves the problem using expected performance indicators to obtain the decision results.

[0062] Specifically, this embodiment also provides an intelligent selection system for a multi-mode matching engine, including: Rule Feature Analysis Module: Parses rule configuration files, extracts rule metadata, and calculates complexity scores.

[0063] Traffic feature acquisition module: Collects network traffic statistics in real time through traffic probes.

[0064] Performance prediction engine: Loads pre-trained machine learning models, inputs rules and traffic features, and outputs performance predictions for each engine.

[0065] Intelligent Decision Engine: The core decision-making unit, which runs multi-objective optimization algorithms and outputs engine selection strategies.

[0066] Engine Management Module: Unifies the lifecycle management of multiple engines such as AC-BM, Hyperscan, and HFA, and supports hot switching.

[0067] Hybrid execution framework: supports rule subset partitioning, multi-engine parallel execution, and result merging.

[0068] Feedback optimization module: Collects actual performance data and updates the prediction model online.

[0069] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A smart selection method for a multi-pattern matching engine, characterized in that, include: Obtain the characteristics of the currently loaded detection rules and the statistical characteristics of network traffic, and predict the expected performance indicators of each engine based on the detection rule characteristics and the statistical characteristics; A multi-objective optimization function is constructed, with the goal of maximizing a single engine or a combination of engines. The expected performance index is used to solve the problem and obtain the decision result.

2. The intelligent selection method of the multi-pattern matching engine according to claim 1, characterized in that, The detection rule features include: rule quantity feature, rule complexity feature, and rule update frequency; The rule quantity characteristics include: the total number of rules and the rule distribution for each protocol type; the rule complexity characteristics include: the proportion of regular expressions, the nesting depth, and the backtracking risk index; the rule update frequency includes: the temporal characteristics of adding / deleting rules.

3. The intelligent selection method of the multi-pattern matching engine according to claim 1, characterized in that, The statistical features include: packet-level features, stream-level features, and content features; The packet-level features include: average packet size, packet size distribution, and entropy value of packet arrival interval; the stream-level features include: number of concurrent connections, rate of new connection establishment, and proportion of long connections; and the content features include: proportion of encrypted traffic, proportion of compressed content, and distribution of protocol types.

4. The intelligent selection method for the multi-pattern matching engine according to claim 1, characterized in that, The expected performance metrics of each engine are predicted based on the detection rule features and the statistical features, including: The detection rule features and statistical features are input into the pre-trained engine performance model to obtain the expected performance indicators of each engine. The engine performance model uses a regression model based on historical data. ； in, To predict processing time, For the number of rules, This represents the percentage of regular expression rules. For data packet size distribution, This is a load-balanced state.

5. The intelligent selection method for the multi-pattern matching engine according to claim 1, characterized in that, Constructing the multi-objective optimization function includes: ； in, For multi-objective optimization functions, , , These are three weighting coefficients, corresponding to the relative importance of the three optimization objectives: throughput, memory usage, and latency. For the target throughput, For actual throughput, Due to memory limitations, This represents the actual memory usage. To handle delays.

6. The intelligent selection method for the multi-pattern matching engine according to claim 1, characterized in that, During the solution process using the expected performance indicators, constraints are set, namely, throughput is not less than the system line speed requirement, memory usage is not greater than the system available memory threshold, and rule compilation latency is not greater than the business tolerance threshold.

7. The intelligent selection method for the multi-pattern matching engine according to claim 1, characterized in that, Obtaining the decision result includes: When the decision result is an engine combination, the rule set is divided into multiple subsets, and different engines are used for different subsets. That is, when the proportion of regular rules in the rule set is greater than the target proportion, the regular rules are given the Hyperscan engine and the literal rules are given the AC-BM engine. When the total number of rules in the rule set is greater than the target value, the high-priority rules are given the HFA engine and the other rules are given the AC-BM engine. When the proportion of regular rules in the rule set is less than the target proportion or the total number is less than the target value, all rules are given the AC-BM engine.

8. The intelligent selection method for the multi-pattern matching engine according to claim 1, characterized in that, The method further includes: comparing the expected performance index with a preset predicted value to update the performance prediction model parameters.

9. An intelligent selection system for a multi-pattern matching engine implemented according to any one of claims 1-8, characterized in that, include: The feature acquisition module is used to acquire the features of the currently loaded detection rules and the statistical features of network traffic; A performance prediction engine is used to predict the expected performance metrics of each engine based on the detection rule features and the statistical features. An intelligent decision engine is used to construct a multi-objective optimization function, with the goal of maximizing a single engine or a combination of engines within the multi-objective optimization function, and to solve the problem using the expected performance indicators to obtain the decision result.

Images