A regular matching optimization method and system based on traffic collection analysis

Abstract

This invention provides a regular expression matching optimization method and system based on traffic collection and analysis, belonging to the field of network security. The method first sets complexity evaluation indicators and thresholds to classify rules into low complexity and high complexity. If a rule is determined to be low complexity, it is marked as a DFA-adaptive rule, and an NFA→DFA conversion is performed. After state compression, the rules are pre-cached in a tiered manner and directly matched. If a rule is determined to be high complexity, it is marked as an NFA-adaptive rule, and safety thresholds for the number of backtracking attempts and matching time are configured. During matching, the number of backtracking attempts and matching time of the NFA engine are collected. If either reaches the safety threshold, a circuit breaker mechanism is triggered, interrupting rule matching, marking the data packet as pending review, and marking the rule as low priority. Rule optimization is performed according to system load, and the optimized rules are reclassified and reloaded. This invention improves service stability, ensures zero interruption of the data collection link, and eliminates security vulnerabilities, making it particularly suitable for high-concurrency API scenarios.

Description

Technical Field

[0001] This invention belongs to the field of network security, specifically relating to a regular expression matching optimization method and system based on traffic collection and analysis. Background Technology

[0002] With the rapid development of the digital economy, network traffic has experienced explosive growth. In scenarios such as large enterprise intranets and internet backbones, data packet processing volumes have reached hundreds of thousands or even millions per second. As a core entry point for security protection, traffic collection and analysis systems need to identify, filter data packets containing illegal content, and extract key identifiers from the collected traffic. Regular expression matching is a commonly used method for traffic collection and analysis.

[0003] In existing technologies, the regular expression matching technology solutions in traffic collection and analysis systems and security servers in the field of network security are based on the basic framework of "rule traversal - engine matching". Depending on the underlying matching engine, they are mainly divided into two mainstream technical paths, which correspond to different traffic processing scenario requirements and rely on two types of regular expression engines to complete the rule matching of data packets. Most lightweight traffic acquisition and analysis systems employ Nondeterministic Finite Automaton (NFA) engines, which rely on regular expressions for matching. They match data packets character by character from the beginning of the regular expression (such as text content in the payload and protocol identifiers in the header). This approach is suitable for scenarios requiring flexible matching of multi-format data packets and is considered a backtracking NFA engine solution. Some high-throughput scenarios (such as Internet backbone traffic acquisition and large data center analysis systems) use Deterministic Finite Automaton (DFA) engines, which perform linear scanning based on the data packet text. This approach requires converting the regular expression into a DFA state machine. During matching, state transitions are triggered sequentially according to the data packet character stream (such as header fields and payload fragments). There is no backtracking throughout the process, and the time complexity is consistently O(n, where n is the length of the data packet text fragment). This approach is suitable for handling large-scale, high-concurrency traffic acquisition tasks and is considered a deterministic DFA engine solution.

[0004] However, with the increasing concurrency of traffic collection and the growing complexity of rules, traffic collection and analysis systems face the following problems when implementing core functions such as traffic feature identification, sensitive information identification, and malicious behavior interception: DFA (Direct Automated Feature Analysis) encounters memory exhaustion or conversion timeouts during the preprocessing stage for highly complex rules, failing to load rules and directly hindering the normal startup and operation of the traffic collection and analysis system in high-traffic scenarios; NFA (Non-Automatic Feature Analysis) needs to try all possible matching paths when encountering extremely long data packets or special character combinations, resulting in an exponential increase in the number of backtracking steps, causing a surge in CPU utilization in a short period and posing a significant potential danger of cascading failures. Therefore, facing the rapidly increasing scale of network traffic, especially in high-concurrency API scenarios, current regular expression matching technology has significant vulnerabilities and cannot guarantee the secure operation of traffic collection and analysis. Summary of the Invention

[0005] In view of the above-mentioned defects or deficiencies in the existing technology, the present invention aims to provide a regular expression matching optimization method and system based on traffic collection and analysis. It constructs a full-process regular expression matching optimization scheme of "rule hierarchical - low-rule optimization - high-rule prevention and control", performs DFA transformation and state compression on low-complexity rules, adopts NFA adaptation for high-complexity rules, and is equipped with an automatic optimization and recovery mechanism after circuit breaking. This ensures service stability, zero interruption of the collection link, and no security vulnerabilities in API high-concurrency scenarios, and realizes the secure and large-scale application of the traffic collection and analysis system.

[0006] To achieve the above objectives, the embodiments of the present invention adopt the following technical solutions: In a first aspect, embodiments of the present invention provide a regular expression matching optimization method based on traffic collection and analysis, the method comprising the following steps: Step S1: Set the regularity rule complexity evaluation index and threshold, and divide the regularity rules into low complexity rules and high complexity rules; Step S2: Calculate the evaluation index of the current regular expression rule and determine the complexity classification of the current regular expression rule; if it is determined to be low complexity, it is used as a DFA adaptation rule and proceeds to step S3; if it is determined to be high complexity, it is used as an NFA adaptation rule and proceeds to step S4. Step S3: Perform NFA→DFA transformation on the DFA adaptation rules, then perform hierarchical pre-caching after state compression, and proceed to step S5; Step S4: For NFA adaptation rules, store core matching segments and configure the safety thresholds for backtracking times and matching time consumption respectively, then proceed to step S5; Step S5: Determine whether the current regular expression rule is the last one or the Nth evaluation; if it is the last one, load all regular expression rules and proceed to step S6; if it is the Nth evaluation, load the current regular expression rule and proceed to step S6; otherwise, increment the regular expression rule number by 1 and proceed to step S2; where N is a positive integer greater than 1. Step S6: When the rule is a DFA-adapted rule, match directly; when the rule is an NFA-adapted rule, collect the backtracking count and matching time of the NFA engine in real time. If either reaches the corresponding safety threshold, proceed to step S7; otherwise, perform NFA rule matching. Step S7: Interrupt the current rule matching, return to risk matching and mark the data packet for review. At the same time, mark the current regular expression rule as low priority and synchronize it to the rule attribute table. The low priority rule will be skipped when matching subsequent data packets. Step S8: Real-time assessment of the current system load and setting of the recovery threshold; when the load is less than the recovery threshold, automatically optimize low-priority rules and delete low-priority identifiers as the current regular expression rule, then proceed to step S2.

[0007] In a preferred embodiment of the present invention, the regular expression rule complexity evaluation index in step S1 includes: character set width, repeating quantifier level, nested quantifier depth and number of branches; for each of the four indexes, a first to a fourth threshold are set respectively.

[0008] In a preferred embodiment of the present invention, step S2 determines the complexity classification of the current regular expression rule, and the determination process includes: If the following conditions are met: character set width > first threshold, repetition quantifier level ≥ second threshold, nested quantifier depth ≥ third threshold, and / or number of branches > fourth threshold, then it is determined to be a high complexity rule; otherwise, it is determined to be a low complexity rule.

[0009] In a preferred embodiment of the present invention, step S3 involves hierarchical pre-caching, which is performed according to the frequency of use; the top 30% of the rules by frequency of use are stored in the memory resident cache, with a loading time of ≤10 milliseconds; the rules by frequency of use of 30% to 70% are stored in the SSD disk cache, with a loading time of ≤500 milliseconds.

[0010] In a preferred embodiment of the present invention, the core matching segment in step S4 includes a violation content feature code and a protocol key identifier.

[0011] In a preferred embodiment of the present invention, when configuring the security threshold in step S4, a personalized security threshold is configured for each NFA adaptation rule, specifically including: For rules containing nested quantifiers, the maximum matching time is 5ms and the maximum number of backtracking attempts is 100. For rules containing wide character set repetitions, the maximum matching time is 3ms and the maximum number of backtracking attempts is 80. For multi-branch rules, the maximum matching time is 4ms and the maximum number of backtracking attempts is 90.

[0012] In a preferred embodiment of the present invention, in step S8, the current system load is determined based on real-time monitoring of the system CPU, and the recovery threshold at this time is the CPU recovery threshold.

[0013] In a preferred embodiment of the present invention, when automatically optimizing low-priority rules in step S8, syntax optimization is adopted, including splitting nested quantifiers and simplifying the matching logic of violation features.

[0014] In a preferred embodiment of the present invention, when performing syntax optimization, a differentiated optimization method is adopted for rules with different feature tags, including: The "nested quantifier" rule is broken down into: <script>.*<\ / script>基础规则+"嵌套层数≤2”长度约束判断；对"多分支”规则，拆分为主协议与次要协议两条子规则，降低单条规则匹配复杂度。

[0015] 第二方面，本发明实施例还提供了一种基于流量采集分析的正则匹配优化系统，所述系统包括：初始配置模块、规则分类模块、DFA预缓存模块、NFA存储模块、NFA安全阈值设置模块、规则加载模块、数据包匹配模块、熔断执行模块和规则优化模块；其中，所述初始配置模块用于设置正则规则复杂性评估指标及阈值，将正则规则分为低复杂性规则和高复杂性规则；所述规则分类模块用于计算当前正则规则的评估指标，判断当前正则规则复杂性分类；；若判定为低复杂性规则，则作为DFA适配规则，并发送给DFA预缓存模块；若判定为高复杂性规则，则作为NFA适配规则，并发送给NFA存储模块；所述DFA预缓存模块用于对DFA适配规则执行NFA→DFA转换，再经状态压缩后进行分级预缓存；所述NFA存储模块用于对NFA适配规则，存储核心匹配片段；所述NFA安全阈值设置模块用于对NFA适配规则分别配置回溯次数和匹配耗时的安全阈值；所述规则加载模块用于判断当前正则规则是否为最后一条，或第N次评估；若为最后一条，则加载所有正则规则，启动数据包匹配模块；若为第N次评估，则加载当前正则规则，返回数据包匹配模块；否则，正则规则序号加1，启动规则分类模块；其中，N为大于1的正整数；所述数据包匹配模块用于当规则为DFA适配规则时，直接进行匹配；当规则为NFA适配规则时，实时采集NFA引擎的回溯次数与匹配耗时，若任一项达到对应的安全阈值，触发熔断执行模块；若未达到，则执行NFA规则匹配；所述熔断执行模块用于中断当前规则匹配，返回风险匹配并标记数据包待复核，同时将当前正则规则标记为低优先级并同步至规则属性表，且后续数据包匹配时跳过低优先级规则；所述规则优化模块用于实时判断当前系统负载情况并设置恢复阈值；当负载小于恢复阈值时，自动优化低优先级规则并删除低优先级标识，作为当前正则规则，启动规则分类模块。

[0016] 本发明实施例所提供的技术方案具有如下有益效果：本发明针对流量采集分析系统 API 高并发正则匹配场景，所提出的基于流量采集分析的正则匹配优化方法及系统，通过"多维度规则分级+低复杂规则精细优化+高复杂规则全周期防控”的组合方案，显著突破现有技术"无差别处理”的局限：其一，本发明通过"字符集宽度+重复限定符层级+嵌套量词深度+多分支数量”四维度量化指标精准划分规则等级，低复杂规则经 "Hopcroft 算法”实现DFA状态数降低，高复杂规则跳过完整DFA转换解决 DFA 状态爆炸问题；其二，增加"分级预缓存”机制，避免规则加载期间采集链路中断；其三，现有技术缺乏回溯监控与恢复机制，本发明为高复杂规则设计 "特征标签-个性化阈值” 绑定及 "毫秒级监控-熔断-自动优化-复用” 全周期流程，回溯风险发生率大大降低，降低了CPU占用，同时实现熔断规则自动优化复用，避免安全防护漏洞，更适配 API 高并发、高规则迭代的生产需求。

[0017] 当然，实施本发明的任一产品或方法并不一定需要同时达到以上所述的所有优点。附图说明

[0018] 为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来说，在不付出创造性劳动的前提下，还可以根据这些附图获得其他附图。

[0019] 图1是本发明实施例所述基于流量采集分析的正则匹配优化方法流程图。具体实施方式

[0020] 本申请发明人在发现上述问题后，对现有的正则匹配技术及流量采集分析过程进行了细致研究。研究发现，在API高并发业务场景中，流量采集分析系统对正则规则库的依赖度显著提升：一方面，为覆盖新型攻击手段(如变异嵌套恶意脚本)、适配新协议类型(如HTTP / 3.0)，规则库需持续迭代更新，导致高复杂性规则占比大幅增加。这类规则普遍包含三类复杂语法：一是宽字符集重复(如匹配长文本业务标识的[a-z0-9]{1,16}，字符集覆盖36种字符，远超常规规则的字符集宽度)；二是深度嵌套量词(如匹配多层违规脚本的((<script>.*< / script> (+), nesting levels of 2 or more require multi-level backtracking verification; thirdly, multi-branch structures (such as matching multiple protocols (http|https|ftp|sftp|tcp|udp): / / , with more than 5 branches, increasing the difficulty of matching path selection), these rules place extremely high demands on the resource capacity of the matching engine. On the other hand, API high-concurrency scenarios have extremely low tolerance for matching latency. The matching time for a single data packet needs to be controlled within milliseconds. Once the rule processing is delayed, it is very easy to cause traffic accumulation, leading to the interruption of the collection link, and thus causing security monitoring to fail. However, the mainstream NFA and DFA engine adaptation solutions cannot meet the needs of API high-concurrency scenarios.

[0021] While backtracking-based NFA engines offer the advantage of flexibly handling multi-format data packets, their character-by-character matching logic, primarily based on regular expressions, requires traversing all possible matching paths when faced with highly complex rules and extremely long data packets (such as violation payloads containing 1000+ characters). This leads to an exponential increase in backtracking counts, causing CPU usage to spike to over 90% in a short period, completely blocking the thread pool, preventing the processing of newly collected traffic, and ultimately triggering a system service avalanche, resulting in data collection link interruption and security monitoring failure. Furthermore, they cannot proactively interrupt the current matching task before backtracking risks spiral out of control, nor do they have a buffer logic for "temporarily downgrading rule priority." They can only passively restart after CPU overload or service crash. Even if a highly complex rule is found to frequently trigger backtracking, manual intervention to disable the rule is still required, which may take several minutes or even hours, resulting in prolonged data collection link interruption and complete failure of security monitoring. Moreover, there is no rule optimization and recovery mechanism after circuit breaking. Disabled highly complex rules are trapped in a vicious cycle of "disabling-crash," unable to be automatically optimized (such as splitting nested quantifiers or simplifying branch logic) and reintroduced into the matching queue, causing long-term security vulnerabilities.

[0022] Deterministic DFA engines achieve linear scanning through pre-built state machines. When adapting to high-throughput scenarios, the number of DFA states corresponding to highly complex rules grows exponentially. For example, with 20 [a-z0-9]{1,16} rules, the theoretical number of states can reach 36^16, requiring more than 10GB of memory storage, leading to frequent memory exhaustion or rule transformation timeouts during the preprocessing stage. Even for low-complexity rules like HTTP / 1.\d and \d {1,3}.\d {1,3}, traditional DFA only performs a "full transformation," without optimizing for state redundancy and repeated transformations. For example, the IP fragment rule \d {1,3}.\d {1,3} has 120 states after traditional transformation, 60% of which are duplicate prefix states (e.g., the preceding matching states of "192.168." and "10.0." can be merged), causing memory waste. Moreover, each system restart or rule update requires a re-transformation, with each transformation taking more than 10 seconds. If the rule iterates 10 times per hour... Each time, the conversion time alone exceeds 240 seconds per day, which seriously consumes CPU resources. Therefore, there are concerns about memory waste and loading time, and the inability to adapt to the low latency and high rule iteration requirements of API scenarios.

[0023] It should be noted that the defects in the above-mentioned prior art solutions are all results obtained by the inventors after practice and careful research. Therefore, the discovery process of the above problems and the solutions proposed by the embodiments of the present invention in the following text should be considered as contributions made by the inventors to the present invention.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. It should be noted that, without conflict, the embodiments and features in the embodiments of the present invention can also be combined with each other.

[0025] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of this invention, the terms "first," "second," "third," "fourth," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0026] After the above in-depth analysis, this invention provides a regular expression matching optimization method and system based on traffic collection and analysis. By constructing a multi-dimensional rule complexity evaluation model of "character set width + repetition level + nesting depth + number of branches", it accurately classifies low-complexity and high-complexity rules, providing a basis for differentiated processing. For low-complexity rules, "DFA transformation + state compression" is performed to reduce memory waste and loading time. For high-complexity rules, NFA adaptation is adopted, and a mechanism of "loading period threshold pre-calculation + runtime real-time monitoring + automatic optimization after circuit breaking" is provided. When the number of backtracking times / time consumption reaches the threshold, the task is actively interrupted. After the load decreases, nesting is automatically split, branches are simplified, and rule availability is restored. This ensures service stability, zero interruption of the collection link, and no security vulnerabilities in API high-concurrency scenarios, realizing the scalable and secure application of the traffic collection and analysis system.

[0027] like Figure 1 As shown, the regular expression matching optimization method based on traffic collection and analysis includes the following steps: Step S1: Based on the network scenario and traffic syntax characteristics, set the regular expression rule complexity evaluation index and threshold; divide the regular expression rules into low complexity rules and high complexity rules according to the evaluation index; and take the first regular expression rule in the current traffic collection and analysis scenario as the current regular expression rule.

[0028] In this step, based on grammatical features and the needs of high-traffic production scenarios, a two-tier optimization scheme of "rule preprocessing - security circuit breaking" is used to achieve high efficiency and stability in the regular expression matching process. During the rule preprocessing stage, the multi-dimensional features of the rules are quantified according to the core needs of the network scenario (such as protocol identification, violation filtering, and key identifier extraction). Regular expression rule complexity evaluation indicators are set, including: character set width, repeating quantifier level, nested quantifier depth, and number of multiple branches; corresponding to these four indicators, first to fourth thresholds are set respectively.

[0029] In one executable embodiment, the first threshold is set to 20; the second threshold is set to 3; the third threshold is set to 2; and the fourth threshold is set to 5.

[0030] Step S2: Calculate the evaluation index of the current regular expression rule, and determine the complexity classification of the current regular expression rule based on the evaluation index result and the corresponding threshold. If it is determined to be low complexity, it is used as a DFA adaptation rule and proceeds to step S3; if it is determined to be high complexity, it is used as an NFA adaptation rule and proceeds to step S4.

[0031] This step categorizes the current regular expression rule and performs differentiated processing on different categories. The specific judgment process includes: If the following conditions are met: character set width > first threshold, repetition quantifier level ≥ second threshold, nested quantifier depth ≥ third threshold, and / or number of branches > fourth threshold, then it is determined as a high complexity rule; otherwise, it is determined as a low complexity rule, ensuring that the classification is more accurately adapted to the needs of high-concurrency API scenarios.

[0032] Step S3: Perform NFA→DFA transformation on the DFA adaptation rules, then perform hierarchical pre-caching after state compression, and proceed to step S5.

[0033] This step optimizes resource consumption and improves loading and matching efficiency through a combination of NFA→DFA transformation, state compression, and hierarchical pre-caching. Specifically, it includes: Step S31: Perform NFA→DFA transformation on the low-complexity rules marked as "DFA adaptation rules" to ensure that the basic matching logic adapts to the linear scan advantage of DFA.

[0034] Step S32: Merge redundant states in the converted DFA.

[0035] In this step, the "Hopcroft algorithm" is used to compress the state of low-complexity rules during merging, reducing the number of optimized states and thus significantly reducing the memory usage of DFA.

[0036] Step S33: The low-complexity rule DFA after state compression is cached hierarchically according to the frequency of use.

[0037] Specifically, the top 30% of rules by usage frequency are stored in a persistent memory cache (the cache is valid indefinitely and is only refreshed when the rule is updated), with a loading time of ≤10 milliseconds; rules by usage frequency of 30%-70% are stored in an SSD disk cache (the cache is valid for 24 hours), with a loading time of ≤500 milliseconds; this avoids repeated conversions and adapts to the requirement of hourly rule iteration.

[0038] Step S4: For NFA adaptation rules, store core matching segments and configure the safety thresholds for backtracking times and matching time consumption, then proceed to step S5.

[0039] In this step, the core matching fragment includes violation content feature codes, protocol key identifiers, etc., which are called as needed during the data packet matching stage. This avoids the risk of memory exhaustion or loading timeout caused by forced DFA transformation of highly complex rules from the source, and avoids state explosion. For example, [a-z0-9]{1,16} is extracted as the basic fragment [a-z0-9] + the length constraint {1,16}, (( <script>.*<\ / script>)+)提取为<script>.*<\ / script>核心片段 + 嵌套层级约束)，并存储至规则库高复杂性专区。

[0040] 在一个可执行的实施例中，本步骤存储时还可以包括：为每条高复杂规则记录规则 ID 与特征标签(如 "宽字符集”"嵌套量词”"多分支”)，进行特征标签绑定，便于后续阈值预计算与优化，提升匹配阶段调用效率。通过将阈值与规则ID、特征标签绑定，存入MySQL规则属性表，以支持匹配阶段实时调用与动态更新。

[0041] 本步骤中，配置安全阈值时，结合规则语法特征与流量处理需求进行，并存储至规则属性表。

[0042] 所述安全阈值包括最大匹配耗时(如设为 5ms，适配每秒 10 万级数据包处理)和最大回溯次数(如设为 100 次)，最大匹配耗时作为匹配耗时的安全阈值，回溯次数作为最大回溯次数的安全阈值。

[0043] 进行安全阈值配置时，为每条NFA适配规则配置个性化安全阈值，具体包括：含嵌套量词的规则(如((<script>.*<\ / script>)+)，特征标签 "嵌套量词”)：最大匹配耗时5ms，最大回溯次数100次；含宽字符集重复的规则(如[a-z0-9]{1,16}，特征标签 "宽字符集”)：最大匹配耗时3ms，最大回溯次数80次；多分支规则(如(http|https|ftp|sftp|tcp):\ / \ / ，特征标签 "多分支”)：最大匹配耗时4ms，最大回溯次数90次。

[0044] 步骤S5，判断当前正则规则是否为最后一条，或第N次评估；若为最后一条，则加载所有正则规则，进入步骤S6；若为第N次评估，则加载当前正则规则，进入步骤S6；否则，正则规则序号加1，转入步骤S2；其中，N为大于1的正整数。

[0045] 步骤S6，数据包匹配阶段，当规则为DFA适配规则时，直接进行匹配；当规则为NFA适配规则时，实时采集NFA引擎的回溯次数与匹配耗时，若任一指标达到安全阈值，则转入步骤S7；若未达到，则执行NFA规则匹配。

[0046] 步骤S7，触发并执行熔断机制，具体包括：中断当前规则匹配，返回风险匹配并标记数据包待复核，同时将当前正则规则标记为低优先级并同步至规则属性表，且后续数据包匹配时跳过低优先级规则。

[0047] 本步骤中，对待复核数据包，通过人工复核进一步确认该数据包存在的问题；而后续数据包匹配时跳过标记为低优先级的规则，使新采集数据包优先匹配低复杂性规则DFA(如 HTTP 协议识别、IP片段提取)及其他正常NFA规则，优先保障正常业务流量处理。同时，本步骤通过将低优先级标识同步至规则属性表，确保所有匹配节点统一执行调度策略。

[0048] 步骤S8，实时判断当前系统负载情况并设置恢复阈值；当负载小于恢复阈值时，自动优化低优先级规则并删除低优先级标识，作为当前正则规则，转入步骤S2。

[0049] 本步骤中，判断当前系统负载情况，根据对系统CPU的实时监控来实现，此时的恢复阈值为CPU恢复阈值，如50%，当 CPU占用降至50%以下时，自动触发低优先级规则优化流程，避免CPU占用飙升导致的采集链路中断与安全监控失效。

[0050] 所述自动优化低优先级规则时，采用语法优化，且包括拆分嵌套量词、简化违规特征匹配逻辑。执行语法优化时，针对不同特征标签的规则采用差异化优化方式，包括：对"嵌套量词”规则(如((<script>.*<\ / script>)+))，拆分为<script>.*<\ / script>基础规则+"嵌套层数≤2”长度约束判断；对"多分支” 规则(如(http|https|ftp|sftp|tcp):\ / \ / )，拆分为主协议(http / https)与次要协议(ftp / sftp / tcp)两条子规则，降低单条规则匹配复杂度。

[0051] 通过返回步骤S2重新执行多维度复杂性评估，实现"熔断-优化-复用”闭环，避免安全防护漏洞。

[0052] 基于同样的思路，本发明实施例还提供了一种基于流量采集分析的正则匹配优化系统，所述系统包括：初始配置模块、规则分类模块、DFA预缓存模块、NFA存储模块、NFA安全阈值设置模块、规则加载模块、数据包匹配模块、熔断执行模块和规则优化模块；其中，所述初始配置模块用于设置正则规则复杂性评估指标及阈值，将正则规则分为低复杂性规则和高复杂性规则；所述规则分类模块用于计算当前正则规则的评估指标，判断当前正则规则复杂性分类；；若判定为低复杂性规则，则作为DFA适配规则，并发送给DFA预缓存模块；若判定为高复杂性规则，则作为NFA适配规则，并发送给NFA存储模块；所述DFA预缓存模块用于对DFA适配规则执行NFA→DFA转换，再经状态压缩后进行分级预缓存；所述NFA存储模块用于对NFA适配规则，存储核心匹配片段；所述NFA安全阈值设置模块用于对NFA适配规则分别配置回溯次数和匹配耗时的安全阈值；所述规则加载模块用于判断当前正则规则是否为最后一条，或第N次评估；若为最后一条，则加载所有正则规则，启动数据包匹配模块；若为第N次评估，则加载当前正则规则，返回数据包匹配模块；否则，正则规则序号加1，启动规则分类模块；其中，N为大于1的正整数；所述数据包匹配模块用于当规则为DFA适配规则时，直接进行匹配；当规则为NFA适配规则时，实时采集NFA引擎的回溯次数与匹配耗时，若任一项达到对应的安全阈值，触发熔断执行模块；若未达到，则执行NFA规则匹配；所述熔断执行模块用于中断当前规则匹配，返回风险匹配并标记数据包待复核，同时将当前正则规则标记为低优先级并同步至规则属性表，且后续数据包匹配时跳过低优先级规则；所述规则优化模块用于实时判断当前系统负载情况并设置恢复阈值；当负载小于恢复阈值时，自动优化低优先级规则并删除低优先级标识，作为当前正则规则，启动规则分类模块。

[0053] 本实施例中各模块通过处理器实现，当需要存储时适当增加存储器。其中，所述处理器可以是但不限于微处理器MPU、中央处理器(Central Processing Unit，CPU)、网络处理器 (Network Processor，NP)、数字信号处理器(DSP)、专用集成电路(ASIC)、现场可编程门阵列(FPGA)、其他可编程逻辑器件、分立门、晶体管逻辑器件、分立硬件组件等。所述存储器可以包括随机存取存储器(Random Access Memory，RAM)，也可以包括非易失性存储器(Non-Volatile Memory，NVM)，例如至少一个磁盘存储器。可选的，存储器还可以是至少一个位于远离前述处理器的存储装置。

[0054] 在上述实施例中，可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本发明实施例所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波等)方式向另一个网站站点、计算机、服务器或数据中心进行传输。

[0055] 另外需要说明的是，本实施例所述基于流量采集分析的正则匹配优化系统与所述基于流量采集分析的正则匹配优化方法是对应的，对所述方法的描述与限定，同样适用于所述系统，在此不再赘述。

[0056] 将本发明实施例所述的基于流量采集分析的正则匹配优化方法及系统，应用于某流量采集分析场景下，参数如下：cpu: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz；mem: 8GB DDR3基于所设置的评估指标和规则分类，对 100 条初始规则逐一评估，最终得到80条低复杂性规则和20条高复杂性规则。其中，低复杂性规则包括 HTTP 协议识别HTTP\ / 1\.\d(字符集宽度=5，无重复 / 嵌套 / 多分支)、IP片段提取\d{1,3}\.\d{1,3}(重复层级 = 1，无宽字符集)、端口匹配:\d{1,5}(字符集宽度 = 5，多分支数量 = 0)等；高复杂性规则包括长标识匹配[a-z0-9]{1,16}(字符集宽度 = 36，重复层级 = 3)、嵌套脚本匹配((<script>.*<\ / script>)+)(嵌套深度 = 2)、多协议匹配(http|https|ftp|sftp|tcp):\ / \ / (多分支数量 = 5，含重复)等。

[0057] 对80条低复杂性规则执行NFA→DFA批量转换，并执行状态压缩和分级预缓存；对对20条高复杂性规则，跳过完整 DFA 转换，直接作为NFA 适配规则，提取并存储规则核心匹配片段；再加载期个性化阈值预计算，运行期实时监控，通过 pyre2 库钩子函数，每 1ms 采集 NFA 引擎的回溯次数与匹配耗时，同步写入内存缓存，确保监控粒度适配毫秒级匹配需求；并在达到安全阈值时触发熔断机制，例如某含 2000 字符的超长违规数据包，匹配((<script>.*<\ / script>)+)规则时，回溯次数达 100 次，触发熔断，此时，立即中断当前规则匹配，返回风险匹配，将数据包标记为"待人工复核”并写入本地日志(含回溯次数、匹配耗时、数据包特征)，避免无效资源占用；将该规则临时标记为低优先级；同时实时监控系统 CPU 占用率，当降至50%以下时，执行规则自动优化恢复。

[0058] 最终的匹配效果如表1所示：表1

[0059] 由表1可以看出，本发明实施例所述基于流量采集分析的正则匹配优化方法及系统，可实现规则加载效率提升 50%以上、单数据包匹配耗时降低 40%-60%、回溯风险发生率降至0.1%以下，完全适配大型企业内网流量监控、互联网骨干网数据采集等高流量生产场景，高效支撑流量特征识别、违规过滤、恶意拦截等核心安全防护功能。

[0060] 由此可见，本发明实施例所提供的基于流量采集分析的正则匹配优化方法及系统，在流量采集分析高并发场景的正则匹配中，使用 "字符集宽度+重复限定符层级+嵌套量词深度+多分支数量” 的四维度规则复杂性评估模型，实现规则精准分级与差异化处理：对低复杂性规则执行DFA状态压缩和分级预缓存，适配规则高频迭代需求；对高复杂性规则仅提取核心匹配片段，跳过完整 DFA 转换，从源头减少内存占用；同时为高复杂性规则设计 "个性化阈值预计算-运行期毫秒级监控 - 熔断后自动优化” 的全周期防护机制：加载期结合规则特征标签预计算差异化安全阈值；运行期通过钩子函数每1ms采集NFA引擎指标，触发阈值即中断匹配并标记规则为低优先级；系统负载降低后自动拆分嵌套、简化分支，优化后重新评估复用，彻底防控灾难性回溯。

[0061] 以上描述仅为本发明的较佳实施例以及对所运用技术原理的说明，并非旨在限制要求保护的本发明的范围，而是仅仅表示本发明的优选实施例。本领域技术人员应当理解，本发明中所涉及的发明范围，并不限于上述技术特征的特定组合而成的技术方案，同时也应涵盖在不脱离所述发明构思的情况下，由上述技术特征或其等同特征进行任意组合而形成的其它技术方案。基于本发明的实施例，本领域技术人员在没有做出创造性劳动的前提下所获得的所有其他实施例，都属于本发明保护的范围。< / script>

Claims

1. A regular expression matching optimization method based on traffic collection and analysis, characterized in that, The method includes the following steps: Step S1: Set the regularity rule complexity evaluation index and threshold, and divide the regularity rules into low complexity rules and high complexity rules; Step S2: Calculate the evaluation index of the current regular expression rule and determine the complexity classification of the current regular expression rule; if it is determined to be low complexity, it is used as a DFA adaptation rule and proceeds to step S3; if it is determined to be high complexity, it is used as an NFA adaptation rule and proceeds to step S4. Step S3: Perform NFA→DFA transformation on the DFA adaptation rules, then perform hierarchical pre-caching after state compression, and proceed to step S5; Step S4: For NFA adaptation rules, store core matching segments and configure the safety thresholds for backtracking times and matching time consumption respectively, then proceed to step S5; Step S5: Determine whether the current regular expression rule is the last one or the Nth evaluation; if it is the last one, load all regular expression rules and proceed to step S6; if it is the Nth evaluation, load the current regular expression rule and proceed to step S6; otherwise, increment the regular expression rule number by 1 and proceed to step S2; where N is a positive integer greater than 1. Step S6: When the rule is a DFA-adapted rule, match directly; when the rule is an NFA-adapted rule, collect the backtracking count and matching time of the NFA engine in real time. If either reaches the corresponding safety threshold, proceed to step S7; otherwise, perform NFA rule matching. Step S7: Interrupt the current rule matching, return to risk matching and mark the data packet for review. At the same time, mark the current regular expression rule as low priority and synchronize it to the rule attribute table. The low priority rule will be skipped when matching subsequent data packets. Step S8: Real-time assessment of the current system load and setting of the recovery threshold; when the load is less than the recovery threshold, automatically optimize low-priority rules and delete low-priority identifiers as the current regular expression rule, then proceed to step S2.

2. The method according to claim 1, characterized in that, In step S1, the evaluation metrics for the complexity of regular expression rules include: character set width, repeating quantifier level, nested quantifier depth, and number of branches; for each of the four metrics, the first to fourth thresholds are set respectively.

3. The method according to claim 1, characterized in that, Step S2 determines the complexity category of the current regular expression rule. The determination process includes: If the following conditions are met: character set width > first threshold, repetition quantifier level ≥ second threshold, nested quantifier depth ≥ third threshold, and / or number of branches > fourth threshold, then it is determined to be a high complexity rule; otherwise, it is determined to be a low complexity rule.

4. The method according to claim 1, characterized in that, Step S3 involves tiered pre-caching, where rules are pre-cached in tiers based on usage frequency. The top 30% of rules by usage frequency are stored in the memory resident cache, with a loading time of ≤10 milliseconds. Rules by usage frequency of 30% to 70% are stored in the SSD disk cache, with a loading time of ≤500 milliseconds.

5. The method according to claim 1, characterized in that, The core matching segment mentioned in step S4 includes the violation content feature code and the protocol key identifier.

6. The method according to claim 1, characterized in that, In step S4, when configuring security thresholds, a personalized security threshold is configured for each NFA adaptation rule, specifically including: For rules containing nested quantifiers, the maximum matching time is 5ms and the maximum number of backtracking attempts is 100. For rules containing wide character set repetitions, the maximum matching time is 3ms and the maximum number of backtracking attempts is 80. For multi-branch rules, the maximum matching time is 4ms and the maximum number of backtracking attempts is 90.

7. The method according to claim 1, characterized in that, In step S8, the current system load is determined based on real-time monitoring of the system CPU. The recovery threshold at this time is the CPU recovery threshold.

8. The method according to claim 1, characterized in that, When automatically optimizing low-priority rules in step S8, syntax optimization is used, including splitting nested quantifiers and simplifying the matching logic for violation features.

9. The method according to claim 8, characterized in that, When performing syntax optimization, differentiated optimization methods are used for rules with different feature labels, including: The "nested quantifier" rule is broken down into: <script>.*<\ / script>基础规则+"嵌套层数≤2”长度约束判断；对"多分支”规则，拆分为主协议与次要协议两条子规则，降低单条规则匹配复杂度。10.一种基于流量采集分析的正则匹配优化系统，其特征在于，所述系统包括：初始配置模块、规则分类模块、DFA预缓存模块、NFA存储模块、NFA安全阈值设置模块、规则加载模块、数据包匹配模块、熔断执行模块和规则优化模块；其中，所述初始配置模块用于设置正则规则复杂性评估指标及阈值，将正则规则分为低复杂性规则和高复杂性规则；所述规则分类模块用于计算当前正则规则的评估指标，判断当前正则规则复杂性分类；若判定为低复杂性规则，则作为DFA适配规则，并发送给DFA预缓存模块；若判定为高复杂性规则，则作为NFA适配规则，并发送给NFA存储模块；所述DFA预缓存模块用于对DFA适配规则执行NFA→DFA转换，再经状态压缩后进行分级预缓存；所述NFA存储模块用于对NFA适配规则，存储核心匹配片段；所述NFA安全阈值设置模块用于对NFA适配规则分别配置回溯次数和匹配耗时的安全阈值；所述规则加载模块用于判断当前正则规则是否为最后一条，或第N次评估；若为最后一条，则加载所有正则规则，启动数据包匹配模块；若为第N次评估，则加载当前正则规则，返回数据包匹配模块；否则，正则规则序号加1，启动规则分类模块；其中，N为大于1的正整数；所述数据包匹配模块用于当规则为DFA适配规则时，直接进行匹配；当规则为NFA适配规则时，实时采集NFA引擎的回溯次数与匹配耗时，若任一项达到对应的安全阈值，触发熔断执行模块；若未达到，则执行NFA规则匹配；所述熔断执行模块用于中断当前规则匹配，返回风险匹配并标记数据包待复核，同时将当前正则规则标记为低优先级并同步至规则属性表，且后续数据包匹配时跳过低优先级规则；所述规则优化模块用于实时判断当前系统负载情况并设置恢复阈值；当负载小于恢复阈值时，自动优化低优先级规则并删除低优先级标识，作为当前正则规则，启动规则分类模块。< / script>

Images