A traffic threat detection method and system supporting massive rules

Abstract

The present invention relates to the technical field of network traffic threat detection, and specifically to a traffic threat detection method and system supporting massive rules, the system comprising a traffic collection module for capturing bad network data, and performing basic traffic analysis on each data to separate the two parts of the transport layer protocol and load; a load scanning module for scanning the load of network data packets; a unit mapping module for extracting the scanned basic unit features and mapping them to number; a sorting module for sorting the numbers of the basic unit features; a character assembly module; a logic scanning module; and a matching position checking module. A network traffic threat detection system is constructed based on this method, which solves the problem that a single block of data needs to match rules repeatedly and cannot achieve integrated optimization at the algorithm level, resulting in repeated calculations and inefficient calculations, and also solves the problem that there is a lot of invalid calculation consumption, resulting in reduced overall performance.

Description

Technical Field

[0001] The present invention relates to the technical field of network traffic threat detection, and in particular to a traffic threat detection method and system supporting massive rules. Background Art

[0002] In the network threat detection system based on network traffic analysis, the matching technology for traffic data features is a widely used key technology. Most network threat detection is achieved based on the matching of relevant rules of traffic features, and a set of network traffic threat detection equipment has at least tens of thousands or even hundreds of thousands of built-in security detection rules. The core of the technology lies in the efficient multi-feature matching capability and the flexible feature rule writing capability. The former directly determines the overall analysis performance of the analysis system, and the latter directly determines the functional perfection of the analysis system.

[0003] With the continuous development of related fields, more and more relevant feature rules have been accumulated in the traffic threat detection system. Various traffic matching engines based on traditional technical solutions have gradually become difficult to perform. It is becoming increasingly difficult to balance complete functional requirements and efficient analysis performance. Traditional traffic matching engines are mainly based on the mode of first parsing the traffic and then gradually matching the rules based on the parsed protocol fields. This solution can have better functions and performance when there are fewer traffic matching rules, but when the number of matching rules reaches a certain scale, the following problems will be encountered:

[0004] A single block of data needs to match rules repeatedly and cannot achieve integrated optimization at the algorithm level, resulting in problems such as repeated and inefficient calculations. At the same time, there is a lot of invalid computing consumption, which reduces its overall performance. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a traffic threat detection method and system that supports massive rules, which solves the problem that a single block of data needs to match rules repeatedly and cannot achieve integrated optimization at the algorithm level, resulting in repeated and inefficient calculations. At the same time, it also solves the problem that there is a lot of invalid computing consumption, resulting in reduced overall performance.

[0006] The technical solution adopted by the present invention to solve the above technical problems is: a traffic threat detection system supporting massive rules,

[0007] include:

[0008] The traffic collection module is used to capture bad network data, analyze the basic traffic of each data, and separate the two parts of the transport layer protocol and load;

[0009] A load scanning module is used to scan the load of network data packets;

[0010] The unit mapping module is used to extract the scanned basic unit features and map them to numbers;

[0011] A sorting module, used to sort the numbers of basic unit features;

[0012] A character assembly module is used to regularize the sorted numbers and compose a string for logical matching based on the sorting order;

[0013] Logical scanning module, used to scan character strings;

[0014] The matching position checking module is used to parse and determine the position of the original basic unit of the hit logic string.

[0015] The present invention also provides a traffic threat detection method that supports massive rules. The method includes a rule compilation stage and a traffic detection stage. The rule compilation stage is used to fry, organize and pre-compile all detection rules for multi-mode matching, and provide support for efficient matching and detection in subsequent stages. The traffic detection stage is used to perform network traffic detection on the results of the rule compilation stage.

[0016] Preferably, the specific steps of the rule compilation stage are as follows:

[0017] S11, rule parsing: read the configuration files of relevant rules to obtain the rules that need to be tested;

[0018] S12, rule logic analysis: sort out the logical relationships in the rules, find the "or" logical relationship, and split it into two rules;

[0019] S13, rule unit extraction: extract the basic matching units in all rules and perform deduplication processing;

[0020] S14, rule unit numbering: sorting the extracted rule units in lexicographic order, and then assigning a digital number in sequence;

[0021] S15, rule logic construction: based on the number of each rule unit, reconstruct the logical relationship based on the number;

[0022] S16, load addition: add all basic units that need to be matched to the multi-mode matching engine, and build a multi-mode scanner for the basic matching units, which is recorded as Scanner1;

[0023] S17, load position record: for each basic matching unit, record the relevant configuration position requirements;

[0024] S18. Logic addition: Generate a multi-mode matching scanner with fast matching logic based on the numbered logical relationship.

[0025] Preferably, the logic adding comprises the following steps:

[0026] S181, numbering regularization: take the maximum number of digits of all basic matching unit numbers and set it to X, fill the front of all numbers with 0 for the first time, so that all numbers have a digital combination of X digits, then fill all numbers with 0 for the second time, so that all matching unit numbers have a digital combination of 2X digits, and finally add a fixed point prefix number "1" to the highest digit of all matching unit numbers;

[0027] S182, pattern construction: convert the final number generated in S181 into a character string, then merge the character strings based on a logical relationship, and fill in the wildcard "*" between adjacent number character strings;

[0028] S183, multi-mode multi-matching addition: add all logical strings formed in the previous step to the multi-mode matching engine, and build a multi-mode scanner for matching logic, recorded as Scanner2.

[0029] Preferably, the specific steps of the flow detection stage are as follows:

[0030] S21. Network traffic collection: The traffic collection module is used to capture bad network data, and the basic traffic of each data is analyzed to separate the two parts of the transport layer protocol and load;

[0031] S22, load scanning: Scan the load of the network data packet in sequence based on Scanner1 through load scanning;

[0032] S23, regular unit mapping: extracting the scanned basic unit features through the unit mapping module, and mapping and numbering them;

[0033] S24, number sorting: sorting the numbers obtained in the previous step by a sorting module;

[0034] S25, logical character assembly: the numbers obtained in the previous step are regularized by a character assembly module with reference to the number regularization step, and then a character string for logical matching is formed based on the sorting order;

[0035] S26, logical scanning: Scan the string obtained in the previous step based on Scanner2 through the logical scanning module, and if it hits, proceed to the next step;

[0036] S27, matching position check: the matching position check module parses and judges the position of the original unit of the hit logic string, and if it meets the rule requirements, the corresponding rule is hit.

[0037] Beneficial effects of the present invention:

[0038] The present invention provides an efficient multi-rule traffic matching engine design method, and builds a network traffic threat detection system based on the method, which transfers a large amount of work in the traffic detection and matching stage to the early stage for one-time processing, and only performs on-demand protocol analysis on the traffic that hits the rules, which greatly reduces the consumption of computing resources and significantly reduces the hardware cost of the threat detection system. In addition, the entire process is optimized and designed based on a multi-mode matching algorithm, and the interference of the number of security rules and the hit situation on the stability of the overall matching performance is also greatly reduced, which provides good technical support for the system to integrate more security detection rules, and provides effective support for the formation and continuous development of the system security capability that is comprehensively superior to the existing technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0040] Figure 1 It is a flowchart of the rule compilation stage in the present invention;

[0041] Figure 2 It is a flow chart of the flow detection stage in the present invention. DETAILED DESCRIPTION

[0042] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the present invention is further explained below in conjunction with specific implementation methods.

[0043] Embodiment 1:

[0044] like Figure 1-Figure 2 As shown, the present invention provides a traffic threat detection system supporting massive rules, including:

[0045] The traffic collection module is used to capture bad network data, analyze the basic traffic of each data, and separate the two parts of the transport layer protocol and load;

[0046] A load scanning module is used to scan the load of network data packets;

[0047] The unit mapping module is used to extract the scanned basic unit features and map them to numbers;

[0048] A sorting module, used to sort the numbers of basic unit features;

[0049] A character assembly module is used to regularize the sorted numbers and compose a string for logical matching based on the sorting order;

[0050] Logical scanning module, used to scan character strings;

[0051] The matching position checking module is used to parse and determine the position of the original basic unit of the hit logic string.

[0052] In the specific work, the bad network data is captured through the traffic collection module, and the basic traffic of each data is analyzed to separate the two parts of the transport layer protocol and load. The load scanning module is used to scan the load of the network data packet based on Scanner1 in turn. The basic unit features scanned are extracted through the unit mapping module, and their mappings are numbered. The numbers obtained in the previous step are sorted through the sorting module. The numbers obtained in the previous step are regularized by the character assembly module with reference to the number regularization step, and then a string for logical matching is composed based on the sorting order. The string obtained in the previous step is scanned by the logic scanning module based on Scanner2. If a hit is found, the next step is entered. The position of the original unit of the hit logic string is analyzed and judged by the matching position check module. If it meets the rule requirements, the corresponding rule is hit.

[0053] Embodiment 2:

[0054] The present invention also provides a traffic threat detection method that supports massive rules, including a rule compilation stage and a traffic detection stage, wherein the rule compilation stage is used to split, organize and pre-compile all detection rules for multi-mode matching, and provide support for efficient matching and detection in subsequent stages, and the traffic detection stage is used to perform network traffic detection on the results of the rule compilation stage.

[0055] The specific steps of the rule compilation phase are as follows:

[0056] S11, rule parsing: read the configuration file of relevant rules to obtain the rules that need to be tested; Figure 1 As shown, rule1: a&&b&&c||a&&d;

[0057] S12, rule logic analysis: sort out the logical relationships in the rules, find the "or" logical relationship, and split it into two rules; split rule1: a&&b&&c||a&&d in S11 into two parts: rule1: a&&b&&c and rule1: a&&d;

[0058] S13, rule unit extraction: extract the basic matching units in all rules and perform deduplication processing; extract the rule units in S12 into: a, b, c, d;

[0059] S14, rule unit numbering: sort the extracted rule units in lexicographic order, and then assign a number to each of them; the rule units in S13 are numbered as: a:1, b:2, c:3, and d:4;

[0060] S15, rule logic construction: based on the number of each rule unit, reconstruct the logical relationship based on the number; the rules in S14 are logically constructed as: rule1: 1&&2&&3 and rule1: 1&&2&&4;

[0061] S16, load addition: add all basic units that need to be matched to the multi-mode matching engine, and build a multi-mode scanner for the basic matching units, denoted as Scanner1; as shown in the figure: a, b, c, d.

[0062] S17. Load location record: for each basic matching unit, record the relevant configuration location requirements; a: unlimited; b: URL; c: TCP-Payload; d: unlimited.

[0063] S18. Logic addition: Generate a multi-mode matching scanner with fast matching logic based on the numbered logical relationship; 101*102*103, 101*104.

[0064] The logic addition includes the following steps:

[0065] S181, numbering regularization: take the maximum number of digits of all basic matching unit numbers and set it to X, fill the front of all numbers with 0 for the first time, so that all numbers have a digital combination of X digits, then fill all numbers with 0 for the second time, so that all matching unit numbers have a digital combination of 2X digits, and finally add a fixed point prefix number "1" to the highest digit of all matching unit numbers;

[0066] S182, pattern construction: convert the final number generated in S181 into a string, then merge the strings based on logical relationships, and fill in the wildcard "*" between adjacent number strings.

[0067] S183, multi-mode multi-matching addition: add all logical strings formed in the previous step to the multi-mode matching engine, and build a multi-mode scanner for matching logic, recorded as Scanner2.

[0068] The specific steps of the flow detection stage are as follows:

[0069] S21. Network traffic collection: The traffic collection module is used to capture bad network data, and the basic traffic of each data is analyzed to separate the two parts of the transport layer protocol and load;

[0070] S22, load scanning: Scan the load of the network data packet in sequence based on Scanner1 through the load scanning module;

[0071] S23, regular unit mapping: extracting the scanned basic unit features through the unit mapping module, and mapping and numbering them;

[0072] S24, number sorting: sorting the numbers obtained in the previous step by a sorting module;

[0073] S25, logical character assembly: the numbers obtained in the previous step are regularized by a character assembly module with reference to the number regularization step, and then a character string for logical matching is formed based on the sorting order;

[0074] S26, logical scanning: Scan the string obtained in the previous step based on Scanner2 through the logical scanning module, and if it hits, proceed to the next step;

[0075] S27, matching position check: the matching position check module is used to parse and judge the position of the original unit that hits the logical string;

[0076] S28, obtain the rule hit result: if the rule requirements are met, the corresponding rule is hit; Figure 2 As shown, name it "rule1".

[0077] It should be noted that the present invention transfers a large amount of work in the traffic detection and matching stages to the early stage for one-time processing, and only performs on-demand protocol analysis for the traffic that hits the rules, which greatly reduces the consumption of computing resources and significantly reduces the hardware cost of the threat detection system. In addition, the entire process is optimized and designed based on a multi-mode matching algorithm, and the number of security rules and the hit situation have greatly reduced the interference on the stability of the overall matching performance, which provides good technical support for the system to integrate more security detection rules, and provides effective support for the formation and continuous development of system security capabilities that is comprehensively superior to existing technical solutions.

[0078] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of the present invention is defined by the attached claims and their equivalents.

Claims

1. A traffic threat detection method supporting massive rules, characterized in that: The method includes a rule compilation phase and a traffic detection phase, wherein the rule compilation phase is used to split, organize and pre-compile all detection rules for multi-mode matching, and to support efficient matching and detection in subsequent phases; The traffic detection phase is used to perform network traffic detection on the results of the rule compilation phase; The rule compilation phase of this method includes the following steps: S11, rule parsing: read the configuration files of relevant rules to obtain the rules that need to be tested; S12, rule logic analysis: sort out the logical relationships in the rules, find the "or" logical relationship, and split it into two rules; S13, rule unit extraction: extract the basic matching units in all rules and perform deduplication processing; S14, rule unit numbering: sorting the extracted rule units in lexicographic order, and then assigning a digital number in sequence; S15, rule logic construction: based on the number of each rule unit, reconstruct the logical relationship based on the number; S16, load addition: add all basic units that need to be matched to the multi-mode matching engine, and build a multi-mode scanner for the basic matching units, which is recorded as Scanner1; S17, load position record: for each basic matching unit, record the relevant configuration position requirements; S18, logic addition: based on the logical relationship of the numbers, a multi-mode matching scanner is generated for fast matching logic; The logic addition includes the following steps: S181, numbering regularization: take the maximum number of digits of all basic matching unit numbers and set it to X, fill the front of all numbers with 0 for the first time, so that all numbers become a digital combination of X digits, then fill all numbers with 0 for the second time, so that all matching unit numbers become a digital combination of 2X digits, and finally add a fixed point prefix number "1" to the highest digit of all matching unit numbers; S182, pattern construction: convert the final number generated in S181 into a character string, then merge the character strings based on a logical relationship, and fill in the wildcard "*" between adjacent number character strings; S183, multi-mode multi-matching addition: adding all logical strings formed in the previous step to the multi-mode matching engine, and constructing a multi-mode scanner for matching logic, recorded as Scanner2; The specific steps of the flow detection stage are as follows: S21. Network traffic collection: The traffic collection module is used to capture bad network data, and the basic traffic of each data is analyzed to separate the two parts of the transport layer protocol and load; S22, load scanning: Scan the load of the network data packet in sequence based on Scanner1 through the load scanning module; S23, regular unit mapping: extracting the scanned basic unit features through the unit mapping module, and mapping and numbering them; S24, number sorting: sorting the numbers obtained in the previous step by a sorting module; S25, logical character assembly: the numbers obtained in the previous step are regularized by a character assembly module with reference to the number regularization step, and then a character string for logical matching is formed based on the sorting order; S26, logical scanning: Scan the string obtained in the previous step based on Scanner2 through the logical scanning module, and if it hits, proceed to the next step; S27, matching position check: the matching position check module parses and judges the position of the original unit of the hit logic string, and if it meets the rule requirements, the corresponding rule is hit.

2. A system using the traffic threat detection method supporting massive rules as claimed in claim 1, characterized in that: The system includes: The traffic collection module is used to capture bad network data, analyze the basic traffic of each data, and separate the two parts of the transport layer protocol and load; A load scanning module is used to scan the load of network data packets; The unit mapping module is used to extract the scanned basic unit features and map them to numbers; A sorting module, used to sort the numbers of basic unit features; A character assembly module is used to regularize the sorted numbers and compose a string for logical matching based on the sorting order; Logical scanning module, used to scan character strings; The matching position checking module is used to parse and determine the position of the original basic unit of the hit logic string.

Images