Cross-domain network threat data high-performance filtering method, device, medium and program
By introducing geometric space segmentation and hash table matching technology into the packet filter matching algorithm, combined with the weighting mechanism of the Bloom filter, the problems of low performance and large redundancy in traditional algorithms under massive rule sets are solved, and high efficiency is achieved. Low redundant packet filtering.
Patent Information
- Application Number
- CN202411459583.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-18
- Publication Date
- 2025-02-14
AI Technical Summary
When traditional packet filtering and matching algorithms process massive data packets, the rule set is too large, resulting in low search performance, and there are rules redundancy problems, which affects efficiency.
Using a packet filtering matching algorithm based on geometric space segmentation and hash table, quickly find matching rules through decision tree traversal and hash table matching, and introduces a Bloom filter for weight determination when dividing subsets of rules, reducing rule redundancy .
It improves the efficiency of packet matching rules, reduces rule redundancy, realizes fast matching and high-performance filtering, and is suitable for large-scale rule sets.
Smart Images
Figure CN119449384A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication technology, and in particular relates to a high-performance filtering method, device, medium and program for cross-domain network threat data. Background Art
[0002] In the application environment of massive data, the rules and volume required for packet filtering are very large. Traditional packet filtering matching algorithms can no longer carry such a large set of rules. Therefore, it is of great significance to develop and adopt efficient packet filtering matching algorithms.
[0003] Currently, the commonly used packet filtering matching algorithms can be divided into the following categories: exhaustive method; packet filtering matching algorithms based on geometric space segmentation, such as Hicuts algorithm, Hypercuts algorithm, etc.; filtering matching algorithms based on tuple space segmentation (tuple), TupleSpaceSearching algorithm, TuplePruning algorithm, etc.; packet filtering matching algorithms based on dimensional decomposition, such as RFC algorithm Pl, BV algorithm, etc.; packet filtering matching algorithms based on Trie tree, such as GirdofTrie algorithm, etc.
[0004] Exhaustive search method: Typical exhaustive search algorithms include sequential matching algorithm and TCAM-based massive parallel search. The idea of the sequential matching algorithm is to sequentially traverse the rule set from the beginning for each incoming data packet until a rule is matched. The sequential matching algorithm has high time complexity and poor search performance, and is not suitable for large-scale rule sets. The idea of the TCAM-based massive parallel search algorithm is to match all single rules in the rule set in parallel at the same time. The algorithm has high search performance, but the hardware consumption is too large and the cost is too high.
[0005] Trie-segmentation-based packet filtering algorithms include basic hierarchical Trie algorithm, set merge Trie algorithm, grid Trie algorithm, etc. The idea of this type of algorithm is to match by rule prefix, multiple prefix matches until the rule is finally hit. The algorithm is relatively simple, but multiple prefix matches take a long time and have high time complexity.
[0006] Packet filtering algorithms based on geometric space segmentation include the Hicuts algorithm and the multi-decision tree algorithm (HyperCuts algorithm). The idea of the Hicuts algorithm is to construct a decision tree through packet header information of different dimensions, and then divide the rule set into rule subsets according to the decision tree, with each leaf node corresponding to a rule subset after division. Due to the need to construct a decision tree and divide the rule set, the preprocessing time of this algorithm is relatively long. The HyperCuts algorithm is an improvement on the Hicuts algorithm, which reduces the preprocessing time consumption by constructing a decision tree through parallel dimension segmentation.
[0007] Packet filtering algorithms based on tuple space segmentation include the basic tuple space (TSS) algorithm and the pruned tuple space algorithm. The tuple space algorithm uses tuples to segment the rule set to shorten the multi-dimensional search range, thereby achieving the purpose of fast search. The tuple space packet filtering algorithm is similar to the packet filtering algorithm based on space segmentation. When the rule set is divided into several rule subsets, the search for each rule subset can be implemented in parallel, but the size of the divided rule subsets and each tuple space cannot be determined, so the packet filtering algorithm based on tuple space is not easy to implement in parallel. The tuple space algorithm has certain advantages in terms of memory space consumption, and compresses the rule set to reduce memory consumption.
[0008] Packet filtering algorithms based on dimensional decomposition include parallel bit vector algorithm, cross product algorithm, recursive flow classification (RFC) algorithm, etc. The idea of the parallel bit vector algorithm is to use bits to form a bit vector, where each dimension corresponds to a bit vector, and each bit of the bit vector corresponds to a rule. For each dimension, a Trie tree is constructed as a search structure, and each dimension is searched independently. Finally, the search results of each dimension are operated and the final result is obtained. The parallel bit vector algorithm is not suitable for the case of a large number of rules. The difference between the cross product algorithm and the parallel bit vector algorithm is that the bit vector is not used to represent the dimension. For each dimension, a unique set of fields is constructed, and each dimension is searched independently. The product table is used to merge the results. The cross product algorithm requires a large amount of memory space due to the use of the product table, and the space complexity is high. Summary of the invention
[0009] The purpose of the present invention is to provide a high-performance filtering method for cross-domain network threat data, which improves the efficiency of data packet matching rules and reduces rule redundancy. The technical solution adopted is as follows:
[0010] A high-performance filtering method for cross-domain network threat data includes the following steps:
[0011] Receiving data packets; wherein each data packet has n dimensions;
[0012] According to the dimension information of the data packet, the decision tree is traversed to find the rule subset to be matched; wherein the decision tree is determined according to at least one of the n dimensions, and the leaf nodes of the decision tree correspond one by one to the rule subset; the rule subset is obtained based on a geometric space segmentation algorithm, and the rule subset is a set of rules based on at least one of the n dimensions;
[0013] Obtaining a hash value of the data packet, and matching it with the hash table of the rule subset to obtain a matching result;
[0014] The data packet is filtered according to the matching result. Specifically, the data packet header information is hash matched with the preset rule set (the matching result is yes or no), so as to determine the rule to which the data packet belongs and execute the corresponding rule action.
[0015] Preferably, the geometric space segmentation algorithm comprises the following steps:
[0016] For all the rules in the rule set, the Bloom filter duplicate detection module selects the rules that have not appeared in all rule subsets and adds them to the minimum rule subset.
[0017] Preferably, the geometric space segmentation algorithm specifically includes the following steps:
[0018] Step 1: Initialize the Bloom filter duplicate detection module, including initializing the bit array to 0 and initializing each rule subset to empty;
[0019] Step 2: traverse the rules in the rule set, and for any rule, obtain the rule subset to which it belongs according to the rule division principle (the same as the division principle in Table 8);
[0020] Step 3: The rule in step 2 passes through the duplicate detection module. If it is determined that it has not appeared in all rule subsets, the bit array position corresponding to the rule is 1, and step 4 is executed;
[0021] Otherwise, skip this rule and go to step 2;
[0022] Step 4: According to the minimum rule subset priority scheduling algorithm, add the rules in step 3 to the minimum rule subset;
[0023] Step 5: Delete the rules in step 4 from the rule set in step 2, execute step 2, and finally obtain the rule subset constituting the decision tree.
[0024] Preferably, there are two cases for the division results in step 2;
[0025] Case 1: There is no rule redundancy, and the partition result is only one rule subset S1;
[0026] Case 2: Redundancy of rules occurs, and the partitioning results contain multiple different rule subsets;
[0027] In step 4, for case 1 in step 2, the rules in step 3 are added to the rule subset S1.
[0028] Preferably, the minimum rule subset priority scheduling algorithm specifically includes the following steps: comparing the sizes of the current rule subsets, and selecting the rule subset with the minimum size to add rules.
[0029] Preferably, the relationship between the number of rule subsets γ and the number of dimensions n is: γ=2 n .
[0030] Preferably, the hash table is a linked list array.
[0031] Preferably, the time complexity of the matching process is O(1).
[0032] Preferably, the specific process of establishing the hash table includes:
[0033] For a rule subset, the rules therein are traversed, and the values obtained after performing hash operations on the rules according to the dimension information are inserted into the hash table.
[0034] Preferably, the hash value of the data packet is obtained, and the specific process includes: performing a hash operation according to the dimension information of the data packet to obtain its corresponding hash value.
[0035] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the high-performance filtering method for cross-domain network threat data when executing the computer program.
[0036] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the high-performance filtering method for cross-domain network threat data is implemented.
[0037] A computer program, when executed by a processor, implements the high-performance filtering method for cross-domain network threat data.
[0038] Compared with the prior art, the advantages of the present invention are:
[0039] (1) Good search performance, short search time, and low time complexity: Based on the geometric space segmentation rules, a hash table is used as the rule storage and search structure to improve the efficiency of data packet matching rules, save the time required for sequential matching rules, and realize fast matching of data packets and filtering rules.
[0040] (2) The required memory space is small, which is suitable for large-scale rule sets: In view of the fact that the packet filtering algorithm that divides the rule subsets has the phenomenon of rule redundancy, a Bloom filter is used to add a duplicate detection process when dividing the rule subsets. For the rules that have been added to the rule subset, no more addition operations are performed, so as to achieve the purpose of reducing rule redundancy.
[0041] (3) Balanced division of rule subsets to improve rule query efficiency: To address the imbalance in rule subset sizes, when dividing rules, if a rule is allowed to be added to different rule subsets, try to add the rule to the smallest rule subset to achieve a balanced distribution of rules in the rule subsets. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is the structure diagram of the packet filtering algorithm HBSP based on geometric space partitioning and hash table;
[0043] Figure 2 Flowchart of a high-performance filtering method for cross-domain network threat data. DETAILED DESCRIPTION
[0044] The cross-domain network threat data high-performance filtering method of the present invention will be described in more detail below in conjunction with a schematic diagram, wherein a preferred embodiment of the present invention is shown, and it should be understood that those skilled in the art can modify the present invention described herein while still achieving the beneficial effects of the present invention. Therefore, the following description should be understood as being widely known to those skilled in the art, and not as a limitation of the present invention.
[0045] A high-performance filtering method for cross-domain network threat data, such as Figures 1-2 shown.
[0046] Firstly, the packet filtering algorithm Hicuts is analyzed, and a packet filtering matching algorithm based on geometric space partition and hash table is proposed, which is called (hash based on space partition) algorithm, or HBSP algorithm for short.
[0047] Secondly, a balanced low-redundancy scheme is adopted to reduce the redundancy of packet filtering algorithm rules. The performance of packet filtering algorithm is further improved by reducing the redundancy of rules and improving the balance of rule subsets.
[0048] Specifically, spatially segmented packet filtering algorithms generally have redundant rules. To address this problem, a rule duplication detection mechanism is introduced during the rule set segmentation process to reduce the redundancy of packet filtering algorithms.
[0049] A minimum rule subset priority scheduling algorithm is designed to improve the balance of rule subsets through reasonable rule scheduling.
[0050] Finally, on the basis of implementing the filtering module designed based on the packet filtering matching algorithm of geometric space segmentation and hash table, the various functions of the filtering module are improved, including real-time dynamic addition, deletion, modification and query functions, which support dynamic addition, deletion, modification and query of rules in the rule set; security level domain control function, which supports domain division of different security levels and inter-domain communication control; preloading backup function, which supports rule set backup, rule set preloading, etc.
[0051] 1) About HBSP algorithm:
[0052] a. Algorithm thinking
[0053] In order to improve the matching efficiency and data packet throughput of packet filtering matching algorithm, a packet filtering matching algorithm based on geometric space partitioning and hash table is proposed. This algorithm uses hash table matching rules to improve matching efficiency based on geometric space partitioning.
[0054] The solution to the hash collision problem in hash table search is as follows: divide the rule set into several smaller rule subsets, and then reduce the probability of hash collision when constructing the hash table from the rule subsets;
[0055] For the inevitable hash collision, the chain address method is adopted to set up a linked list in each unit of the hash table. When a hash collision occurs, other data items that are also mapped to this position only need to be added to the linked list, which ensures the feasibility of the algorithm.
[0056] Due to the use of hash tables, this algorithm occupies a large amount of memory space.
[0057] b. Algorithm steps
[0058] The HBSP algorithm is divided into the decision tree and hash table establishment process and the data packet matching process.
[0059] The process of building a decision tree and hash table includes the following two stages:
[0060] In the first stage, the rule set is spatially geometrically divided according to the matching dimension to construct a decision tree;
[0061] In the second stage, a hash table is established based on the matching dimension information in the different rule subsets after division.
[0062] The packet matching process includes the following two stages:
[0063] In the first stage, according to the matching dimension information of the data packet (source address, destination address, source port, destination port, protocol), the decision tree is traversed to determine the hash table to be matched;
[0064] In the second stage, a hash operation is performed based on the dimension information of the data packet to obtain its corresponding hash value, thereby determining the final matching rule.
[0065] Set the following parameters:
[0066] Data packet set (P), decision tree (T), decision tree root node (T0), decision tree middle new node (R (n) ), decision tree leaf nodes (T x ), number of dimensions (M), hash table (RT i ).
[0067] c. Construction of decision tree and hash table
[0068] Division of rule sets and construction of decision trees:
[0069] A rule set consists of rules. Assume that the rule set R = {r1, r2, ..., r n}, there are n rules in total, and the expressions of the rules are as follows:
[0070] r i ={sip i , dip i , sport i , dport i ,proto i ,deal i}, i = 1, 2, 3, ..., n
[0071] As shown above, the rule is determined by the source address sip i 、Destination address dip i , source port sport i 、Destination port dport i 、protocol i There are five matching dimensions and the processing actions for the data packet. i composition.
[0072] Among them, i represents an identifier, belongs to 1-n, and is used for rule counting.
[0073] The rule set is divided into two subsets according to the protocol dimension, the source port dimension is divided into two subsets, the destination port dimension is divided into two subsets, the source address dimension is divided into two subsets, and the destination address dimension is divided into two subsets. The 32 subspaces formed are divided into 32 rule subsets.
[0074] The relationship between the number of rule subsets and the number of dimensions is as follows:
[0075] γ=2 n ,n=dim1,dim2,...,dim n
[0076] Among them, dim n Represents the nth dimension.
[0077] n represents the number of the dimension.
[0078] The process of building a decision tree is as follows:
[0079] Generate a decision tree starting from the root node.
[0080] First, a decision tree is generated for the protocol dimension. Since the protocol dimension is divided into two subsets, the first layer of the decision tree has two branches;
[0081] Next, a decision tree is generated based on the source port dimension. According to the number of divided subsets, each has 2 branches, and the second layer has 4 branches in total.
[0082] Then, a decision tree is generated according to the target port dimension, each with 2 branches, and the third layer of the decision tree has a total of 8 branches;
[0083] Then, a decision tree is generated based on the source address dimension, each with 2 branches. The fourth layer of the decision tree has a total of 16 branches.
[0084] Finally, a decision tree is generated according to the target address dimension, each with 2 branches. The fifth layer of the decision tree has a total of 32 branches, that is, 32 rule subsets.
[0085] Hash table construction:
[0086] According to the divided rule subset SR i , i = 1, 2, 3, ..., 32, the process of constructing a hash table is as follows: one rule subset corresponds to one hash table. i represents the number of the rule subset.
[0087] For the rule subset, traverse the rules in it and insert the value obtained after the rule is hashed according to the dimension information into the hash table HT i Among them, HT i represents the i-th bucket.
[0088] In the structure of the hash table storing the rule subset, KV represents the array in the bucket, and i represents the subscript of the array, that is, the number of the bucket.
[0089] The essence of this hash table is a linked list array. The value obtained by hashing the rule dimension information is used as the array subscript, and a sequential linked list is used as the array element.
[0090] When inserting a rule, first look up the array index based on the hash value obtained by hashing the rule dimension information, and then check whether there is a rule at that position after confirming the position.
[0091] If it does not exist, insert the rule into the linked list; if it does exist, insert the rule into the end of the linked list.
[0092] Use hashing to achieve matching with approximately O(1) time complexity;
[0093] Use the zipper method to solve the hash conflict problem;
[0094] Hash-match the packet header information with the preset rule set to determine the rule to which the data packet belongs and execute the corresponding action.
[0095] In the final rule set storage structure, KEY represents the key and VALUE represents the hash value.
[0096] d. Packet search and matching process
[0097] From the construction process of decision tree and hash table, we can know that the data packet search and matching process can be divided into decision tree search stage and hash table search stage.
[0098] In the decision tree search phase, the decision tree is traversed according to the matching dimension information of the data packet until the last leaf node, that is, the rule subset.
[0099] The hash table to be searched in the next stage is determined based on the leaf node found.
[0100] In the hash table search phase, a hash operation is performed based on the dimension information of the data packet to obtain its corresponding hash value.
[0101] Then match the linked list in the hash table according to the hash value.
[0102] The time complexity of this matching process is O(1).
[0103] If a hash collision occurs, the linked list is matched sequentially, and the time complexity of the matching process is O(n).
[0104] 2) Balanced rule merging solution
[0105] In packet filtering algorithms that use segmented rule sets, rule redundancy is a common problem, which results in large space consumption for this type of packet filtering algorithm and affects the universality of packet filtering algorithm applications. There are two reasons for rule redundancy:
[0106] One is that the filtering rule is a segment rule, which is a range on one or more domains, and when the rule set is segmented on these domains, the same rule is divided into multiple different rule subsets, resulting in rule redundancy.
[0107] Another reason is that when splitting the rule set, there is overlap in the scope of dividing the rule subsets on one or more domains, which may also cause the same rule to be divided into multiple different rule subsets, resulting in rule redundancy.
[0108] The reasons for the redundancy of these two rules are described in detail below.
[0109] Generally speaking, for IPv4 filtering rules, the source IP address and destination IP address are 32 bits, the source port and destination port are 16 bits, and the protocol number is 8 bits. Taking the three dimensions of source port, destination port, and protocol number as an example, for the convenience of description, it is assumed that the three dimensions of source port, destination port, and protocol number are all 4 bits, and the rule set can be described in the way shown in Table 8.
[0110] Table 8 Rule Set-1
[0111]
[0112]
[0113] The three dimensions of source port, destination port, and protocol number are represented by D, F, and G respectively, where the size of each dimension is 4 bits, and the value range of each dimension is 0-15.
[0114] Assume that the principle of dividing regular subsets is: evenly divide into two equal sub-segments in each dimension, and the three-dimensional space formed by three dimensions can be divided into eight subspaces of equal size.
[0115] In dimension D, two sub-segments are formed: In dimension F, two sub-segments are formed: In dimension G, two sub-segments are formed:
[0116] The eight subspaces formed are described as:
[0117] Z1:d1f1g1, Z2:d1f1g2, Z3:d1f2g1, Z4:d1f2g2, Z5:d2f1g1, Z6:d2f1g2, Z7:d2f2g1, Z8:d2f2g2.
[0118] According to the principle of dividing rule subsets, the eight rules in the table are divided into various subspaces. The results of the division are as follows:
[0119] Z1:d1f1g1 contains rules r3, r6, r7;
[0120] Z2:d1f1g2 contains rule r7;
[0121] Z3:d1f2g1 contains rule r4;
[0122] Z4:d1f2g2 contains rule r4;
[0123] Z5:d2f1g1 contains rules r3 and r6;
[0124] Z6:d2f1g2 contains rules r1, r5, r8;
[0125] Z7:d2f2g1 contains rule r2;
[0126] Z8:d2f2g2 contains rules r2 and r8.
[0127] The eight subspaces divide the rule set into eight rule subsets. Corresponding to eight rule subsets
[0128] The complete rule set before segmentation contains 8 rules. It can be seen that the number of rules contained in all rule subsets after segmentation is 14, which is greater than 8. There are 6 repeated rules, and the rule redundancy rate is 75%.
[0129] If there is no rule redundancy problem, the number of rules contained in the rule subset after segmentation should be equal to the number of rules contained in the complete rule set before segmentation.
[0130] Another situation that leads to rule redundancy is that the value of each dimension is a fixed value, but the range of dividing the rule subsets overlaps, which also leads to rule redundancy.
[0131] Also taking the three dimensions of source port, destination port, and protocol number as an example, the sizes of the three dimensions are 4 bits respectively, and the rule set is shown in Table 9.
[0132] Table 9 Rule Set-2
[0133] rule Source port (4 bits) Destination port (4 bits) Protocol number (4 bits) <![CDATA[r1]]> 0001 0110 1111 <![CDATA[r2]]> 1011 1110 0111 <![CDATA[r3]]> 0010 1101 0001 <![CDATA[r4]]> 0110 1010 1001 <![CDATA[r5]]> 1000 0011 1100 <![CDATA[r6]]> 0000 0100 0101 <![CDATA[r7]]> 0010 0111 1000 <![CDATA[r8]]> 1010 1111 1101
[0134] D, F, and G are used to represent the three dimensions of source port, destination port, and protocol number respectively.
[0135] The principle of dividing the rule subsets is: on dimension D, two sub-segments are formed: In dimension F, two sub-segments are formed: In dimension G, two sub-segments are formed:
[0136] In the space composed of three dimensions D, F, and G, the space is divided into eight subspaces according to the division principle.
[0137] The eight subspaces formed can be described as: Z1: d1f1g1, Z2: d1f1g2, Z3: d1f2g1, Z4: d1f2g2, Z5: d2f1g1, Z6: d2f1g2, Z7: d2f2g1, Z8: d2f2g2.
[0138] The complete rule set contains 8 rules. According to the principle of division, the rule division results are as follows:
[0139] Z1: d1f1g1 contains rules r4, r6, r7;
[0140] Z2: d1f1g2 contains rules r1, r4, r5, r6, r7;
[0141] Z3: d1f2g1 contains rules r2, r3, r4, r6, r7;
[0142] Z4: d1f2g2 contains rules r1, r2, r4, r6, r7, r8;
[0143] Z5: d2f1g1 contains rule r4;
[0144] Z6: d2f1g2 contains rules r4 and r5;
[0145] Z7: d2f2g1 contains rules r2 and r4;
[0146] Z8: d2f2g2 contains rules r2, r4, r8.
[0147] The eight subspaces divide the complete rule set into eight rule subsets. Corresponding to eight rule subsets respectively
[0148] R1 contains 3 rules, R2 contains 5 rules, R3 contains 5 rules, R4 contains 6 rules, R5 contains 1 rule, R6 contains 2 rules, R7 contains 2 rules, and R8 contains 3 rules, with a total of 27 rules. 19 rules are repeated, and the rule redundancy rate is 237.5%.
[0149] It can be seen that this division principle leads to serious rule redundancy. The reason is that the two fields (0, 11) and (4, 15) overlap more, making the rule more likely to fall in the overlapping area and then be divided into multiple rule subsets, resulting in rule redundancy.
[0150] From the rule redundancy problem described above, we can see that if rule r i When redundancy occurs, it will be divided into rule subsets S1 and S2 according to the division principle, and the rule subclusters of multiple different rule subsets are defined, and ρ(r i |S1, S2). You can choose to add it to any of the rule subsets.
[0151] Assume that all rule redundancy occurs and the rule subcluster contains the rule subset S a , add these rules to the rule subset S a This will lead to an uneven distribution of rule subsets. a The scale is particularly large, containing a large number of rules, while other rule subsets are particularly small, containing only a small number of rules.
[0152] Unbalanced distribution of rule subsets will lead to a decrease in packet filtering rate because when the rule subset is large, it takes more time to query the rule subset. Moreover, the larger the rule subset is, the more rules it contains. When a data packet arrives, there is a greater probability that it will match a larger rule subset when filtering the data packet. Then, the rule subset needs to be queried, which consumes more query matching time, thereby reducing packet filtering efficiency.
[0153] a. Solution idea
[0154] The balanced low rule redundancy solution consists of the following two parts:
[0155] The first part is low rule redundancy.
[0156] Bloom filter is a high-performance, low-storage-space-consuming query algorithm with excellent query efficiency and short response time. In view of the redundancy of rules in the packet filtering algorithm for dividing rule subsets, Bloom filter is used to add a duplicate detection process when dividing rule subsets, and the rules that have been added to the rule subset are no longer added. This achieves the purpose of reducing rule redundancy.
[0157] The second part is a subset of split-balance rules.
[0158] To solve the problem of uneven rule subset sizes, when dividing rules, if a rule is allowed to be added to different rule subsets, try to add the rule to the smallest rule subset to achieve balanced distribution of rules in the rule subsets.
[0159] b. Program steps
[0160] Define the complete rule set as: R f :{r1,…,r rs}, the total number of rules is rs, and the balanced low rule redundancy scheme can be divided into the following steps:
[0161] Step 1: Initialize the Bloom filter duplicate detection module, including initializing the bit array to "0" and initializing each rule subset to empty.
[0162] Step 2: Traverse the rule set R f , for any rule r i , according to the rule division principle, the rule subset to which it belongs is obtained.
[0163] There are two cases for the partition results. Case 1: There is no rule redundancy, and the partition result is only a rule subset S1.
[0164] Case 2: Redundancy of rules occurs, and the partitioning results contain multiple different rule subsets.
[0165] Step 3: Rule r i Through the weight detection module, if it is judged that r i If it has not appeared before, set the corresponding bit array position to "1" and execute step 4.
[0166] If we judge r i If it appears in the rule subset, then skip rule r i , return to step 2 and continue traversing the rule set Rf.
[0167] Step 4: For case 1 in step 2, change rule r i Add to rule subset S y middle.
[0168] If it is case 2, it is necessary to perform rule scheduling according to the rule scheduling algorithm, and r i Added to the optimal rule subset.
[0169] The rule r i After adding to the rule subset, from the complete rule set R f Delete r i , return to step 2 and continue traversing the rule set.
[0170] Rule Set R f After the traversal is completed, if there is a misjudgment problem in the duplicate judgment module, the rule set R f There will still be a small number of rules left in the set, which need to be divided again. Re-traverse the remaining rule set Rf and then perform the division process in step 2 and the process of adding rules to the rule subset in step 4.
[0171] When the complete rule set R f After all the divisions are completed, for the subsequent addition of rules r k , through the duplicate detection module to detect duplicates, if the judgment rule r k If it is not recorded in the duplicate detection module, the corresponding bit array position is set to "1", and then r k Add to the rule subset; if the judgment rule r k If repeated, no adding operation will be performed.
[0172] From the above, in order to improve the data packet filtering performance, the present invention cuts in from two aspects: data packet matching rules and reducing rule redundancy, and proposes a packet filtering algorithm based on geometric space partitioning and hash table and a balanced low rule redundancy solution.
[0173] The former introduces hash tables as rule storage and search structures based on geometric space segmentation to improve the efficiency of packet matching rules. Geometric space segmentation divides the geometric space composed of various dimensions into different subspaces by dividing the subintervals in each dimension in a certain way. The rule set is mapped to the geometric space. Different rules fall in different spatial positions. The rules falling in the same subspace constitute a rule subset. According to the distribution of the subspaces formed after segmentation, the rule set can be divided into several rule subsets. Then, the hash table is used as the storage and search structure of the rule subset. Through the fast matching of the hash table, the purpose of high-performance packet filtering is achieved.
[0174] In a balanced low rule redundancy scheme, two problems that need to be solved are firstly considered: one is the rule redundancy problem, and the other is the problem of uneven distribution of rules in the rule subset.
[0175] Then a solution to the problem is proposed. In the process of building a decision tree, a rule duplication detection mechanism based on Bloom filter design is introduced to reduce the redundancy of rules, and a minimum rule subset priority scheduling algorithm is designed to balance the distribution of rules.
[0176] The rule duplication mechanism uses experiments to analyze the impact of the Bloom filter parameter bit array length m and the number of hash functions k on the probability of misjudgment. When the number of hash functions k is 5 and the bit array storage space reaches 4KB, rule misjudgment can be avoided and the rule redundancy problem can be solved. As for the minimum rule subset priority scheduling algorithm, its idea is that when rules can be added to multiple rule subsets, the size of each current rule subset is compared, and the minimum rule subset is selected to add the rule. Experiments have shown that the minimum rule subset priority scheduling algorithm can avoid the situation where the rule subset is too large or too small, so that the rules are evenly distributed in each rule subset.
[0177] The above is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any technician in the relevant technical field, without departing from the scope of the technical solution of the present invention, makes any form of equivalent replacement or modification to the technical solution and technical content disclosed in the present invention, which does not depart from the content of the technical solution of the present invention and still falls within the protection scope of the present invention.
Claims
1. A high-performance filtering method for cross-domain network threat data, characterized in that: The following steps are involved: Receiving data packets; wherein each data packet has n dimensions; According to the dimension information of the data packet, the decision tree is traversed to find the rule subset to be matched; wherein the decision tree is determined according to at least one of the n dimensions, and the leaf nodes of the decision tree correspond one by one to the rule subset; the rule subset is obtained based on a geometric space segmentation algorithm, and the rule subset is a set of rules based on at least one of the n dimensions; Obtaining a hash value of the data packet, and matching it with the hash table of the rule subset to obtain a matching result; The data packet is filtered according to the matching result.
2. The high-performance filtering method for cross-domain network threat data according to claim 1 is characterized in that: The geometric space segmentation algorithm comprises the following steps: For all the rules in the rule set, the Bloom filter duplicate detection module selects the rules that have not appeared in all rule subsets and adds them to the minimum rule subset.
3. The high-performance filtering method for cross-domain network threat data according to claim 2 is characterized in that: The geometric space segmentation algorithm specifically includes the following steps: Step 1: Initialize the Bloom filter duplicate detection module, including initializing the bit array to 0 and initializing each rule subset to empty; Step 2: traverse the rules in the rule set, and for any rule, obtain the rule subset to which it belongs according to the rule division principle; Step 3: The rule in step 2 passes through the duplicate detection module. If it is determined that it has not appeared in all rule subsets, the bit array position corresponding to the rule is 1, and step 4 is executed; Otherwise, skip this rule and go to step 2; Step 4: According to the minimum rule subset priority scheduling algorithm, add the rules in step 3 to the minimum rule subset; Step 5: Delete the rules in step 4 from the rule set in step 2, execute step 2, and finally obtain the rule subset constituting the decision tree.
4. The high-performance filtering method for cross-domain network threat data according to claim 3 is characterized in that: There are two cases for the division results in step 2: Case 1: There is no rule redundancy, and the partition result is only one rule subset S1; Case 2: Redundancy of rules occurs, and the partitioning results contain multiple different rule subsets; In step 4, for case 1 in step 2, the rules in step 3 are added to the rule subset S1.
5. The high-performance filtering method for cross-domain network threat data according to claim 3 is characterized in that: The minimum rule subset priority scheduling algorithm specifically includes the following steps: comparing the sizes of the current rule subsets, and selecting the rule subset with the minimum size to add rules.
6. The cross-domain network threat data high-performance packet filtering method according to claim 1 is characterized in that: The relationship between the number of rule subsets γ and the number of dimensions n is: γ = 2 n .
7. The high-performance filtering method for cross-domain network threat data according to claim 1 is characterized in that: The hash table is a linked list array.
8. The high-performance filtering method for cross-domain network threat data according to claim 1 is characterized in that: The time complexity of the matching process is O(1).
9. The high-performance filtering method for cross-domain network threat data according to claim 1, characterized in that: The specific process of establishing the hash table includes: For a rule subset, the rules therein are traversed, and the values obtained after performing hash operations on the rules according to the dimension information are inserted into the hash table.
10. The high-performance filtering method for cross-domain network threat data according to claim 1, characterized in that: Obtaining the hash value of the data packet, the specific process includes: performing a hash operation according to the dimension information of the data packet to obtain its corresponding hash value.
11. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the high-performance filtering method for cross-domain network threat data as described in any one of claims 1 to 10 is implemented.
12. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the high-performance filtering method for cross-domain network threat data as described in any one of claims 1 to 10 is implemented.
13. A computer program, characterized in that When the computer program is executed by a processor, the high-performance filtering method for cross-domain network threat data as described in any one of claims 1 to 10 is implemented.