High performance network packet classification method and system

Abstract

The application provides a high-performance network message classification method and system, comprising the following steps: S1, decomposing the five-tuple <source IP, destination IP, source port number, destination port number, protocol number> of a message into attributes with preset value ranges satisfying preset requirements; S2, configuring a set of bit sets for each attribute with a value range satisfying preset requirements, and using the bit sets to record the matching state of the current message attribute value and the corresponding attribute; S3, extracting the value of the decomposed attribute in the message, and searching for the bit set corresponding to the attribute value; S4, performing logical AND or logical OR operation on the searched bit sets to obtain a candidate matching rule; S5, performing further matching degree precision check on the candidate matching rule according to the port number to obtain a final matching rule; and S6, classifying the message by using the obtained final matching rule.

Description

Technical Field

[0001] The present invention relates to the technical field of message classification, and in particular, to a high-performance network message classification method and system. More specifically, the present invention proposes a new high-performance network message classification algorithm based on a bit set index data structure, logical operations, and an aggregation compression algorithm. Background Art

[0002] Messages must be forwarded according to corresponding rules in network devices or software services. In order to minimize the forwarding delay of messages in the network, a message classification algorithm that supports high-performance search needs to be designed.

[0003] Patent document CN107016080A (application number: 201710197897.5) discloses an efficient network message classification method. Its decision tree-based classification method has a simple structure, is easy to form rules, is highly efficient, and does not require additional knowledge. It is suitable for situations with large data volumes and diverse data formats. In particular, for systems such as real-time message classification that require high speed, the decision tree method is a reasonable choice. The decision tree algorithm based on information entropy proposed in the present invention is also based on a heuristic strategy and is an improvement on the multi-branch hierarchical tree. In the hierarchical tree, the present invention sequentially determines the rule subset to which it belongs based on the header bit information, which easily forms a phenomenon of uniform division of the rule subsets. Based on observations of a specific rule base, it is found that if the most distinguishing bits are extracted from it, and based on the entropy characteristics of these bits, bits with larger entropy gains are obtained, the rules can be evenly divided into subsets, thereby reducing the complexity of the worst case.

[0004] The network message classification algorithm proposed in the present invention is a set of rules with priorities defined based on the five-tuple of <source IP, destination IP, source port number, destination port number, protocol number>. By classifying the messages, the highest-priority rule that matches them is found. Since many applications are very sensitive to network transmission delays, this places very high performance requirements on the message search algorithm involved in routing and forwarding. First, the classification delay requirement for a single message is sub-microsecond or even nanosecond; second, the algorithm should show stable classification performance as much as possible on various rule sets with different characteristics; finally, the algorithm should have good maintainability and practicality, that is, it should support the rapid insertion and deletion of rules, the memory space occupied by the data structure should be controllable, and whether the algorithm needs to be trained and the training time for different rule sets are also aspects that need to be considered. Summary of the Invention

[0005] In view of the defects in the prior art, the object of the present invention is to provide a high-performance network message classification method and system.

[0006] A high-performance network message classification method provided by the present invention includes:

[0007] Step S1: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into attributes with preset value ranges that meet preset requirements;

[0008] Step S2: configuring a set of bit sets for each attribute whose value range meets the preset requirements, and using the bit sets to record the matching status between the current message attribute value and the corresponding attribute;

[0009] Step S3: extract the value of the decomposed attribute in the message and retrieve the bit set corresponding to the attribute value;

[0010] Step S4: performing a logical AND or logical OR operation on the retrieved bit set to obtain a candidate matching rule;

[0011] Step S5: Perform an accurate match check on the candidate matching rules on the port number to obtain the final matching rules;

[0012] Step S6: Classify the message using the final matching rule obtained.

[0013] Preferably, the preset bit sets are compressed according to a certain aggregation ratio, and the compressed bit sets are subjected to a logical OR / AND operation to obtain a candidate matching rule set.

[0014] Preferably, the step S1 adopts:

[0015] Step S1.1: Decompose the source IP and destination IP attributes into four attributes with a value range of 256, each attribute corresponding to a byte in the IP;

[0016] Step S1.2: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into four attributes obtained by decomposing the source IP attributes, four attributes obtained by decomposing the destination IP attributes, the source port number attribute, the destination port number attribute, and the protocol number attribute.

[0017] Preferably, the attribute value ranges of the two attributes, the source port number and the destination port number, are decomposed into c subspaces according to preset requirements.

[0018] Preferably, the step S2 adopts:

[0019] Step S2.1: configuring a bit set for each value of the decomposed source IP and destination IP attributes;

[0020] Step S2.2: configure a bit set for each protocol number, and configure a bit set for marking a wildcard rule for the protocol number;

[0021] Step S2.3: Configure a bit set for each decomposed port number subspace.

[0022] Preferably, step S4 adopts: when a reverse query algorithm based on a bit set is adopted, a logical OR operation is performed on the bit set in the non-matching state of the record rule to obtain a candidate matching rule; when a forward query algorithm based on a bit set is adopted, a logical AND operation is performed on the bit set in the matching state of the record rule to obtain a candidate matching rule.

[0023] A high-performance network message classification system provided by the present invention includes:

[0024] Module M1: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into attributes with preset value ranges that meet preset requirements;

[0025] Module M2: configures a set of bits for each attribute whose value range meets the preset requirements, and uses the bit set to record the matching status between the current message attribute value and the corresponding attribute;

[0026] Module M3: extracts the value of the decomposed attribute in the message and retrieves the bit set of the corresponding attribute value;

[0027] Module M4: performs logical AND or logical OR operations on the retrieved bit sets to obtain candidate matching rules;

[0028] Module M5: performs a precise matching check on the candidate matching rules on the port number to obtain the final matching rules;

[0029] Module M6: Classify the message using the final matching rule obtained.

[0030] Preferably, the preset bit sets are compressed according to a certain aggregation ratio, and the compressed bit sets are subjected to a logical OR / AND operation to obtain a candidate matching rule set.

[0031] Preferably, the module M1 adopts:

[0032] Module M1.1: Decompose the source IP and destination IP attributes into four attributes with a value range of 256, each corresponding to a byte in the IP.

[0033] Module M1.2: Decompose the packet's five-tuple (source IP, destination IP, source port number, destination port number, protocol number) into four attributes derived from the source IP attribute, four attributes derived from the destination IP attribute, the source port number attribute, the destination port number attribute, and the protocol number attribute.

[0034] The attribute value domains of the source port number and the destination port number are decomposed into c subspaces according to preset requirements.

[0035] Preferably, the module M2 adopts:

[0036] Module M2.1: Configure a bit set for each value of the decomposed source IP and destination IP attributes;

[0037] Module M2.2: Configure a bit set for each protocol number and configure a bit set to mark the protocol number wildcard rule;

[0038] Module M2.3: configure a bit set for each port number subspace after decomposition;

[0039] The module M4 adopts: when adopting a reverse query algorithm based on a bit set, a logical OR operation is performed on the bit set in the record rule mismatch state to obtain a candidate matching rule; when adopting a forward query algorithm based on a bit set, a logical AND operation is performed on the bit set in the record rule match state to obtain a candidate matching rule.

[0040] Compared with the prior art, the present invention has the following beneficial effects:

[0041] 1. The present invention uses a bit set to store partial mismatch results or matching results of a rule: the source IP and destination IP are decomposed into four attributes by bytes, thereby reducing the value range of the attribute to 0-255, making it possible to configure a bit set for each attribute value to separately store its mismatch rule or matching rule;

[0042] 2. The present invention mainly performs bit set retrieval and logical operations during the search process based on the bit set method, which has high execution efficiency and fully utilizes the filtering effect of each attribute, making the algorithm's classification efficiency high and able to achieve stable performance on various tilted rule sets;

[0043] 3. The present invention can balance the memory overhead by adjusting the number of divided subspaces in terms of port number attributes, making full use of the memory space to reduce the algorithm's search time;

[0044] 4. Experimental results show that on a rule set with weak locality, the average search time of the proposed method reaches 171ns, which is 99.5%, 83.5%, 99.5%, and 48.3% faster than the existing algorithms PSTSS, CutSplit, CutTSS, and PartitionSort, respectively;

[0045] 5. The present invention proposes an aggregated bit set optimization, compressing the original bit set according to a certain aggregation ratio to obtain a secondary bit set, thereby improving the search performance by reducing the number of logical operations and comparison operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0047] Figure 1 Schematic diagram of six sample rules of the present invention.

[0048] Figure 2 Schematic diagram of the predicate interval after decoding the sample rule in the present invention.

[0049] Figure 3 Schematic diagram of the reverse bit set index constructed based on sample rules in the present invention.

[0050] Figure 4 The flowchart of the high-performance network message classification method is shown in FIG.

[0051] Figure 5 The flowchart of the high-performance network message classification method is shown in FIG. DETAILED DESCRIPTION

[0052] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0053] Example 1

[0054] According to a high-performance network message classification method provided by the present invention, Figures 4 to 5 Shown, including:

[0055] Step S1: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into attributes with preset value ranges that meet preset requirements;

[0056] Step S2: configuring a set of bit sets for each attribute whose value range meets the preset requirements, and using the bit sets to record the matching status between the current message attribute value and the corresponding attribute;

[0057] Step S3: extract the value of the decomposed attribute in the message and retrieve the bit set corresponding to the attribute value;

[0058] Step S4: performing a logical AND or logical OR operation on the retrieved bit set to obtain a candidate matching rule;

[0059] Step S5: Perform an accurate match check on the candidate matching rules on the port number to obtain the final matching rules;

[0060] Step S6: Classify the message using the final matching rule obtained.

[0061] Specifically, a preset bit set is compressed according to a certain aggregation ratio, and the compressed bit set is subjected to a logical OR / AND operation to obtain a candidate matching rule set.

[0062] Specifically, the step S1 adopts:

[0063] Step S1.1: Decompose the source IP and destination IP attributes into four attributes with a value range of 256, each attribute corresponding to a byte in the IP;

[0064] Step S1.2: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into four attributes obtained by decomposing the source IP attributes, four attributes obtained by decomposing the destination IP attributes, the source port number attribute, the destination port number attribute, and the protocol number attribute.

[0065] Specifically, the attribute value ranges of the two attributes, the source port number and the destination port number, are decomposed into c subspaces according to preset requirements.

[0066] Specifically, the step S2 adopts:

[0067] Step S2.1: configuring a bit set for each value of the decomposed source IP and destination IP attributes;

[0068] Step S2.2: configure a bit set for each protocol number, and configure a bit set for marking a wildcard rule for the protocol number;

[0069] Step S2.3: Configure a bit set for each decomposed port number subspace.

[0070] Specifically, step S4 adopts: when a reverse query algorithm based on a bit set is adopted, a logical OR operation is performed on the bit set in the record rule mismatch state to obtain a candidate matching rule; when a forward query algorithm based on a bit set is adopted, a logical AND operation is performed on the bit set in the record rule matching state to obtain a candidate matching rule.

[0071] A high-performance network message classification system provided by the present invention includes:

[0072] Module M1: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into attributes with preset value ranges that meet preset requirements;

[0073] Module M2: configures a set of bits for each attribute whose value range meets the preset requirements, and uses the bit set to record the matching status between the current message attribute value and the corresponding attribute;

[0074] Module M3: extracts the value of the decomposed attribute in the message and retrieves the bit set of the corresponding attribute value;

[0075] Module M4: performs logical AND or logical OR operations on the retrieved bit sets to obtain candidate matching rules;

[0076] Module M5: performs a precise matching check on the candidate matching rules on the port number to obtain the final matching rules;

[0077] Module M6: Classify the message using the final matching rule obtained.

[0078] Specifically, a preset bit set is compressed according to a certain aggregation ratio, and the compressed bit set is subjected to a logical OR / AND operation to obtain a candidate matching rule set.

[0079] Specifically, the module M1 adopts:

[0080] Module M1.1: Decompose the source IP and destination IP attributes into four attributes with a value range of 256, each corresponding to a byte in the IP.

[0081] Module M1.2: Decompose the packet's five-tuple (source IP, destination IP, source port number, destination port number, protocol number) into four attributes derived from the source IP attribute, four attributes derived from the destination IP attribute, the source port number attribute, the destination port number attribute, and the protocol number attribute.

[0082] The attribute value domains of the source port number and the destination port number are decomposed into c subspaces according to preset requirements.

[0083] Specifically, the module M2 adopts:

[0084] Module M2.1: Configure a bit set for each value of the decomposed source IP and destination IP attributes;

[0085] Module M2.2: Configure a bit set for each protocol number and configure a bit set to mark the protocol number wildcard rule;

[0086] Module M2.3: configure a bit set for each port number subspace after decomposition;

[0087] The module M4 adopts: when adopting a reverse query algorithm based on a bit set, a logical OR operation is performed on the bit set in the record rule mismatch state to obtain a candidate matching rule; when adopting a forward query algorithm based on a bit set, a logical AND operation is performed on the bit set in the record rule match state to obtain a candidate matching rule.

[0088] Example 2

[0089] Example 2 is a preferred example of Example 1

[0090] The message classification problem addressed by this invention is that of quintuples. A quintuple is defined as <source IP address, destination IP address, source port number, destination port number, and protocol number> and can be viewed as five fields or attributes with different value ranges and characteristics. The data being queried is the rule set, and the data used for the query is the message set. Each rule consists of five predicates, defining an interval constraint for each attribute. Each message has a fixed value for each of the five attributes.

[0091] For the two attributes of source IP and destination IP, the range is [0,2 32 -1] (i.e., the range of values ​​that can be represented by an unsigned 32-bit integer). The interval predicate defined on the IP attribute is defined by an IP value and a mask. The mask is a non-negative integer whose range is [0,32] and can be represented by m. The mask stipulates that the m binary bits starting from the high bit of the IP value are fixed bits, that is, the message IP must be the same as the IP defined in the rule in these bits to match. Conversely, the 32-m binary bits starting from the low bit are free bits, and each bit can be 0 or 1, so the predicate width on the IP attribute can be expressed as 2 32-m When the source IP mask m = 0, the source IP of the message can be any value and will match the rule. In this case, the rule can be called a wildcard on the source IP. When the source IP mask m = 32, the source IP value of the message must be exactly equal to the source IP value defined by the rule to match the rule.

[0092] The value range of the source port number and destination port number attributes is [0,2 16 -1], the interval predicate defined on the port number is directly defined by two 16-bit unsigned integers as the high value and low value of the predicate respectively, so the predicate width is not necessarily a power of 2.

[0093] The protocol number attribute is defined using two 8-bit unsigned binary integers. The first number is a protocol number, and the second is a protocol mask, which can only take the values ​​0 and 255. When the protocol mask is 0, the protocol number has 0 fixed bits, and the packet's protocol number can take any value. In this case, the match probability is 1. When the protocol mask is 255, the protocol number has 8 fixed bits, meaning it must be equal to the first 8-bit unsigned binary integer for a match to occur. The width of the predicate defined on the protocol number attribute is 1.

[0094] A packet matches a rule only if all five attributes have values ​​that fall within the five intervals defined by the rule. Each rule has a different priority. The goal of packet classification is to quickly find the highest-priority rule from the rule set that matches the incoming packet. A single packet can typically match multiple rules in a rule set. Rules are sorted in descending order of priority. The last rule, with the lowest priority, is typically a wildcard rule that matches any packet. Therefore, the rule set ensures that a single rule exists for each packet as the classification result.

[0095] To achieve rapid message classification, the present invention first proposes a reverse lookup algorithm and a forward lookup algorithm based on bit sets. Bit sets are used to record the numbers of unmatched and matched rules, respectively, and candidate matching rule sets are obtained through logical OR and AND operations. Based on this primary bit set, the present invention proposes a bit set aggregation and compression method, compressing the primary bit set into a secondary bit set at a specific aggregation ratio. During classification, a rapid rule search is first performed on the secondary bit set. If the conditions are not met, a fine-grained rule search is then performed on the primary bit set.

[0096] The message classification algorithm based on bit set reverse lookup includes:

[0097] The core idea of ​​this algorithm is to use bit sets to store the distribution information of predicate values ​​in advance according to the characteristics of different attributes when inserting rules. This serves as a cache. When querying, the bit set storing unmatched rules on the corresponding attribute is directly found through the value of the message. Finally, the unmatched rule sets on each attribute are merged through a logical OR operation.

[0098] Protocol Number Attribute

[0099] For the protocol number attribute, its value is relatively small. Suppose there are a total of R p If the protocol number is R p The protocol values ​​are mapped to 1, 2, ..., R p Directly set a bit set for each protocol number (i represents the i-th protocol number) and records the rule numbers that do not match when the message protocol number is the i-th protocol. The number of bits in each bit set is the total number of rules, and each bit represents the matching status of a rule, 1 indicates a mismatch, and 0 indicates a match.

[0100] The insertion process of the rule on the protocol number attribute is: when the rule with priority r defines that the protocol number must be p, This R p The rth bit of the -1 bit set is marked as 1, indicating no match; when the rule is wildcarded in the protocol number, the mark bit set is not required.

[0101] The process of searching for the protocol number attribute of a message is as follows: (1) When the protocol number of the message is a protocol number p that appears in the rule set, All the rules that do not match the message in the protocol number are recorded. The first 0 in the bit is the rule with the highest priority that matches the message based on the protocol number attribute; (2) When the protocol number of the message is a new protocol number that has not appeared in the rule set, all rules that have constraints on the protocol number attribute will not match. Therefore, a bit set initialized to all 1s can be added. Indicates that each rule does not match by default. When inserting a rule, if the rule is wildcarded in the protocol number attribute, Unmark the rule, and finally It records all the rules with predicate constraints on the protocol number attribute, that is, as a mismatch rule set on the protocol number, which can be used to classify messages whose protocol numbers do not appear in the rule set.

[0102] IP attributes

[0103] For the two IP address type attributes, since their value ranges are too large, the value range space cannot be directly divided equally. Note the following two observations: (1) The IP address is composed of 4 bytes, and the value range of each byte is 0-255; (2) According to the mask, the interval constraints defined on each byte can be directly obtained. For example, 1.2.3.4 / 16, the constraints defined on the four bytes from high to low are [1, 1], [2, 2], [0, 255], [0, 255]. By inverting each interval, we can get the value range of each byte that does not match. Therefore, the two IP attributes can be split into eight independent attributes, and eight interval predicates of the rules can be obtained. This process is called decoding. Let the four attributes from high byte to low byte of the source IP be attributes 1 to 4, and the four attributes from high byte to low byte of the destination IP be attributes 5 to 8. The value range of each attribute is 256. Like the protocol number attribute, a bit set can be set for each value v to record all the rules that do not match when the message takes the value v on this attribute. Thus, a total of 2048 bit sets are required, namely The protocol number can be used as the ninth attribute, and the corresponding bit set can be expressed as

[0104] Port Number Attribute

[0105] For the port number attribute, its value range is 0-65535. On the one hand, setting a bit set for each value like the protocol number attribute will inevitably lead to very large memory consumption. On the other hand, the port number attribute does not specify the constraint interval through a mask. If a port number attribute is forcibly divided into two new attributes according to the high and low bytes like the IP attribute, then the two are not independent of each other, and the value of the low-byte attribute depends on the value of the high-byte attribute. Therefore, the present invention considers a fuzzy search strategy, first dividing the value range of the port number attribute into c subspaces, and the size of each subspace is Then, each subspace is treated as an attribute value, and a bit set is used to store the rule number that may not be matched when the value of the message falls into the subspace.

[0106] For example, if c = 256, there are 256 subspaces in total, and the range of each subspace is also 256. By configuring a separate bit set for each subspace, it is possible to store all mismatched rules whose predicate constraints do not include this subspace. Suppose a rule defines an interval predicate on the port number attribute as [4596, 24021], spanning subspaces 18 to 94. Therefore, the mismatched subspaces are 1 to 17 and 95 to 256. The rule can be marked as mismatched directly in the bit sets corresponding to these subspaces.

[0107] However, the interval predicate [4596, 24021] does not fully encompass subspaces 18 and 94. If a packet's port number falls within subspace 18 but is less than 4596, or falls within subspace 94 but is greater than 24021, it will not match the rule. In other words, if the difference between the packet's port value and the rule's interval constraint boundary on the port number attribute is less than the subspace size of 256, the match is uncertain. Therefore, when classifying packets, bitset indexing can filter out most of the mismatched rules whose interval predicate boundary value differs significantly from the packet's port value. Finally, a precise comparison of the packet's port number with a small number of candidate rules is performed to determine whether the rule matches.

[0108] In summary, the message classification based on the port number attribute can be implemented by fuzzy search, where the source port number and destination port number can be used as the 10th and 11th attributes, and the corresponding bit set can be expressed as

[0109] Bitset-based reverse lookup steps

[0110] The five attributes of protocol number, IP address, and port number are decomposed into 11 attributes with smaller value ranges, and a bit set index is constructed for each attribute to store mismatch rules. The total number of bit sets is 2048+R p+1+2c, the length of each bit set is equal to the number of rules. Specifically, the source IP and destination IP are decomposed into 8 attributes, each attribute value range is 256, one value corresponds to one bit set, so 8*256=2048; suppose there are a total of R p There are protocol numbers, each protocol corresponds to a bit set, and another bit set is needed to mark the wildcard case, so R p +1 bit set; each port is divided into c subspaces, each subspace is represented by a bit set, so there are 2c bit sets.

[0111] The length of each bit set is equal to the number of rules. Each bit in the bit set represents the matching status of the corresponding rule, where forward lookup is used to mark the matching status and reverse lookup is used to mark the non-matching status.

[0112] The final search steps for message classification are:

[0113] (1) Retrieve the bit sets of 11 records that do not match the rules based on the values ​​of the 11 attributes of the message;

[0114] (2) Perform a maximum of 10 OR operations on these bit sets for every 64 binary bits. Each time an OR operation is performed, check whether the result is 64 binary 1s. If so, it means that the rules represented by these 64 bits do not match. At this time, the OR operation can be stopped in this round and the next 64 bits can be processed in a loop.

[0115] (3) If some bits are still 0 after 10 OR operations are performed on 64 bits, it means that there is a rule with the highest priority that may be matched. It may also be a false positive rule caused by fuzzy check on the port number attribute. You only need to check the port number to determine whether it matches and get the query result.

[0116] Bitset-based reverse lookup example

[0117] This invention provides a specific example of reverse lookup based on bit sets. Since the value range of the original problem is too large and too many bit sets need to be configured, it is not convenient to express them all. Here, a simplified example with only IP attributes is used to describe the algorithm: the IP attribute is simplified and split into two attributes a1 and a2. Each attribute value is represented by 3 bits of binary with a value range of [0, 7]. The mask m of the two attributes can be 0 to 6. Given six sample rules such as Figure 1 As shown, the rule ID also represents the priority of the rule. The smaller the ID, the higher the priority.

[0118] According to the construction method of the index on the IP attribute, the predicate defined by each rule on the IP attribute is first decoded into two predicates, that is, the reasonable value range of each rule on a1 and a2 is obtained, such as Figure 2 shown.

[0119] By inverting the constraint interval defined by the rule on each attribute, we can get the values ​​on the attribute that do not match the rule. In the bit set corresponding to these values, the rule is marked as 1 to indicate mismatch. The final inverted bit set index structure is as follows Figure 3 As shown, each attribute has eight bits, corresponding to attribute values ​​0 to 7. Each bit set, from left to right, records the matching status of rules 1 through 6 with the corresponding attribute value. For example, for attribute a1, when the message value is 2, R1, R2, R4, and R5 do not match. Therefore, bit set B2 corresponding to attribute value 2 for attribute a1 is 110110.

[0120] When classifying a message, since there are only two attributes, we can extract the corresponding two bit sets based on the values ​​of the two attributes and perform an OR operation to get the search result. For example, if the message value is 51, it can be decoded into two attributes to get a1·a2=6.3. and Performing an OR operation yields B(101110). The first unmarked bit is bit 2, so R2 is the highest-priority matching rule. This shows that the bitset-based reverse lookup algorithm essentially eliminates the need to compare the high and low values ​​of the rule predicates (which requires port checking on the small number of filtered candidate rules to avoid false positives), completely eliminating arithmetic operations and marking of mismatched predicates during the lookup process.

[0121] Message Classification Algorithm Based on Bitset Forward Lookup

[0122] Based on the same data structure, forward or reverse search algorithms can be designed to store and search for rules that match or mismatch predicates. The bit set supporting reverse search records the rule number corresponding to the mismatch predicate. If the bit set records the rule number corresponding to the matching predicate, a bit set AND operation can be used during search to obtain rules that match the message in all attributes. This allows the design of a message classification algorithm based on forward bit set search.

[0123] Specifically, for the protocol number attribute, when the rule defines that the protocol number must be p, you can directly set the bit set corresponding to p. When the rule is wildcarded on the protocol number attribute, the rule can be marked as a match in the bit set corresponding to each protocol number.

[0124] When the protocol number of the message is a protocol number p that appears in the rule set, It records all the rules that match the message on the protocol number. The first 1 in the bit is the rule with the highest priority that matches the message based on the protocol number attribute.

[0125] When the protocol number of the message is a new protocol number that does not appear in the rule set, only the rule with wildcard on the protocol number attribute is matched. Therefore, a bit set initialized to all 0 can be added. Indicates that none of them match by default. When inserting a rule, if the rule has a wildcard in the protocol number attribute, The rule is marked as matched in the final All the rules that are wildcarded on the protocol number are recorded. The difference from the insertion process based on the bit set reverse lookup strategy is that when the rule is wildcarded on the protocol number, it needs to be marked with R when inserted. p +1 bit set.

[0126] For two IP attributes, a rule can decode eight interval predicates. Each attribute value within these eight intervals matches the rule. Insertion is completed by simply marking the rule as a match in the bit sets corresponding to these values. During message search, the eight values ​​of these eight attributes are first decoded from the message. The eight bit sets corresponding to these eight values, which contain all matching rules for each attribute, are then extracted. A logical AND operation is then performed to obtain a rule that matches the message on all eight decoded attributes.

[0127] For the port number attribute, simply mark the rule in the bit set corresponding to the subspace that the rule fully or partially covers. This allows the message's port number to directly retrieve the bit set storing all matching rules for that attribute. However, as with reverse lookup, there may be false positive rules, requiring further inspection of the port numbers of candidate matching rules. In the example of the port number attribute, [4596,24021] spans subspaces 18 to 94. For forward lookup, the rule is simply marked in the bit set corresponding to these subspaces.

[0128] The bitset-based reverse lookup algorithm uses a bitset to mark all mismatching rules when a message takes a certain value on a certain attribute. The forward lookup algorithm uses a bitset to mark all matching rules when a message takes a certain value on a certain attribute. During the search, the former iteratively performs a 64-bit logical OR operation on the bitsets of the mismatching states of the 11 attributes determined by the message value to obtain candidate matching rules. The latter iteratively performs a 64-bit logical AND operation on the bitsets of the matching states of the 11 retrieved rules to obtain candidate matching rules. Both algorithms require precise port number checks on a small number of candidate matching rules to determine whether further logical operations are needed to find the highest-priority matching rule. For the example in Section 2, the forward lookup algorithm omits the negation step and directly marks the rule in the bitset corresponding to each value in the interval shown in Table 2. This is reflected in Table 3, where each bitset changes from 1 to 0 and vice versa, and the marked bit changes from indicating a mismatch to indicating a match.

[0129] Bitset aggregation optimization

[0130] Based on the following two observations: (1) Each message matches very few rules, meaning the matching degree is very low. The bit set supporting forward search will inevitably have many 0s and sparsely distributed 1s, while the reverse is true for the bit set supporting reverse search. (2) The performance bottleneck in the search process is bit set operations and comparison operations. The amount of computation is inversely proportional to the priority of the matching rule; the lower the priority, the greater the amount of computation. Therefore, it is possible to consider aggregating and compressing k binary bits into a single binary bit representation. One bit set operation can replace the previous k bit set operations, thereby reducing the number of bit set operations and the number of comparisons required to determine whether to stop the bit set operation early in each 64-bit iteration, thereby alleviating the performance bottleneck.

[0131] Specifically, let the aggregation compression ratio be k (hereinafter referred to as the aggregation ratio), which means that every k bits are aggregated into 1 bit. Let the bit set before aggregation be the primary bit set, the bit set after aggregation be the secondary bit set or the aggregated bit set, the binary bits on the primary bit set are the primary bits, and the binary bits on the secondary bit set are the secondary bits or the aggregated bits. Let the total number of rules be n, then the primary bit set consists of n bits, and the secondary bit set consists of n bits. In actual implementation, a 64-bit unsigned integer array is used to represent the bit set, with each 64 bits being a storage unit. A first-level bit set consists of units, a secondary bit set consists of It consists of units.

[0132] For forward search, if the i-th consecutive k bits in the first-level bit set are all 0, indicating that the k rules from i·k+1 to (i+1)·k are not matched, then the i-th aggregation bit of the second-level bit set is 0; otherwise, as long as at least one of the i-th consecutive k bits in the first-level bit set is 1, indicating that there may be a matching rule, then the i-th aggregation bit of the second-level bit set is 1.

[0133] The steps for bitset-based forward lookup after aggregation optimization are:

[0134] (1) Retrieve the secondary bit sets on 11 attributes based on the message value;

[0135] (2) Process each storage unit of these 11 secondary bit sets, that is, perform a logical AND operation on every 64 bits, and perform a maximum of 10 logical AND operations on the 11 storage units;

[0136] (3) Each time a logical AND operation is performed, the result of the operation is judged to be 0. If it is 0, it means that the 64 k rules represented by these 64 aggregation bits are not matched. The 10 unfinished 64-bit logical AND operations in this round can be stopped in time, and the next round of 64 aggregation bits can be processed;

[0137] (4) Otherwise, if the result is still not 0 after 10 logical AND operations, find out which aggregation bits are 1, and then retrieve the storage units in the 11 first-level bit sets represented by them (the retrieval logic of the first-level bit set storage units is different when k < 64 and k ≥ 64);

[0138] (5) Directly perform 10 64-bit logical AND operations on the 11 storage units of the 11 first-level bit sets to determine which one or several rules may be matched, and finally check whether the port number matches to determine the message search result.

[0139] Bitset aggregation optimization can also be performed on bitsets that support reverse lookup. The difference is that only when all consecutive k bits are 1, that is, all do not match, can they be aggregated and compressed into 1; otherwise, they are aggregated into 0.

[0140] Those skilled in the art will appreciate that, in addition to implementing the system, device, and various modules provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same program in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, embedded microcontrollers, and the like by logically programming the method steps. Therefore, the system, device, and various modules provided by the present invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; the modules for implementing various functions can also be considered both software programs for implementing the method and structures within the hardware component.

[0141] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art may make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. The embodiments of this application and the features in the embodiments may be combined with each other in any manner unless there is a conflict.

Claims

1. A high-performance network message classification method, characterized in that: include: Step S1: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into attributes with preset value ranges that meet preset requirements; Step S2: configuring a set of bit sets for each attribute whose value range meets the preset requirements, and using the bit sets to record the matching status between the current message attribute value and the corresponding attribute; Step S3: extract the value of the decomposed attribute in the message and retrieve the bit set corresponding to the attribute value; Step S4: performing a logical AND or logical OR operation on the retrieved bit set to obtain a candidate matching rule; Step S5: Perform a precise matching check on the candidate matching rule on the port number. When the current matching rule matches the current port number, the final matching rule is obtained. Step S6: Classify the message using the final matching rule obtained; Compress the bit set according to a certain aggregation ratio, and obtain a candidate matching rule set based on the compressed bit set using logical OR / AND operations; The step S1 adopts: Step S1.1: Decompose the source IP and destination IP attributes into four attributes with a value range of 256, each attribute corresponding to a byte in the IP; Step S1.2: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into four attributes obtained by decomposing the source IP attribute, four attributes obtained by decomposing the destination IP attribute, the source port number attribute, the destination port number attribute, and the protocol number attribute; Decompose the attribute value ranges of the source port number and destination port number attributes into c subspaces according to the preset requirements; The step S2 adopts: Step S2.1: configuring a bit set for each value of the decomposed source IP and destination IP attributes; Step S2.2: configure a bit set for each protocol number, and configure a bit set for marking a wildcard rule for the protocol number; Step S2.3: configure a bit set for each decomposed port number subspace; The step S4 adopts: when a reverse query algorithm based on a bit set is adopted, a logical OR operation is performed on the bit set in the record rule mismatch state to obtain a candidate matching rule; when a forward query algorithm based on a bit set is adopted, a logical AND operation is performed on the bit set in the record rule match state to obtain a candidate matching rule.

2. A high-performance network message classification system, characterized in that: include: Module M1: Decompose the five-tuple of the message <source IP, destination IP, source port number, destination port number, protocol number> into attributes with preset value ranges that meet preset requirements; Module M2: configures a set of bits for each attribute whose value range meets the preset requirements, and uses the bit set to record the matching status between the current message attribute value and the corresponding attribute; Module M3: extracts the value of the decomposed attribute in the message and retrieves the bit set of the corresponding attribute value; Module M4: performs logical AND or logical OR operations on the retrieved bit sets to obtain candidate matching rules; Module M5: performs a precise matching check on the candidate matching rules on the port number to obtain the final matching rules; Module M6: classify the message using the final matching rule obtained; Compress the preset bit sets according to a certain aggregation ratio, and use logical OR / AND operations on the compressed bit sets to obtain a candidate matching rule set; The module M1 adopts: Module M1.1: Decompose the source IP and destination IP attributes into four attributes with a value range of 256, each corresponding to a byte in the IP. Module M1.2: Decompose the packet's five-tuple (source IP, destination IP, source port number, destination port number, protocol number) into four attributes derived from the source IP attribute, four attributes derived from the destination IP attribute, the source port number attribute, the destination port number attribute, and the protocol number attribute. Decompose the source port number and destination port number attributes into c subspaces according to the preset requirements; The module M2 adopts: Module M2.1: Configure a bit set for each value of the decomposed source IP and destination IP attributes; Module M2.2: Configure a bit set for each protocol number and configure a bit set to mark the protocol number wildcard rule; Module M2.3: configure a bit set for each port number subspace after decomposition; The module M4 adopts: when adopting a reverse query algorithm based on a bit set, a logical OR operation is performed on the bit set in the record rule mismatch state to obtain a candidate matching rule; when adopting a forward query algorithm based on a bit set, a logical AND operation is performed on the bit set in the record rule match state to obtain a candidate matching rule.

Images