A Tuple Space Search Method for Accelerating Package Classification Based on Learning Indexes

Abstract

This invention provides a method for accelerating tuple space search and package classification based on learned indexes. It constructs ordered tuples based on hash tables by setting partition thresholds and tuple insertion constraints, and uses a linear model based on gap arrays to learn the index to achieve stable and efficient rule lookup and fast rule updates, while ensuring low memory overhead.

Description

Technical Field

[0001] This invention belongs to the technical field of packet classification in the field of computer networks, specifically relating to a tuple space search method for accelerating packet classification based on a learned index. Background Technology

[0002] Packet classification is a fundamental component of high-performance systems and infrastructure for the Web and WOT. By finding the best-matching rule and applying specific operations to the packet, it supports many web applications, including web firewalls, intrusion detection, and web service routers and gateways. The tuple space search packet classification algorithm is widely used due to its high rule update speed and low memory consumption. In this algorithm, the rule set is divided into tuples based on the prefix length of the rules. At runtime, all tuples need to be traversed to find the best-matching rule. Therefore, as the number of rules increases, the explosive growth in the number of tuples slows down the search speed.

[0003] Existing solutions accelerate tuple traversal by partitioning tuples, combining them with other packet classification techniques, and merging tuples. However, these methods cannot scale with the number and distribution of rules. Specifically, partition-based solutions result in high tree depths under highly skewed rule distributions and large rule sets, while combination-based solutions lead to extensive rule duplication in the decision tree under large rule sets. Merging-based solutions take a long time to find the update position due to tuple collisions caused by a wide range of rules. In contrast, learned indexes trained according to rules can adapt to different data sizes and distributions, and achieve faster search speeds and limited memory overhead compared to traditional tree indexes. Summary of the Invention

[0004] To address the shortcomings and deficiencies of existing technologies, this invention provides a tuple space search packet classification acceleration method based on a learned index, enabling fast rule lookup and rule updates while ensuring limited memory overhead. By setting partition thresholds and tuple insertion constraints, ordered tuples based on hash tables are constructed. A learned index model combining a linear model with gap arrays is used to find the tuple containing the packet matching rule, achieving stable and efficient rule lookup and fast rule updates while ensuring low memory consumption.

[0005] The specific technical solution adopted by this invention to solve its technical problem is as follows:

[0006] A method for accelerating bag classification based on tuple space search using learned indexes, characterized by:

[0007] Constructing ordered tuples based on hash tables and leveraging learned indexes to improve the rule lookup speed of the tuple space search algorithm to maintain fast rule updates and ensure low memory overhead includes the following steps:

[0008] The element group partition is determined based on the distribution of the rule prefix length;

[0009] Based on the characteristics of the rule set, constraints are set, and the rules are inserted into different tuples;

[0010] Based on the partitioning results, construct an ordered tuple data structure based on a hash table;

[0011] Based on existing tuple information, construct a learning index;

[0012] Achieve fast rule matching and rule updating within existing data structures.

[0013] Furthermore, the threshold for partitioning the metagroup is determined based on the rule set trie. First, the trie is constructed using the address field of the rule: the tree depth is used to represent the distribution of the rule prefix length, and the weight of a node is defined as the number of rules in the subtree rooted at that node, representing the number of rules that satisfy the current prefix; the tree depth and the weight of the node together determine the threshold for partitioning.

[0014] The specific threshold determination method is as follows: traverse the trie from bottom to top, calculate the weight of each node, and stop traversing the branch when the following condition is met:

[0015] The number of node layers is less than half the average depth;

[0016] The change in node weight is greater than half of the cumulative rule count.

[0017] Furthermore, the rule insertion is subject to constraints such as the number of rules in the tuple, the overlap of internal rules, and the range of tuples. When a rule does not meet the insertion conditions of the current tuple, it is placed in another tuple. When a rule does not meet the insertion conditions of all existing tuples, a new tuple is constructed and the rule is inserted.

[0018] Furthermore, the specific constraints for rule insertion are as follows:

[0019] When the rule allows insertion of a tuple, the following conditions must be met:

[0020] The number of rules in a tuple does not exceed the set maximum number of rules per tuple;

[0021] The number of rule overlaps in a tuple does not exceed the set maximum number of rule overlaps.

[0022] The scope of a rule is smaller than the scope defined by a tuple.

[0023] Furthermore, in the hash table-based ordered tuple data structure, different tuples represent different prefix lengths, the rules in the same tuple are divided into several groups, the rules in each group share the same hash key, and the rules in the group are stored in a linked list in order of priority.

[0024] Furthermore, the learning index consists of multiple layers of models, each of which uses a linear regression model. The output of each model is the range of tuples to be searched. The root node contains only one model, corresponding to the entire tuple space. The tuple space is divided into more refined subspaces until the error bound is reached.

[0025] Furthermore, the sub-model is represented as LM. i，j Where 1≤i≤k and 1≤j≤m; k and m represent the depth and width of the index, respectively; LM i，j It consists of a linear regression model, namely:

[0026] N i，j (x)=ωx+b

[0027] Where N i，j (x) is the output of this type of linear model, where x is the scalar input derived from the data packet, and ω and b are the weight and bias row vectors; the model LM i，j The output is represented as M i，j (x), that is:

[0028] M i，j (x)=H(N i，j (x))

[0029] Where H(x) is a function used to ensure that the output is normalized to [0, 1). Since each node retains its first position node.start and slope node.slope, the initial calculation of the input key position is as follows:

[0030] pred pos (x)=(x-node.start)×node.slope

[0031] The error of the learned index based on the linear model is as follows, for

[0032] error = max(pred pos (x)-true pos (x))

[0033] Therefore, the range of the actual index is estimated as follows:

[0034] true pos (x)∈[pred pos (x), pred pos (x)+error]

[0035] Considering the presence of a large number of tuples, the error range of the leaf nodes in the index structure may cover multiple tuples; therefore, for the training part of the linear regression model learning the index, the loss function is defined as follows:

[0036]

[0037]

[0038] Where y and These are the actual location and the predicted location, respectively. If the original destination IP is used, then n=2; sample m data packets and recursively train the model for each layer using gradient descent, i.e.:

[0039]

[0040] Where α is the pre-configured learning rate, Indicate j th sample i th feature.

[0041] Furthermore, to achieve rapid updates, a gap array is used in the learning index.

[0042] Furthermore, the rule lookup is achieved by setting two independent hash functions; hash 1 is used to handle short prefix rule conflicts within a tuple, and hash 2 is used to handle rule conflicts with equal prefixes; during matching, hash 2 is executed first, followed by hash 1.

[0043] As a preferred approach, the two hash functions can be implemented in a way that is independent of each other and responds to the two types of collisions separately.

[0044] Furthermore, the rule update classifies different types of rule insertions; for inserting exact matching rules, it is treated as a data packet; based on its position, it is indexed to the corresponding tuple and inserted; for rules with prefix or range matching, constraint-based rule insertion is performed: if no tuple can be inserted, the rule is inserted into the remaining tuples; when inserting a large number of rules, tuples with too many rules are split, and the corresponding nodes in the index structure are also split.

[0045] Compared with existing technologies, this invention and its preferred embodiment propose a learning index-based tuple space search packet classification acceleration method, which can accelerate tuple space search in packet classification, while achieving fast rule updates and low memory usage. Attached Figure Description

[0046] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0047] Figure 1 This is a schematic diagram of the main steps of an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of an ordered tuple data structure based on a hash table, according to an embodiment of the present invention.

[0049] Figure 3 This is a schematic diagram of the structure of the learning index in the linear regression model according to an embodiment of the present invention.

[0050] Figure 4 This is a schematic diagram of the overall process of an embodiment of the present invention. Detailed Implementation

[0051] To make the features and advantages of this patent more apparent and understandable, specific embodiments are provided below for detailed explanation:

[0052] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0053] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0054] like Figure 4 As shown, the embodiment of the present invention employs the following method: determining tuple partitions based on the rule prefix length distribution; setting constraints based on rule set characteristics and inserting rules into different tuples; constructing an ordered tuple data structure based on a hash table based on the partitioning results; building a learning index based on existing tuple information; and achieving fast rule matching and rule updating within the existing data structure. This includes the following steps:

[0055] 1) Determine the partition threshold based on the rule set trie. A trie is constructed using the rule address field. The tree depth represents the distribution of rule prefix lengths, and the weight of a node is defined as the number of rules in the subtree rooted at that node, representing the number of rules satisfying the current prefix. Both together determine the partition threshold.

[0056] 2) Rule insertion with constraints. Rule insertion is subject to constraints such as the number of rules in the tuple, the overlap of internal rules, and the tuple range. When a rule does not meet the insertion conditions of the current tuple, it is placed in another tuple. When a rule does not meet the insertion conditions of all existing tuples, a new tuple is constructed for it and the rule is inserted.

[0057] 3) Construct an ordered tuple data structure based on hash tables. Rules in the same tuple are divided into several groups, and the rules in each group share the same hash key. The rules within a group are stored in a linked list in order of priority.

[0058] 4) Construct a linear model learning index based on a gap array. The learning index consists of multiple layers of models, each employing a linear regression model. The output of each model represents the range of tuples to be searched. The root node contains only one model, corresponding to the entire tuple space. The tuple space is further subdivided into finer subspaces until the error bound is reached. To achieve rapid updates, a gap array is used in the learning index.

[0059] 5) Fast lookup based on dual hash functions. The fast lookup method is implemented by setting two independent hash functions. Hash 1 handles short prefix rule collisions within a tuple, and hash 2 handles rule collisions with equal prefixes. During matching, hash 2 is executed first, followed by hash 1. In the design of the hash functions, the two hash functions can be implemented independently, each responding to one of the two types of collisions respectively.

[0060] 6) Fast Rule Update. The fast rule update method categorizes different types of rule insertion. For inserting exact match rules, it is treated as a data packet. Based on its position, it is indexed to the corresponding tuple and inserted. For rules with prefix or range matches, a constraint-based rule insertion similar to that in tuple construction is performed; if no tuple can be found for insertion, the rule is inserted into the remaining tuples. When inserting a large number of rules, tuples with too many rules are split, and the corresponding nodes in the index structure are also split.

[0061] The design of the present invention will be further described below with a specific embodiment example.

[0062] Please refer to Figure 1 The overall scheme of this invention includes three parts: tuple data structure construction, rule lookup, and rule set update.

[0063] (1) Construction of tuple data structure

[0064] The construction process consists of three parts: tuple partitioning, rule insertion, and tuple construction.

[0065] For a metagroup partition, the threshold is determined by the rule set trie. The trie is constructed using the rule address field; its depth represents the distribution of rule prefix lengths; and the weight of a node is defined as the number of rules in the subtree rooted at that node, representing the number of rules satisfying the current prefix. Both together determine the partition threshold. Specifically, the trie is traversed from bottom to top, calculating the weight of each node. Traversal of a branch stops when the following condition is met:

[0066] 1) The number of node layers is less than half the average depth;

[0067] 2) The change in node weight is greater than half of the cumulative rule count.

[0068] The insertion of rules into each tuple is constrained by the number of rules in the tuple, the overlap of internal rules, and the tuple's range. When a rule does not meet the insertion conditions of the current tuple, it is placed into another tuple. When a rule does not meet the insertion conditions of all existing tuples, a new tuple is constructed for it and the rule is inserted, as shown in the constraint-based rule insertion algorithm. When a rule is allowed to be inserted into a tuple, the number of rules in the tuple does not exceed the set maximum number of rules in the tuple, the overlap of rules in the tuple does not exceed the set maximum overlap of rules, and the range of the rule is smaller than the range specified by the tuple.

[0069]

[0070]

[0071] For ordered tuple data structures based on hash tables, please refer to [reference needed]. Figure 2 Rules within the same tuple are divided into several groups, and the rules in each group share the same hash key. The rules within a group are stored in a linked list in order of priority.

[0072] (2) Rule lookup

[0073] For the rule lookup process, a learned index and a double hash function are used to accelerate the search. The range of tuples where rules matching the data packet might exist is obtained by searching the learned index. A double hash function is then used within these tuples to speed up the search, and finally, the remaining tuples are searched. Combining these search steps, the matching rule is obtained, and the corresponding operation is performed.

[0074] Please refer to the learning index. Figure 3 The learning index consists of multiple layers of models, each employing a linear regression model. The output of each model represents the range of tuples to be searched. The root node contains only one model, corresponding to the entire tuple space. The tuple space is further subdivided into finer subspaces until the error bound is reached.

[0075] The model calculation formula is as follows, and the sub-model is represented as LM. i，j , where 1≤i≤k and 1≤j≤m. k and m represent the depth and width of the index. LM i，j It consists of a linear regression model, namely:

[0076] N i，j (x)=ωx+b

[0077] Where N i，j(x) is the output of this type of linear model, where x is the scalar input derived from the data packet, and ω and b are the weight and bias row vectors. Then the model LM... i，j The output is represented as M i，j (x), that is

[0078] M i，j (x)=H(N i，j (x))

[0079] Where H(x) is a function that ensures the output is normalized to [0, 1); since each node retains its first position and slope, the approximate position of the input key is calculated as follows:

[0080] pred pos (x)=(x-node.start)×node.slope

[0081] The error of the learned index based on the linear model is as follows, for

[0082] error = max(pred pos (x)-true pos (x))

[0083] Therefore, the range of the actual index can be estimated as follows:

[0084] true pos (x)∈[pred pos (x), pred pos (x)+error]

[0085] When there are a large number of tuples, the error range of the leaf nodes in the index structure may cover multiple tuples.

[0086] For the training part of the linear regression model learning index, the loss function is defined as follows:

[0087]

[0088]

[0089] Where y and These are the actual and predicted locations. Since the original and destination IPs are used, n=2. The m data packets are sampled, and gradient descent is used to recursively train the model for each layer.

[0090]

[0091] Where α is the pre-configured learning rate, Indicate j th sample i th feature.

[0092] To enable rapid updates, a gap array was used in the learning index.

[0093] For a dual hash function, hash1 handles short prefix rule collisions within a tuple, while hash2 handles rule collisions with equal prefixes. During matching, hash2 is executed first, followed by hash1. In the design of the hash function, the two hash functions can be implemented independently, each responding to one of the two types of collisions separately.

[0094] (3) Rule Update

[0095] The rule update method categorizes different types of rule insertion. For inserting exact match rules, they are treated as a data packet. Based on their position, the corresponding tuple is indexed and inserted. For rules with prefix or range matches, a constraint-based rule insertion similar to that in tuple construction is performed; if no tuple can be found for insertion, the rule is inserted into the remaining tuples. When inserting a large number of rules, tuples with too many rules are split, and the corresponding nodes in the index structure are also divided.

[0096] The rule update process is shown in Algorithm 2.

[0097]

[0098] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0099] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0100] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0101] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0102] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

[0103] This patent is not limited to the above-described preferred embodiment. Anyone can derive other forms of learning index-based tuple space search package classification acceleration methods based on this patent. All equivalent changes and modifications made within the scope of this patent application shall fall within the scope of this patent.

Claims

1. A method for accelerating bag classification based on tuple space search using learned indexes, characterized by: Constructing ordered tuples based on hash tables and leveraging learned indexes to improve the rule lookup speed of the tuple space search algorithm to maintain fast rule updates and ensure low memory overhead includes the following steps: The element group partition is determined based on the distribution of the rule prefix length; Based on the characteristics of the rule set, constraints are set, and the rules are inserted into different tuples; Based on the partitioning results, construct an ordered tuple data structure based on a hash table; Based on existing tuple information, construct a learning index; Achieve fast rule matching and rule updating within existing data structures; The learning index consists of multiple layers of models, each layer employing a linear regression model. The output of each model is the range of tuples to be searched. The root node contains only one model, corresponding to the entire tuple space. The tuple space is further divided into finer subspaces until the error bound is reached. The model is represented as ,in and ; and These represent the depth and width of the index, respectively. : in It is the output of the model. It is a scalar input derived from the data packet. and These are the weight and bias row vectors; the model The output is represented as ,Right now: in It is a function used to ensure that the output is normalized. Because each node maintains its first position and slope Therefore, the preliminary calculation of the input key position is as follows: The error of the learned index based on the linear regression model is as follows, for : Therefore, the range of the actual index is estimated as follows: Considering the presence of a large number of tuples, the error range of the leaf nodes in the index structure may cover multiple tuples; therefore, for the training part of the linear regression model learning the index, the loss function is defined as follows: in and These are the actual location and the predicted location, respectively. If the original destination IP is used, then... ;right Data packets are sampled, and gradient descent is used to recursively train the model for each layer, i.e.: in It is a pre-configured learning rate. express The i-feature of the sample; To enable rapid updates, a gap array is used in the learning index.

2. The method for accelerating tuple space search and bag classification based on learning indexes according to claim 1, characterized in that: The threshold for the metagroup partitioning is determined based on the rule set trie. First, the trie is constructed using the address field of the rule: the tree depth is used to represent the distribution of the rule prefix length, and the weight of a node is defined as the number of rules in the subtree rooted at that node, representing the number of rules that satisfy the current prefix. The tree depth and node weights together determine the partition threshold; The specific threshold determination method is as follows: traverse the trie from bottom to top, calculate the weight of each node, and stop branching when the following condition is met: The number of node layers is less than half the average depth; The change in node weight is greater than half of the cumulative rule count.

3. The method for accelerating tuple space search and bag classification based on learning indexes according to claim 1, characterized in that: The rule insertion is subject to constraints based on the number of rules in the tuple, the overlap of internal rules, and the range of tuples. If a rule does not meet the insertion conditions of the current tuple, it is inserted into another tuple. If a rule does not meet the insertion conditions of all existing tuples, a new tuple is constructed and the rule is inserted.

4. The method for accelerating tuple space search and bag classification based on learning indexes according to claim 3, characterized in that: The specific constraints for rule insertion are as follows: When the rule allows insertion of a tuple, the following conditions must be met: The number of rules in a tuple does not exceed the set maximum number of rules per tuple; The number of rule overlaps in a tuple does not exceed the set maximum number of rule overlaps. The scope of a rule is smaller than the scope defined by a tuple.

5. The method for accelerating tuple space search and bag classification based on learning indexes according to claim 1, characterized in that: In the hash table-based ordered tuple data structure, different tuples represent prefixes of different lengths. Rules in the same tuple are divided into several groups, and the rules in each group share the same hash key. The rules within a group are stored in a linked list in order of priority.

6. The method for accelerating tuple space search and bag classification based on learning indexes according to claim 1, characterized in that: The rule lookup is achieved by setting two independent hash functions: hash 1 is used to handle short prefix rule collisions within a tuple, and hash 2 is used to handle rule collisions with equal prefixes. When matching, hash 2 is executed first, followed by hash 1.

7. The method for accelerating tuple space search and bag classification based on learning indexes according to claim 1, characterized in that: The rule update is performed by classifying different types of rule insertions; for inserting exact match rules, they are treated as a data packet; based on their position, they are indexed to the corresponding tuple and inserted; for rules with prefix or range matching, constraint-based rule insertion is performed: if no tuple can be inserted, the rule is inserted into the remaining tuples; when inserting a large number of rules, tuples with too many rules are split, and the corresponding nodes in the index structure are also split.

Images