Method and device for preprocessing decision tree rule sets
By preprocessing the decision tree rule set, large rules are identified and labeled, and divided into small rules and large rules. Large rules are then searched in parallel using hardware, which solves the problems of excessive decision tree creation time and excessive storage resource consumption, thus achieving efficient decision tree creation.
Patent Information
- Application Number
- CN202111038482.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-06
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2041-09-06
AI Technical Summary
The problems of excessive decision tree creation time and excessive storage resource consumption are mainly due to the high number of rule replications.
By using a decision tree rule set preprocessing method, large rules are identified and labeled. New thresholds are set for tree height and the number of rules in leaf nodes. The rule set is divided into two parts: small rules and large rules. Large rules are searched in parallel using hardware, while the decision tree only creates small rules.
It significantly reduces decision tree creation time and storage resource consumption, reduces tree height, reduces logical resources, and improves processing efficiency.
Smart Images

Figure CN113723548B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of communication technology, and in particular relates to a method and device for preprocessing a decision tree rule set. Background Art
[0002] Packet classification algorithms are commonly used in networks. Their purpose is to classify packets based on their header information, typically including source IP address, destination IP address, source port number, destination port number, and protocol number, allowing for differentiated processing. Common applications include access control lists, firewalls, and flow-based billing and statistics. They are used in both core and edge network devices. Decision trees are a common method among packet classification algorithms.
[0003] The key to packet classification using a decision tree is tree creation. This involves dividing the rules of a parent node into multiple child nodes as evenly as possible based on the characteristics of the rule set. The root node contains all the rules in the rule set. After multiple levels of tree node division, the leaf nodes contain only a specified number of rules. This allows a large rule set to be divided into multiple smaller rule sets, allowing packet searches to be performed within the specified smaller rule sets. When searching for a packet, the search starts at the root node, using the information generated during decision tree creation, and continues until the leaf node is reached. The search is then performed within the smaller rule sets contained within the leaf nodes.
[0004] Figure 1 The following example shows a rule: source IP address value / source IP mask, destination IP address value / destination IP mask, source port number range, destination port number range, and protocol number value / protocol number mask value. Each rule represents a range, and a rule with a smaller mask represents a larger range.
[0005] In decision tree implementations, since rules represent a range, when splitting rules from a parent node to child nodes, the same rule may be split into multiple child nodes. This situation is referred to as rule duplication in this patent. Rule duplication can lead to excessive decision tree creation time and excessive storage resource consumption. After a decision tree is created, the number of times each rule in the rule set is replicated varies. To ensure efficient decision tree creation, rules with many replications can be removed from the rule set and the decision tree recreated. In this patent, rules with many replications are referred to as large rules.
[0006] The first type of large rules is a rule with a large scope, i.e. a rule with a small IP mask, a large port number range, or a small protocol number mask. Figure 2Rule 5 in the example above represents a relatively large scope, and its number of replications is relatively high. The second category: rules that overlap each other. Although the scope of a single rule is small, the scopes of multiple similar rules overlap each other. Figure 3 The scopes represented by rules 1 through 5 are not large, but they overlap and cannot be evenly divided into different subnodes, which also results in an increase in rule replication. Therefore, based on the number of replications of these five rules, a rule with a relatively high number of replications should be identified as a large rule.
[0007] The first category of large rules can be easily identified by the size of the IP mask and the port number range. However, identifying several groups of overlapping rules among tens of thousands or even hundreds of thousands of rules, and determining which rules within each overlapping rule group are large rules, is more challenging. Embodiments of the present invention provide a method for preprocessing rule sets that accurately identifies the two categories of large rules within a rule set. Preprocessing the rule set before creating a decision tree reduces rule duplication and addresses the issues of lengthy decision tree creation times and excessive storage resource consumption. Summary of the Invention
[0008] (1) Technical issues to be resolved
[0009] The technical problem to be solved by the present invention is how to provide a method and device for preprocessing a decision tree rule set to solve the problems of long decision tree creation time and excessive consumption of storage resources.
[0010] (2) Technical solution
[0011] In order to solve the above technical problems, the present invention proposes a method for preprocessing a decision tree rule set, which includes the following steps:
[0012] Step 1: Determine whether the number of current child node rules is greater than the upper limit threshold of leaf node rules pre_leaf_rules_max. If it is less than or equal to the threshold, execute step 7 and subsequent steps; otherwise, execute step 2.
[0013] Step 2: Determine whether the child node still has a valid division dimension; if the current child node does not have a valid division dimension, execute step 7 and subsequent steps; otherwise, execute step 3.
[0014] Step 3: Determine whether the specified dimension still has a partition bit; if the specified dimension does not have a partition bit, execute step 7 and subsequent steps; otherwise, execute step 4;
[0015] Step 4: Determine whether the height of the current child node exceeds the tree height upper limit threshold pre_tree_height_max; if the tree height of the current child node is greater than or equal to the threshold, execute step 7 and subsequent steps; otherwise, execute step 5 and step 6;
[0016] Step 5: The number of copies of each rule of this child node is increased by 1. Step 6: This child node serves as the new parent node and continues to divide the rules level by level. After executing step 6, the child node serves as the new parent node and executes steps 1 to 4 again until all nodes have been divided by rules and each child node can execute step 7 and subsequent steps.
[0017] Step 7: The child node is set as a leaf node. Step 8: The number of copies of each rule of the child node is increased by 1, indicating that the pre-tree is successfully built.
[0018] Furthermore, multiple thresholds are pre-set before the decision tree is created: the upper limit of tree height tree_height_max, the upper limit of the number of leaf node rules leaf_rules_max, and in the preprocessing stage, new thresholds are set for the pre-built tree: the upper limit of tree height pre_tree_height_max and the upper limit of the number of leaf node rules pre_leaf_rules_max, the upper limit of tree height pre_tree_height_max is greater than tree_height_max, and the upper limit of leaf node rules pre_leaf_rules_max is greater than leaf_rules_max.
[0019] Furthermore, a counting variable is maintained for each rule. In the initial state, all rules in the rule set belong to the root node of the decision tree, and all counting variables are initialized to zero. In the process of creating the tree, the rules are divided layer by layer from the root node to the lower nodes until the number of rules contained in the lower node is less than the upper limit of the number of leaf node rules pre_leaf_rules_max, and the node is marked as a leaf node; each time a rule is divided from a parent node to a child node, the counting variable corresponding to the rule is incremented by one, that is, the number of times the rule is copied is incremented by one.
[0020] Furthermore, each node in the decision tree includes the following key information:
[0021] 1) Current node tree height tree_height: indicates the number of layers from the root node to the current node;
[0022] 2) The current node's partitioning dimension cut_dim: 5 bits in total, with bits 0 to 4 representing the protocol number, destination port number, source port number, destination IP address, and source IP address. When a bit in cut_dim is 1, it indicates that the corresponding dimension is used as the partitioning dimension.
[0023] 3) The number of bits used by the current node as the partitioning bits in the specified partitioning dimension cut_num[5]: represented by an array containing 5 elements, from cut_num[0] to cut_num[4], which represent the number of partitioning bits in the five dimensions of protocol number, destination port number, source port number, destination IP address and source IP address respectively;
[0024] 4) The sum of the number of partition bits of each node on the path from the root node to the parent node of the current node is called the cumulative number of partition bits of the current node, represented by accu_cut_num[5]. From accu_cut_num[0] to accu_cut_num[4], they represent the cumulative number of partition bits of the five dimensions of protocol number, destination port number, source port number, destination IP address, and source IP address respectively;
[0025] 5) Pointer to child node memory chi ldren_ptr: Multiple child nodes of the parent node are stored in a continuous memory block, and chi ldren_ptr points to the first address of the memory block;
[0026] 6) Number of child nodes chi ld_num: Calculate the sum of the number of partition bits corresponding to the partition dimension, and the number of child nodes chi ld_num is: 2 to the power of cut_num_sum.
[0027] Furthermore, the method for dividing parent node rules into child nodes includes: when the parent node uses only one dimension i as a division dimension, the number of division bits is cut_num[i], and the cumulative number of division bits is accu_cut_num[i]; when a rule r of the parent node needs to be divided into a child node, the value of the rule in dimension i from the accu_cut_num[i]th bit to the accu_cut_num[i]+cut_num[i]-1th bit is obtained, and this value is the child node index value idx, indicating that the rule r will be divided into the idxth child node of the parent node, and when the mask value mask of dimension i is less than accu_cut_num[i]+cut_num[i], that is, the mask bit to the accu_cut_num[i]+cut_num[i]th bit of dimension i is wildcard, so the rule r will be divided into multiple child nodes.
[0028] Furthermore, the method for dividing the parent node rule into child nodes includes: when two dimensions i and j are used as division dimensions, obtaining the value IDXi of the division dimension i from the accu_cut_num[i]th bit to the accu_cut_num[i]+cut_num[i]-1th bit; obtaining the value IDXj of the division dimension j from the accu_cut_num[j]th bit to the accu_cut_num[j]+cut_num[j]-1th bit; after shifting IDXj left by cut_num[i] bits, performing a bitwise OR operation with IDXi to obtain the index value idx, which is the index of the child node, indicating that the rule r will be divided into the idxth child node of the parent node; similarly, due to the wildcard mask, the rule r may be divided into multiple child nodes.
[0029] Furthermore, the judgment method of step 2 is as follows: the maximum number of bits occupied by the data of each dimension is fixed. When the cumulative number of division bits accu_cut_num of each dimension of the child node is equal to the corresponding maximum number of bits, there is no valid division dimension for the child node;
[0030] The judgment method of step 3 is: if the masks of all rules are less than or equal to the cumulative number of partition bits accu_cut_num in all specified partition dimensions, then it is determined that there is no partition bit in the specified dimension;
[0031] The judgment method of step 4 is: the node information includes the tree height tree_height of the current node, and directly compares whether the tree_height in the child node is greater than or equal to the tree height threshold pre_tree_height_max.
[0032] Furthermore, the step 8 also includes the following steps: after the pre-tree is successfully built, each leaf node of the pre-created decision tree is traversed. If the number of rules rule_num contained in the leaf node is greater than the leaf node rule number threshold leaf_rules_max when the decision tree is created, the rules in the leaf node are sorted from small to large according to the number of rule replications, and the rules from leaf_rules_max+1 to rule_num-1 are marked as large rules.
[0033] Furthermore, after preprocessing, the user rule set is divided into two parts: a small number of large rules and a large number of small rules. A large number of small rules are used to create a decision tree, and the decision tree information is saved to the hardware and searched through the decision tree search module. A small number of large rules are searched in parallel with the help of hardware.
[0034] The present invention also provides a device for preprocessing a decision tree rule set. The device includes a processor, and the processor is used to execute the above method.
[0035] (3) Beneficial effects
[0036] The present invention proposes a method and apparatus for preprocessing a decision tree rule set. By comparing the time and memory usage of decision tree creation before and after rule set preprocessing, it is found that this solution has the following significant advantages:
[0037] 1) Significantly reduces the decision tree creation time.
[0038] 2) The FPGA on-chip RAM resources occupied by decision tree nodes are significantly reduced.
[0039] 3) Reduce the tree height, reduce the corresponding logical resources, and reduce processing delays. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] Figure 1 For example of rules;
[0041] Figure 2 For Big Rule 1: Example of a large-scale rule;
[0042] Figure 3 For Big Rule 2: Examples of rules that override each other;
[0043] Figure 4 The memory layout of the parent node and child nodes;
[0044] Figure 5 Preprocessing process for rule sets;
[0045] Figure 6 For the overall processing flow. DETAILED DESCRIPTION
[0046] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.
[0047] The present invention identifies a small number of large rules in a rule set by pre-building a decision tree. Usually, multiple thresholds are set before the decision tree is created: tree height upper limit tree_height_max, leaf node rule number upper limit leaf_rules_max, etc. In the pre-building stage, in order to ensure the success of pre-building and reduce the pre-building time and storage resource usage, new thresholds are set for pre-building: tree height upper limit
[0048] pre_tree_height_max (greater than tree_height_max) and the upper limit of leaf node rules pre_leaf_rules_max (greater than leaf_rules_max).
[0049] Before pre-building the tree, all the rules in the rule set are assigned to the root node. The process of decision tree pre-creation is similar to that of decision tree creation. Since the focus of this patent is the pre-processing of the rule set, the decision tree creation process will not be described in detail. The pre-creation of the decision tree is a recursive process. This patent describes the process of a single recursive process in detail to illustrate the principle of accurately marking large rules through decision tree pre-creation. Single recursive processing is the process of dividing the rules of the parent node into child nodes, and then processing each child node separately according to different thresholds.
[0050] First, we'll explain how to count rule replications: A counting variable is maintained for each rule. Initially, all rules in the rule set belong to the decision tree root node, and all counting variables are initialized to zero. During the decision tree construction process, rules are partitioned from the root node to lower-level nodes. A node is marked as a leaf node until the number of rules it contains falls below the pre_leaf_rules_max limit for leaf node rules. Each time a rule is partitioned from a parent node to a child node, the corresponding counting variable is incremented by one, indicating the number of rule replications.
[0051] Each node in the decision tree includes the following key information:
[0052] 1) Current node tree height tree_height: Indicates the number of tree levels from the root node to the current node. The root node has a tree height of 0, the root node's child node has a tree height of 1, the root node's grandchild node has a tree height of 2, and so on.
[0053] 2) Cut_dim: The cut dimension of the current node. This field is 5 bits long, with bits 0 through 4 representing the protocol number, destination port number, source port number, destination IP address, and source IP address. When a cut_dim bit is 1, the corresponding dimension is used as the cut dimension.
[0054] Method for obtaining partitioning dimensions: save the different numbers of rules contained in the current node in each dimension in the array diff_num[5] respectively, calculate the average value avg_num of the five members of diff_num[5], and use the dimension where diff_num[i] is greater than avg_num as the partitioning dimension, and select up to three dimensions.
[0055] 3) The number of bits used by the current node as the partitioning bits in the specified partitioning dimension cut_num[5]: represented by an array containing 5 elements. From cut_num[0] to cut_num[4], they represent the number of partitioning bits in the five dimensions of protocol number, destination port number, source port number, destination IP address, and source IP address.
[0056] To determine the number of partition bits, try different numbers of partition bits, divide the parent node rule into multiple child nodes, and calculate the standard deviation of the number of child node rules as a measure of the average degree of rule partitioning. The number of partition bits with the highest average degree of rule partitioning is used as the final number of partition bits.
[0057] 4) The sum of the number of partition bits of each node on the path from the root node to the parent node of the current node is called the cumulative number of partition bits of the current node, represented by accu_cut_num[5]. From accu_cut_num[0] to accu_cut_num[4], they represent the cumulative number of partition bits of the five dimensions: protocol number, destination port number, source port number, destination IP address, and source IP address.
[0058] 5) Pointer to child node memory chi ldren_ptr: Multiple child nodes of the parent node are stored in a continuous memory block, and chi ldren_ptr points to the first address of the memory block.
[0059] 6) Number of child nodes chi ld_num: After determining the partition dimension cut_dim and the number of partition bits cut_num[5], the number of child nodes is also determined.
[0060] The method for calculating the number of child nodes is to calculate the sum of the number of partition bits corresponding to the partition dimension, and the number of child nodes, chi ld_num, is 2 to the power of cut_num_sum. The memory layout of the parent node and the child node is as follows: Figure 4 As shown in the figure, the index of child nodes starts from 0, and the index value of child node n in the figure is child_num–1.
[0061] Method for dividing parent node rules into child nodes:
[0062] 1) When the parent node uses only one dimension i as the partitioning dimension, the number of partitioning bits is cut_num[i], and the cumulative number of partitioning bits is accu_cut_num[i]. When a rule r of the parent node needs to be partitioned into a child node, the value of the rule in the dimension i from the accu_cut_num[i]th bit to the accu_cut_num[i]+cut_num[i]-1th bit is obtained (note: the highest bit is recorded as the 0th bit, the second highest bit is recorded as the 1st bit, and so on). This value is the child node index value idx, indicating that the rule r will be partitioned into the idxth child node of the parent node. When the mask value mask of dimension i is less than accu_cut_num[i]+
[0063] When cut_num[i], that is, the mask bit to the accu_cut_num[i]+cut_num[i]th bit of dimension i are wildcards, so the rule r will be divided into multiple child nodes.
[0064] 2) When the parent node uses multiple dimensions as partitioning dimensions, the processing method is similar to when only one dimension is used. For the sake of convenience, it is assumed here that two dimensions i and j are used as partitioning dimensions. Get the value IDXi of the partitioning dimension i from the accu_cut_num[i]th bit to the accu_cut_num[i]+cut_num[i]-1th bit; get the value IDXj of the partitioning dimension j from the accu_cut_num[j]th bit to the accu_cut_num[j]+cut_num[j]-1th bit. After shifting IDXj left by cut_num[i] bits, perform a bitwise OR operation with IDXi to obtain the index value idx, which is the index of the child node, indicating that the rule r will be divided into the idxth child node of the parent node. Similarly, due to the wildcard mask, the rule r may be divided into multiple child nodes. The case of using 3 dimensions as partitioning dimensions is similar to the case of using 2 dimensions, and will not be repeated here.
[0065] 3) For other rules in the parent node, the same processing method as rule r is adopted and they are divided into child nodes in turn.
[0066] The rule set preprocessing process is as follows Figure 5 This figure details the processing flow after the parent node rules are divided into child nodes.
[0067] 1) Step 1: Determine whether the number of rules in the current child node is greater than the threshold pre_leaf_rules_max. If it is less than or equal to the threshold, execute step 7 and subsequent steps; otherwise, execute step 2.
[0068] 2) Step 2: Determine whether the child node still has a valid partition dimension.
[0069] Judgment method: The maximum number of bits occupied by data in each dimension is fixed. For example, the source IP and destination IP each occupy 32 bits, the source port and destination port each occupy 16 bits, and the protocol number occupies 8 bits. When the cumulative number of partitioned bits accu_cut_num for each dimension of a child node equals the corresponding maximum number of bits, no valid partitioning dimension exists for that child node.
[0070] The reason why the effective division dimensions are exhausted and the rules are not divided is that there are too many overlapping rules. In this case, the current child node needs to be marked as a leaf node in advance.
[0071] If the current child node does not have a valid division dimension, execute step 7 and subsequent steps; otherwise, execute step 3.
[0072] 3) Step 3: Determine whether there are still partition bits in the specified dimension.
[0073] Judgment method: If in all specified partition dimensions, the masks of all rules are less than or equal to the cumulative number of partition bits accu_cut_num of the corresponding dimensions, it is determined that there is no partition bit in the specified dimension.
[0074] If there is no valid partition bit, it means that the current child node contains many large rules. In this case, the current child node needs to be marked as a leaf node in advance.
[0075] If the partition bit does not exist in the specified dimension, execute step 7 and subsequent steps; otherwise, execute step 4.
[0076] 4) step 4: Determine whether the height of the current child node exceeds the tree height threshold pre_tree_height_max.
[0077] Judgment method: The node information contains the current node's tree height tree_height. Simply compare the tree_height in the child nodes to see if it is greater than or equal to the tree height threshold pre_tree_height_max.
[0078] To ensure search efficiency, the height of the decision tree needs to be limited. When the tree height of the current child node is greater than or equal to the tree height threshold pre_tree_height_max, the current child node needs to be marked as a leaf node in advance.
[0079] If the tree height of the current child node is greater than or equal to the threshold, execute step 7 and subsequent steps; otherwise, execute step 5 and step 6.
[0080] 5) Step 5: The replication count for each rule of this child node is incremented by 1. Step 6: This child node serves as the new parent node and continues to partition rules level by level. After step 6, the child node serves as the new parent node. Repeat steps 1 through 4. This continues until all nodes have been partitioned and all child nodes can execute step 7 and subsequent steps. Step 7: This node becomes a leaf node. Step 8: The replication count for each rule of this node is incremented by 1. This indicates that the pre-tree has been successfully built.
[0081] 6) After the pre-tree is successfully built, traverse each leaf node of the pre-created decision tree. If the number of rules contained in the leaf node (rule_num) is greater than the leaf node rule number threshold (leaf_rules_max) when the decision tree is created, sort the rules in the leaf node from small to large according to the number of rule replications, and mark the rules from leaf_rules_max+1 to rule_num-1 as large rules.
[0082] 7) In a rule set with practical significance, the number of large rules is small, and this embodiment can accurately identify them, and a small number of large rules are searched in parallel by hardware.
[0083] 8) During rule set preprocessing, the major rules have been marked. The large number of unmarked rules are used as the rule set when creating the decision tree.
[0084] Rule set preprocessing only needs to be performed once, and the preprocessed rule set can be used for subsequent possible multiple decision tree creations.
[0085] A small number of large rules separated from the original rule set are searched in parallel by hardware; the remaining large number of small rules are searched through the decision tree. The overall processing flow is as follows Figure 6 :
[0086] After the user rule set is preprocessed by the rule set described in this patent, it is divided into two parts: a small number of large rules and a large number of small rules. A large number of small rules are used to create a decision tree, and the decision tree information is saved to the hardware (such as FPGA). A small number of large rules need to be searched in parallel with the hardware. When the hardware queries which rule a data packet matches, the search is performed separately through the decision tree search module and the parallel search module. If multiple rules from two modules match the data packet, it needs to go through the priority arbitration module, and the rule with the highest priority is output as the final matching rule.
[0087] By comparing the time and memory usage of decision tree creation before and after rule set preprocessing, we found that this solution has the following obvious advantages:
[0088] 1) Significantly reduces the time required to create a decision tree.
[0089] 2) The FPGA on-chip RAM resources occupied by decision tree nodes are significantly reduced.
[0090] 3) Reduce the tree height, reduce the corresponding logical resources, and reduce processing delays.
[0091] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.
Claims
1. A method for preprocessing decision tree rule sets, It is characterized in that The method comprises the following steps: Step 1: Determine whether the number of current child node rules is greater than the upper limit threshold of leaf node rules pre_leaf_rules_max. If it is less than or equal to the threshold, execute step 7 and subsequent steps; otherwise, execute step 2; Step 2: Determine whether the child node still has a valid division dimension; if the current child node does not have a valid division dimension, execute step 7 and subsequent steps; otherwise, execute step 3; Step 3: Determine whether there are still partition bits in the specified dimension; if there are no partition bits in the specified dimension, execute step 7 and subsequent steps; otherwise, execute step 4; Step 4: Determine whether the height of the current child node exceeds the upper threshold of the tree height pre_tree_height_max; if the tree height of the current child node is greater than or equal to the threshold, execute step7 and subsequent steps; otherwise, execute step5 and step6; Step 5: The number of copies of each rule of the child node is increased by 1. Step 6: The child node serves as the new parent node and continues to divide the rules level by level. After executing step 6, the child node serves as the new parent node and executes steps 1 to 4 again until all nodes are divided by rules, and each child node can execute step 7 and subsequent steps. Step 7: The child node is set as a leaf node. Step 8: The number of copies of each rule of the child node is increased by 1, indicating that the pre-tree is successfully built. in, Each node in the decision tree includes the following key information: 1) Current node tree height tree_height: indicates the number of layers from the root node to the current node tree; 2) The partition dimension of the current node cut_dim: a total of 5 bits, from bit 0 to bit 4, respectively, representing the five dimensions of protocol number, destination port number, source port number, destination IP address, and source IP address. When a bit of cut_dim is 1, it means that the corresponding dimension is used as the partition dimension; 3) The number of bits used by the current node as the partitioning bits in the specified partitioning dimension cut_num[5]: represented by an array containing 5 elements, from cut_num[0] to cut_num[4], which represent the number of partitioning bits in the five dimensions of protocol number, destination port number, source port number, destination IP address and source IP address respectively; 4) The sum of the number of partition bits of each node on the path from the root node to the parent node of the current node is called the cumulative number of partition bits of the current node, represented by accu_cut_num[5]. From accu_cut_num[0] to accu_cut_num[4], they represent the cumulative number of partition bits of the five dimensions of protocol number, destination port number, source port number, destination IP address and source IP address respectively; 5) Pointer children_ptr pointing to the child node memory: multiple child nodes of the parent node are stored in a continuous memory block, and children_ptr points to the first address of the memory block; 6) Number of child nodes chi ld_num: Calculate the sum of the number of partition bits corresponding to the partition dimension, cut_num_sum. The number of child nodes chi ld_num is: 2 to the power of cut_num_sum.
2. The method for preprocessing a decision tree rule set according to claim 1, It is characterized in that Before creating a decision tree, multiple thresholds are pre-set: the upper limit of tree height tree_height_max, the upper limit of the number of leaf node rules leaf_rules_max. In the preprocessing stage, new thresholds are set for the pre-built tree: the upper limit of tree height pre_tree_height_max and the upper limit of the number of leaf node rules pre_leaf_rules_max. The upper limit of tree height pre_tree_height_max is greater than tree_height_max, and the upper limit of leaf node rules pre_leaf_rules_max is greater than leaf_rules_max.
3. The method for preprocessing a decision tree rule set according to claim 1, It is characterized in that A counting variable is maintained for each rule. In the initial state, all rules in the rule set belong to the root node of the decision tree, and all counting variables are initialized to zero. In the process of creating the tree, the rules are divided layer by layer from the root node to the lower nodes until the number of rules contained in the lower node is less than the upper limit of the number of leaf node rules pre_leaf_rules_max, and the node is marked as a leaf node; each time a rule is divided from a parent node to a child node, the counting variable corresponding to the rule is incremented by one, that is, the number of rule replications is incremented by one.
4. The method for preprocessing a decision tree rule set according to claim 1, It is characterized in that The method for dividing parent node rules into child nodes includes: when the parent node only uses one dimension i as the division dimension, the number of division bits is cut_num[i], and the cumulative number of division bits is accu_cut_num[i]; when a rule r of the parent node needs to be divided into child nodes, the value of the rule from the accu_cut_num[i]th bit to the accu_cut_num[i]+cut_num[i]-1th bit in dimension i is obtained, and the value is the child node index value idx, indicating that the rule r will be divided into the idxth child node of the parent node. When the mask value mask of dimension i is less than accu_cut_num[i]+cut_num[i], that is, the mask bit to the accu_cut_num[i]+cut_num[i]th bit of dimension i is wildcarded, so the rule r will be divided into multiple child nodes.
5. The method for preprocessing a decision tree rule set according to claim 1, It is characterized in that The method for dividing a parent node rule into child nodes includes: when two dimensions i and j are used as division dimensions, obtaining the value IDXi of the division dimension i from the accu_cut_num[i]th bit to the accu_cut_num[i]+cut_num[i]-1th bit; obtaining the value IDXj of the division dimension j from the accu_cut_num[j]th bit to the accu_cut_num[j]+cut_num[j]-1th bit; shifting IDXj left by cut_num[i] bits, performing a bitwise OR operation with IDXi to obtain an index value idx, which is the index of the child node, indicating that the rule r will be divided into the idxth child node of the parent node; similarly, due to the wildcard mask, the rule r may be divided into multiple child nodes.
6. The method for preprocessing a decision tree rule set according to any one of claims 1 to 5, It is characterized in that The judgment method of step 2 is: the maximum number of bits occupied by the data of each dimension is fixed. When the cumulative number of division bits accu_cut_num of each dimension of the child node is equal to the corresponding maximum number of bits, there is no valid division dimension for the child node; The judgment method of step 3 is: if in all specified division dimensions, the masks of all rules are less than or equal to the cumulative number of division bits accu_cut_num of the corresponding dimensions, then it is determined that there is no division bit in the specified dimension; The judgment method of step 4 is: the node information includes the tree height tree_height of the current node, and directly compares whether the tree_height in the child node is greater than or equal to the tree height threshold pre_tree_height_max.
7. The method for preprocessing a decision tree rule set as claimed in claim 6, It is characterized in that The step 8 also includes the following steps: after the pre-tree is successfully built, traverse each leaf node of the pre-created decision tree. If the number of rules rule_num contained in the leaf node is greater than the leaf node rule number threshold leaf_rules_max when creating the decision tree, the rules in the leaf node are sorted from small to large according to the number of rule replications, and the rules from leaf_rules_max+1 to rule_num-1 are marked as large rules.
8. The method for preprocessing a decision tree rule set as claimed in claim 7, It is characterized in that After preprocessing, the user rule set is divided into two parts: a small number of large rules and a large number of small rules. A large number of small rules are used to create a decision tree, and the decision tree information is saved to the hardware and searched through the decision tree search module. A small number of large rules are searched in parallel with the help of hardware.
9. A device for preprocessing a decision tree rule set, It is characterized in that The apparatus comprises a processor configured to execute the method of any one of claims 1-8.
