Event matching method and system based on blue-green tree data structure

Abstract

This invention provides an event matching method and system based on a blue-green tree data structure, comprising: constructing a blue-green tree data structure, storing predicates in subscriptions using blue and green type nodes; searching for matching and non-matching predicates through the blue-green tree data structure; and completing event matching based on the matching and non-matching predicates using corresponding forward counting and reverse marking event matching algorithms. To further improve the algorithm's search performance, this invention proposes corresponding optimization methods from five aspects: bit set optimization, node scaling, load balancing, attribute filtering, and multi-way trees. This invention first classifies and stores predicates according to their distribution to ensure load balancing. The setting of a virtual and real attribute layer enables the matching algorithm to guarantee search performance under high-dimensional and high-load conditions, ultimately achieving a more efficient and stable event matching algorithm. By setting multiple split points on a single node, the stability of the blue-green tree structure is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, specifically to an event matching method and system based on a blue-green tree data structure. More particularly, it relates to an event matching algorithm and optimization method in a content-based publish-subscribe system. Background Technology

[0002] With the high-quality development of society and the arrival of the big data era, the scale of data is growing larger and larger. People are putting forward increasingly granular requirements for data distribution rules, while the need for low latency in data distribution remains unchanged. This poses a greater challenge to data distribution systems. In a content-based publish-subscribe system, the proxy server executes an event matching algorithm, stores the constraints or predicates proposed by subscribers, and for events published in real time by publishers, quickly and accurately finds subscribers interested in the event based on the stored predicates and certain matching logic. This minimizes the processing latency of events on the proxy server and improves the overall data distribution throughput of the system by enhancing the performance of event matching on a single machine.

[0003] However, many factors influence the performance of event matching algorithms, including the characteristics of the subscription dataset (number of subscriptions, subscription size, predicate width, attribute distribution of subscriptions, distribution of predicate values, frequency of subscription insertion and deletion, dimensionality of the content space, and size of the attribute value range), the characteristics of the event dataset (event size, attribute distribution of events, distribution of event values, and matching degree between events and subscriptions), as well as interference from other tasks in the system and platform hardware characteristics. Generally speaking, when designing new data structures and matching algorithms, various application scenarios should be considered as much as possible to ensure the algorithm's versatility and robustness.

[0004] Patent document CN103984760A discloses a data structure for a content publishing and subscription system and its corresponding hybrid event matching method. It includes a list of subscriptions with the same predicate and a predicate with the same predicate and a predicate with the same predicate, a hybrid subscription list, and a hybrid predicate table. Through three stages of adding subscriptions, deleting subscriptions, and event matching, the subscription information is organized into an index structure that considers the relationship between predicates and subscriptions. The subscription coverage relationship is separated from the matching network algorithm and introduced into the predicate index structure.

[0005] However, there is a market need for an event matching method and system based on the blue-green tree data structure that can solve the problem of efficient distribution of fine-grained data in content-based publish-subscribe systems. Summary of the Invention

[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide an event matching method and system based on a blue-green tree data structure.

[0007] An event matching method based on a blue-green tree data structure provided by the present invention includes:

[0008] Step S1: Construct a blue-green tree data structure, using blue and green nodes to store the predicates in the subscription;

[0009] Step S2: Find matching and non-matching predicates using the blue-green tree data structure;

[0010] Step S3: Based on the matching predicate and the non-matching predicate, complete the event matching using the corresponding forward counting event matching algorithm and reverse tagging event matching algorithm.

[0011] Preferably, the method further includes matching algorithm optimization steps: bit set optimization, dynamic scaling optimization based on node size, interval splitting point optimization based on load balancing of historical data, attribute filtering optimization, and multi-branch tree optimization.

[0012] Preferably, the blue-green tree data structure is divided into three categories based on whether the interval formed by the high and low values ​​of the predicates defined on the attribute crosses the median of the attribute value range: the first category is defined as both the high and low values ​​are less than or equal to the median; the second category is defined as both the high and low values ​​are greater than the median; and the third category is defined as the high and low values ​​cross the median.

[0013] The first and second categories are represented by blue nodes in a blue-green tree data structure, and the third category is represented by green nodes in a blue-green tree data structure. Predicates are stored using both blue and green nodes, and a blue-green tree data structure is constructed by recursively splitting nodes. The high and low values ​​of the third category are stored separately, with low values ​​stored in low-green nodes and high values ​​stored in high-green nodes.

[0014] The number of blue nodes in the h-level blue-green tree is 2. h -1, and by solving, the number of green nodes is (h-2)×2. h +2, the formula is as follows:

[0015] G(h)=G(h-1)+2×T(h-1)+2×2 h-2

[0016] T(h)=2×T(h-1)+2×2 h-2

[0017] Where h≥1, G(h) represents the number of all green nodes in the h-th layer blue-green tree, and T(h) represents the number of green nodes in the h-th layer. h-2 This represents the number of blue nodes in the (h-1)th layer.

[0018] Preferably, in the positive counting event matching algorithm, there is only one blue node that needs to be processed, and each predicate stored in the blue node needs to be compared and checked twice. One or more green nodes are either completely matched or need to be compared and checked once.

[0019] In the reverse-marking event matching algorithm, there are exactly h-1 blue nodes that are completely mismatched and need to be reverse-marked, and exactly one blue node that requires two comparisons to determine whether the predicate matches. There are one or more green nodes that are completely mismatched or require one comparison check, where h represents the number of layers in the blue-green tree data structure.

[0020] Preferably, the bit set optimization includes converting the counting arithmetic operation type of the forward counting matching algorithm into the logical operation type and optimizing the performance of the reverse marking matching algorithm through the bit set;

[0021] The step of converting the counting arithmetic operation type of the forward counting matching algorithm to the logical operation type includes:

[0022] Step S1.1: When inserting the predicate, use a bit set B for each attribute. null Record subscriptions for which no predicate is defined on the attribute, where the bit set length is the total number of subscriptions, each bit represents a subscription, and a mark of 1 indicates that the subscription does not define a predicate on the attribute;

[0023] Step S1.2: When performing event matching, first initialize a result bit set B, wherein the length of the result bit set is the total number of subscriptions, and initialize all flags to 1 to indicate that all are matching subscriptions;

[0024] Step S1.3: For each event attribute, first copy the B of the attribute. null The matching subscriptions found using a forward counting matching algorithm are marked in the copy bit set to obtain all matching subscriptions for the event attribute. A bitwise AND operation is then performed between the copy bit set and the result bit set B. For empty attributes with undefined event values, the B value of the empty attribute is directly set. null Perform a bitwise AND operation with B, and the bits in B that are 1 represent matching subscriptions.

[0025] The performance optimization steps for the reverse-label matching algorithm include:

[0026] Step S2.1: Set a threshold u, and determine whether the number of predicates stored in the node is less than or equal to u. If yes, store the subscription ID in a dynamic array; otherwise, store the subscription ID in a bit set with a length equal to the total number of subscriptions.

[0027] Step S2.2: When processing nodes that store completely mismatched predicates, i.e., nodes that do not match without comparison, if the storage method of the node is a bit set, then perform an OR operation between the bit set corresponding to the mismatched node and the result bit set.

[0028] Preferably, dynamic scaling optimization based on node size includes:

[0029] A preset threshold V is set. When the node size is less than or equal to the threshold V, recursive splitting will stop. When the node size is greater than V, dynamic splitting will be performed.

[0030] If, after deleting the predicate, the size of the parent node of the leaf node is less than or equal to the threshold V, then the child node corresponding to the parent node is deleted, and the parent node is used as the leaf node.

[0031] Preferably, the attribute filtering optimization includes constructing two attribute layers, virtual and real, for event matching, including the following sub-steps:

[0032] Step A: Treat each event attribute as a virtual attribute, retrieve the matching predicates on the virtual attribute layer, and obtain the candidate matching subscription set;

[0033] Step B: Treat each event attribute as a real attribute, obtain the matching relationship between the predicate on the real attribute and the event, and store the matching result of each real attribute in the bit set;

[0034] Step C: Traverse the candidate matching subscriptions obtained from each virtual attribute layer, and check whether the real attribute predicates of the candidate subscriptions are all matched based on the result bit set of the real attribute layer, thereby determining the final matching subscription.

[0035] Preferably, the multi-branch tree optimization includes setting multiple split points on a single node to increase the branches of each node, thereby making the tree wider and shorter, forming a multi-branch tree.

[0036] An event matching system based on a blue-green tree data structure, provided by the present invention, includes:

[0037] Module M1: Constructs a blue-green tree data structure, using blue and green nodes to store predicates in the subscription;

[0038] Module M2: Uses the blue-green tree data structure to find matching and non-matching predicates;

[0039] Module M3: Based on the matching predicate and the non-matching predicate, it completes event matching through the corresponding forward counting event matching algorithm and reverse tagging event matching algorithm.

[0040] Preferably, the blue-green tree data structure is divided into three categories based on whether the interval formed by the high and low values ​​of the predicates defined on the attribute crosses the median of the attribute value range: the first category is defined as both the high and low values ​​are less than or equal to the median; the second category is defined as both the high and low values ​​are greater than the median; and the third category is defined as the high and low values ​​cross the median.

[0041] The first and second categories are represented by blue nodes in a blue-green tree data structure, and the third category is represented by green nodes in a blue-green tree data structure. Predicates are stored using both blue and green nodes, and a blue-green tree data structure is constructed by recursively splitting nodes. The high and low values ​​of the third category are stored separately, with low values ​​stored in low-green nodes and high values ​​stored in high-green nodes.

[0042] The number of blue nodes in the h-level blue-green tree is 2. h -1, and by solving, the number of green nodes is (h-2)×2. h +2, the formula is as follows:

[0043] G(h)=G(h-1)+2×T(h-1)+2×2 h-2

[0044] T(h)=2×T(h-1)+2×2 h-2

[0045] Where h≥1, G(h) represents the number of all green nodes in the h-th layer blue-green tree, and T(h) represents the number of green nodes in the h-th layer. h-2 This represents the number of blue nodes in the (h-1)th layer.

[0046] Compared with the prior art, the present invention has the following beneficial effects:

[0047] 1. This invention first classifies and stores predicates according to their distribution, fully considering the problem of distribution skew and ensuring load balance.

[0048] 2. The present invention proposes the setting of virtual and real attribute layers, which enables the matching algorithm to ensure search performance under high-dimensional and high-load conditions, and ultimately achieves a more efficient and stable event matching algorithm.

[0049] 3. This invention improves the stability of the blue-green tree structure by setting multiple split points on a single node.

[0050] 4. This invention uses bit sets for logical operations to reduce redundant operations, thereby accelerating the matching process and making full use of the hardware features. Attached Figure Description

[0051] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0052] Figure 1 This is a schematic diagram of a blue-green tree data structure.

[0053] Figure 2 This is a schematic diagram of the forward matching algorithm for blue nodes.

[0054] Figure 3 This is a schematic diagram of the positive matching algorithm for low green nodes.

[0055] Figure 4 This is a schematic diagram of the positive matching algorithm for high-green nodes.

[0056] Figure 5 This is a schematic diagram of the reverse matching algorithm for blue nodes.

[0057] Figure 6 This is a schematic diagram of the reverse matching algorithm for low green nodes.

[0058] Figure 7 This is a schematic diagram of the reverse matching algorithm for high-green nodes.

[0059] Figure 8 This is a schematic diagram of the workflow of the present invention. Detailed Implementation

[0060] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the scope of protection of the present invention.

[0061] The following exist in the prior art:

[0062] Based on the number of attributes processed in a single event matching process, the data structure of event matching algorithms can be divided into two categories: single-dimensional processing and multi-dimensional processing. (1) Single-dimensional processing algorithms set up a separate data structure instance for each attribute. This single-dimensional processing can only retrieve predicates that match or do not match on each attribute. Further aggregation of the partial matching results obtained on each attribute is required to obtain the final matching result. (2) Multi-dimensional processing algorithms concatenate the classification structures of predicates on multiple attributes, and can obtain predicates that match or do not match on multiple attributes at once. Most data structures, including the blue-green tree of this invention, belong to the first type. Their differences mainly lie in the classification and storage strategies of predicates on the same attribute.

[0063] The REIN algorithm uses a single-value classification method to store predicates. First, it separates the high and low values ​​of an interval predicate into two unrelated attribute value spaces, and then divides the value spaces into multiple subspaces, each of which stores data using a dynamic array. Then, based on the two subspaces into which the high and low values ​​of the interval predicate fall, it stores the high value and subscription ID, and the low value and subscription ID, respectively, into the dynamic arrays corresponding to the two subspaces.

[0064] The TAMA algorithm uses a span classification method to store predicates. Similar to segment trees and binary trees, TAMA divides the value space of each attribute into layers, with each node corresponding to a subspace. Each node uses a dynamic array to store the subscription IDs of the interval predicates covering that subspace. The subspace corresponding to each layer of nodes represents a partition of the value space. The number of subspaces between layers grows exponentially with respect to base 2; the higher the layer, the more subspaces there are. This partitioning scheme ensures that the interval predicates are covered by as many subspaces as possible, requiring only a small number of nodes to store them.

[0065] PS-Tree and NeuvoMatch combine the first two classification methods, employing a dynamic value domain subspace partitioning approach. First, the set of predicates for a given attribute is deduplicated, and the values ​​of the retained predicates are used as the dividing points of the value domain space. Therefore, each predicate is covered by exactly multiple subspaces. By storing the subscription ID corresponding to the predicate in these subspaces, matching subscriptions can be directly found based on the event value.

[0066] For the second type of data structure that supports multidimensional processing, H-Tree uses a single-level overlapping subspace classification method, MO-Tree uses a multi-level overlapping subspace classification method, and GSEC uses a multi-attribute subspace combination classification method.

[0067] However, the existing technology has a relatively complex matching logic, which is difficult to implement, modify and apply in practice. It also has poor scalability and poor performance in terms of dynamism and load balancing.

[0068] Example 1

[0069] An event matching method based on a blue-green tree data structure is provided by the present invention, such as Figure 8 Shown, including:

[0070] Step S1: Construct a blue-green tree data structure, using blue and green nodes to store predicates in the subscription. The blue-green tree data structure uses a distribution-based classification method to store predicates. Based on whether the interval between the high and low values ​​of the predicate defined on the attribute crosses the median of the attribute value range, it is divided into three categories: Category 1 (both high and low values ​​are less than or equal to the median), Category 2 (both high and low values ​​are greater than the median), and Category 3 (both high and low values ​​cross the median). Category 1 and Category 2 are represented by blue nodes in the blue-green tree data structure, and Category 3 is represented by green nodes. Predicates are stored using both blue and green nodes, and the blue-green tree data structure is constructed by recursively splitting nodes. In Category 3, high and low values ​​are stored separately, with low values ​​stored in low-green nodes and high values ​​stored in high-green nodes.

[0071] In a blue-green tree, the subtree rooted at a green node is essentially a standard binary search tree, and the tree formed by all blue nodes is also a binary search tree. Therefore, the blue-green tree is essentially a pseudo-binary tree, formed by splicing multiple smaller binary search trees into a larger binary search tree. Thus, the number of blue nodes in a h-level blue-green tree is 2h-1, and the number of green nodes can be derived using the following recursive formula:

[0072] G(h)=G(h-1)+2×T(h-1)+2×2 h-2

[0073] T(h)=2×T(h-1)+2×2 h-2

[0074] Where h≥1, G(h) represents the number of all green nodes in the h-th layer blue-green tree, and T(h) represents the number of green nodes in the h-th layer. h-2 Let represent the number of blue nodes in the (h-1)th layer. Solving the above equation, we get:

[0075] G(h) = (h-2) × 2 h +2

[0076] T(h) = (h-1) × 2 h-1

[0077] Therefore, the number of green nodes in the blue-green tree at level h is (h-2)×2. h +2.

[0078] Specifically, the value range of the set attributes is normalized to [0, 1]. Each attribute maintains a separate instance of the blue-green tree. Taking the median value 0.5 of the value range as the splitting point, all interval predicates defined on each attribute are divided into three categories: both the high value and the low value are less than or equal to 0.5 (l≤h≤0.5), both the high value and the low value are greater than 0.5 (0.5<l≤h), and the high and low values span 0.5 (l≤0.5<h). The three categories of intervals do not overlap and cover all interval predicates defined in the attribute value range space, so any interval predicate belongs to only one of them. The first two categories limit the high and low values of the interval predicate within a range and can be represented by the same type of node. In the blue-green tree, this type of node is called a "blue node". The third category indicates that the high and low values of the predicate belong to two different ranges. In the blue-green tree, this type of node is called a "green node". Since the matching semantics of the event value for the high and low values of the predicate are opposite, in the blue-green tree, the third category of predicates stores its high value and low value separately. The low value is stored in the low green node, and the high value is stored in the high green node. Finally, the blue-green partitioning scheme is carried out recursively. For example, taking 0.25 and 0.75 as the splitting points, the two sub-intervals of 0 to 0.5 and 0.5 to 1 of the blue node are classified into three categories respectively, and only a binary classification is required for the green node. In this way, it can be continuously and recursively split until the height / layer of the tree reaches the preset threshold.

[0079] Furthermore, the blue-green tree data structure in the present invention is specifically described as follows in combination with the accompanying drawings:

[0080] As Figure 1 shown, the accompanying drawing shows a three-layer blue-green tree data structure. For the convenience of description, the node numbers are given by black balls in each node. Among them, node 2 is the left blue child node of node 1, and node 13 is the right blue child node of node 1. The "L"-shaped green node represents the low green node, and the "H"-shaped green node represents the high green node, which store the low value and the high value of the predicate respectively. Nodes 7 and 10 are the low green child node and the high green child node of node 1 respectively. Nodes 8 and 9 are the left low green child node and the right low green child node of node 7 respectively. Nodes 11 and 12 are the left high green child node and the right high green child node of node 10 respectively.

[0081] The root node of the blue-green tree is a blue node, which contains the entire value range interval of the attribute and has no filtering effect. No data is stored in the first layer. The remaining blue nodes store the subscription number and the high and low values of the predicate. The low green node stores the subscription number and the low value of the predicate, and the high green node stores the subscription number and the high value of the predicate. For example, given an interval predicate [0.3, 0.4], it is classified and stored in blue nodes 2 and 6; the interval predicate [0.3, 0.8] is split into two parts: the low value 0.3 and the high value 0.8, which are stored in nodes 7, 9 and nodes 10, 12 respectively; the interval predicate [0.6, 0.9] is stored in nodes 13, 15 and 16. The above is the three-layer blue-green tree data structure and its storage logic for the predicate. The leaf nodes of the third layer can continue to be recursively subdivided.

[0082] Step S2: Find matching and non-matching predicates using the blue-green tree data structure.

[0083] Step S3: Based on the matching predicate and the non-matching predicate, complete the event matching using the corresponding forward counting event matching algorithm and reverse tagging event matching algorithm.

[0084] In the forward counting event matching algorithm, there is one and only one blue node to process, and each predicate stored in the blue node requires two comparison checks. One or more green nodes are complete matches or require one comparison check. That is, the blue node can be further filtered out using distribution classification logic to remove some mismatched predicates. The two child nodes of the green node may be either one complete match and the other requiring recursive processing, or one complete mismatch and the other requiring recursive processing. Figure 1 Taking the data structure shown as an example, given an event value of 0.7, by searching... Figure 1 The system may contain nodes that match the predicates; nodes 11, 12, 14, and 15 can be found. Node 12 stores all matching subscriptions. The predicates in node 11 require comparing whether the higher value is greater than or equal to 0.7 to determine if a match exists. The predicates in node 15 require comparing whether the lower value is less than or equal to 0.7. Node 14 requires comparing both the higher and lower values ​​of the predicates to determine if a match exists.

[0085] For a blue node, if it's a leaf node, iterate through the predicates stored in the node, performing two comparisons for each predicate to determine if a match exists. If it's not a leaf node, the event value must fall within the scope of one of its blue or green child nodes, and then these two child nodes are processed recursively. The pseudocode for forward matching of blue nodes is as follows: Figure 2 As shown.

[0086] For green nodes, since the matching semantics of low-green and high-green are opposite, they need to be treated differently. If it is a green leaf node, then each predicate stored in it is traversed, and a single comparison is used to determine whether the predicates match. The pseudocode for the forward matching of low-green and high-green nodes is as follows: Figure 3 and Figure 4 As shown.

[0087] (1) If it is a low green branch node, then either the left low green child node needs to be processed recursively and the right low green child node is completely mismatched, or the left low green child node is completely matched and the right low green child node needs to be processed recursively.

[0088] (2) If it is a high green branch node, then either the left high green child node needs to be processed recursively and the right high green child node is completely matched, or the left high green child node is completely unmatched and the right high green child node needs to be processed recursively.

[0089] In the reverse-marking event matching algorithm, only h-1 blue nodes need to be processed as completely mismatched nodes requiring reverse marking, and only one node requires two comparisons to determine if the predicate matches. Green nodes have one or more completely mismatched nodes or require one comparison check, where h represents the level of the blue-green tree data structure. That is, branch blue nodes can continue searching for mismatched predicates in their child nodes based on distribution classification logic; one of their two blue child nodes must be a completely mismatched node, and the other needs recursive processing. Similarly, for branch green nodes, only one of their two child nodes needs recursive processing, just like in forward matching. For branch green nodes, one of their two child nodes also needs recursive processing, just like in forward matching; the other is either a completely matching or completely mismatched child node, and if it's a mismatch, it needs to be directly traversed and marked. The comparison check includes three types: zero-time, one-time, and two-time comparison checks on the predicate. Zero-time check means the predicate's matching can already be determined through the blue-green number filtering logic. One-time comparison check means the matching of either the high or low value of the predicate needs to be determined through one comparison before the final predicate match can be confirmed. Two-time comparison check means neither the high nor low value of the predicate can determine whether it matches the event, requiring two comparisons to confirm. Figure 1 Taking the data structure shown as an example, given an event value of 0.7, by searching... Figure 1 The system may store nodes with mismatched predicates; nodes 2, 11, 14, 15, and 17 can be found. Nodes 2 and 17 store mismatched subscriptions. Nodes 11 and 15 require comparing the high value of the predicate (less than 0.7) and the low value of the predicate (greater than 0.7) to determine if the predicates match. Blue node 14 requires two comparisons for each stored predicate to determine a match. For blue nodes, if they are leaf nodes, the predicate match is determined by two comparisons; if they are branch nodes, the event value must fall within the scope of one of their blue and green child nodes, requiring recursive processing of one blue and one green child node, and marking the other blue child node. The pseudocode for reverse matching of blue nodes is as follows: Figure 5 As shown.

[0090] For green nodes, if they are green leaf nodes, the stored mismatched predicates can be found through a single comparison. For branch nodes, two cases are discussed. One of the child nodes of a low-green branch node must be processed recursively; the other either doesn't need processing or is a completely mismatched node that needs to be traversed and reverse-marked. The reverse matching of high-green branch nodes can be derived similarly. The pseudocode for reverse matching of low-green and high-green nodes is as follows: Figure 6 and Figure 7 As shown.

[0091] The event matching method based on the blue-green tree data structure provided by this invention also includes matching algorithm optimization steps: bit set optimization, dynamic scaling optimization based on node size, interval splitting point optimization based on load balancing of historical data, attribute filtering optimization, and multi-branch tree optimization.

[0092] Bit set optimization includes converting the counting arithmetic operations of the forward counting matching algorithm into logical operations and optimizing the performance of the reverse marking matching algorithm using bit sets. The step of converting the counting arithmetic operations of the forward counting matching algorithm into logical operations includes:

[0093] Step 1: When inserting the predicate, use a bit set B for each attribute. null Record subscriptions for which no predicate is defined on the attribute, where the bit set length is the total number of subscriptions, each bit represents a subscription, and a mark of 1 indicates that the subscription has no predicate defined on the attribute.

[0094] Step 2: When performing event matching, first initialize a result bit set B, where the length of the result bit set is the total number of subscriptions, and initialize all flags to 1 to indicate that all are matching subscriptions.

[0095] Step 3: For each event attribute, first copy the B of the attribute. null The matching subscriptions found using a forward counting matching algorithm are marked in the copy bit set to obtain all matching subscriptions for the event attribute. A bitwise AND operation is then performed between the copy bit set and the result bit set b. For empty attributes with undefined event values, the bit set b of the empty attribute is directly set to b. null Perform a bitwise AND operation with B, and the bits in b that are 1 represent matching subscriptions.

[0096] The above describes optimizations to the forward algorithm. For the bBG-Tree reverse-marking matching algorithm based on blue-green trees, when a node stores a large number of predicates, it requires traversing and marking them one by one. Performance optimization using bit sets includes: setting a threshold u; determining if the number of predicates stored in a node is less than or equal to u; if so, storing the subscription IDs in a dynamic array; otherwise, storing the subscription IDs in a bit set of length equal to the total number of subscriptions. When handling nodes storing completely mismatched predicates (nodes where mismatches are known without comparison), if the node's storage method is a bit set, performing an OR operation between the bit set corresponding to the mismatched node and the resulting bit set. For the optimized forward counting matching, i.e., forward marking matching, this method can also be used to optimize the process when traversing and marking a green node storing matching subscriptions. The OR operation marks all subscriptions in the node onto a copied bit set at once.

[0097] If the tree height is fixed, a blue-green tree is a "full tree." Some branches may store very few predicates, resulting in many empty nodes; while other branches may store too many predicates, leaving many predicates to check even after filtering to the leaf nodes, leading to large leaf nodes and high load. To address this issue, a static layering design can be changed to a dynamic layering design, optimizing based on dynamic scaling of node size.

[0098] The dynamic scaling optimization based on node size includes: a preset threshold V; when the node size is less than or equal to the threshold V, recursive splitting stops; when the node size is greater than V, dynamic splitting is performed. After deleting a predicate, if the parent node size of a leaf node is less than or equal to the threshold V, then the child nodes corresponding to that parent node are deleted, and the parent node becomes a leaf node. This optimization effectively avoids the problem of nodes being too small or too large, reducing space usage and improving filtering capabilities.

[0099] When the distribution of predicate values ​​is skewed, the load may be concentrated in the green child nodes of a blue node, in which case the subtree rooted at that blue node degenerates into two binary trees; alternatively, the load may be concentrated in the blue child nodes of a blue node, leaving the subtree rooted at the green child nodes of that blue node almost empty, thus reducing the filtering effect of that blue node. To ensure load balance as much as possible, the strategy of using the median of the value range as the dividing point for interval partitioning can be changed. Instead, the interval partitioning point for achieving load balance among child nodes can be determined based on historical load data analysis. For example, for the root node, its corresponding value range is 0 to 1. If the number of predicates in the interval between 0 and 0.3 accounts for 33%, the number of predicates in the interval between 0.3 and 1.0 accounts for 33%, and the number of predicates in the interval crossing 0.3 accounts for 34%, then the attribute value range space should be partitioned based on 0.3 instead of 0.5. Green nodes can also have their partitioning points dynamically determined for interval bisection. After optimization, the filtering effect of the nodes becomes more stable.

[0100] Attribute filtering optimization involves constructing both virtual and real attribute layers for event matching. This optimization improves matching efficiency in high-dimensional spaces and under high load conditions. Predicates with good filtering effects are stored in the virtual attribute layer to fully utilize their filtering capabilities and reduce the number of candidate predicates. Then, the matching status of other predicates with weaker filtering capabilities is checked to determine the final matching result. This includes the following sub-steps:

[0101] Step A: Treat each event attribute as a virtual attribute, retrieve the matching predicates on the virtual attribute layer, and obtain the candidate matching subscription set.

[0102] Step B: Treat each event attribute as a real attribute to obtain the matching relationship between the predicate on the real attribute and the event, and store the matching result of each real attribute in the bit set.

[0103] Step C: Traverse the candidate matching subscriptions obtained from each virtual attribute layer, and check whether the real attribute predicates of the candidate subscriptions are all matched based on the result bit set of the real attribute layer, thereby determining the final matching subscription.

[0104] There are two ways to measure the filtering effect: the attributes defined by the predicate with the sparsest distribution or the smallest interval width in the subscription are used as virtual attributes, and these predicates are stored in the virtual attribute layer. Other attributes are real attributes, and the predicates defined on the real attributes are stored in the real attribute layer. To save space, instead of cascading the virtual and real attribute layers like OpIndex, virtual and real attribute predicates can be stored separately in the nodes of the same tree instance, similar to PS-Tree. This two-layer structure, separating predicates with special characteristics from a single storage structure, is why this optimization is named virtual-real attribute filtering. In this way, many subscriptions with empty event attributes as virtual attributes are mismatched, and this workload is filtered out, saving computational resources. This method has a certain degree of versatility and can be applied to existing matching algorithms, enabling them to support high-performance lookups in high-dimensional spaces and under high loads, just like OpIndex and PS-Tree.

[0105] Multi-way tree optimization involves setting multiple split points on a single node to increase the number of branches at each node, thus widening and shortening the tree to form a multi-way tree. This optimization avoids the imbalance caused by branches that are too long or too short. Specifically, for simplicity, equal-load split point optimization is not used here; the intervals are still statically divided using an equal-length method. Assuming each node has k-1 split points, its responsible value range is divided into k disjoint sub-intervals. Each blue node's sub-interval is numbered starting from 0, and its blue child nodes, low-green child nodes, and high-green child nodes are also numbered starting from 0. Let l be the space number where the low predicate value falls, and h be the space number where the high predicate value falls.

[0106] In the original classification strategy, k=2, the 0th sub-interval of the root node is [0,0.5], and the 1st sub-interval is (0.5,1.0). If l and h are both 0 or both are 1, a blue child node is inserted; otherwise, a low-green child node and a high-green child node are inserted. For green nodes, simply bisect the range of values ​​they are responsible for.

[0107] Extending to the case where k=3, the 0th, 1st, and 2nd sub-intervals of the root node are [0,0.33], (0.33,0.66], and (0.66,1.0) respectively. If l and h are both 0, 1, or 2, then blue child nodes 0, 1, or 2 are inserted respectively; if l=0 and h=1, then low-green child node 0 and high-green child node 0 are inserted; if l=0 and h=2, then low-green child node 1 and high-green child node 1 are inserted; if l=1 and h=2, then low-green child node 2 and high-green child node 2 are inserted. Therefore, among the blue nodes, there are 1+2=3 types each for low-green and high-green child nodes. For green nodes, simply divide the range of values ​​they are responsible for into equal sub-intervals 0, 1, and 2; the predicate value will only belong to one of these sub-intervals.

[0108] Extending to k-1 split points, i.e., the case of a k-ary tree: (1) If l = h, then insert the blue child node l; (2) If l ≠ h, pair different subintervals together, and... In this case, the node numbers for inserting the low-green child node and the high-green child node are: For green nodes, simply divide the interval they are responsible for into k equal sub-intervals, and recursively store the subscription ID into the corresponding green child node according to the sub-interval to which the predicate value belongs.

[0109] Example 2

[0110] The present invention also provides an event matching system based on a blue-green tree data structure. Those skilled in the art can implement the event matching system based on a blue-green tree data structure by executing the steps of the event matching method based on a blue-green tree data structure. That is, the event matching method based on a blue-green tree data structure can be understood as a preferred embodiment of the event matching system based on a blue-green tree data structure.

[0111] An event matching system based on a blue-green tree data structure, provided by the present invention, includes:

[0112] Module M1: Constructs a blue-green tree data structure, using blue and green nodes to store predicates in the subscription. The blue-green tree data structure is divided into three categories based on whether the interval formed by the high and low values ​​of the predicates defined on the attribute crosses the median of the attribute value range: Category 1: both high and low values ​​are less than or equal to the median; Category 2: both high and low values ​​are greater than the median; Category 3: both high and low values ​​cross the median. Category 1 and Category 2 are represented by blue nodes in the blue-green tree data structure, and Category 3 is represented by green nodes. Predicates are stored using both blue and green nodes, and the blue-green tree data structure is constructed through recursive node splitting. In Category 3, high and low values ​​are stored separately: low values ​​are stored in low-green nodes, and high values ​​are stored in high-green nodes. The number of blue nodes in the h-level blue-green tree is 2. h-1, and by solving, the number of green nodes is (h-2)×2. h +2, the formula is as follows:

[0113] G(h)=G(h-1)+2×T(h-1)+2×2 h-2

[0114] T(h)=2×T(h-1)+2×2 h-2

[0115] Where h≥1, G(h) represents the number of all green nodes in the h-th layer blue-green tree, and T(h) represents the number of green nodes in the h-th layer. h-2 This represents the number of blue nodes in the (h-1)th layer.

[0116] Module M2: Uses the blue-green tree data structure to find matching and non-matching predicates.

[0117] Module M3: Based on the matching predicate and the non-matching predicate, it completes event matching through the corresponding forward counting event matching algorithm and reverse tagging event matching algorithm.

[0118] In the forward counting event matching algorithm, there is one and only one blue node that needs to be processed, and each predicate stored in the blue node needs to be compared and checked twice. One or more green nodes are either complete matches or need to be compared and checked once.

[0119] In the reverse-marking event matching algorithm, there are exactly h-1 blue nodes that are completely mismatched and need to be reverse-marked, and exactly one that requires two comparisons to determine whether the predicate matches. There are one or more green nodes that are completely mismatched or require one comparison check, where h represents the level of the blue-green tree data structure. The comparison check includes three types: zero-time check, one-time check, and two-time check. Zero-time check means that the matching of the predicate can be obtained through the filtering logic of the blue-green tree. One-time check means that the matching of either the high or low value of the predicate needs to be determined by one comparison before the final predicate can be determined. Two-time check means that neither the high nor low value of the predicate can determine whether it matches the event, and two comparisons are required to determine the match.

[0120] The event matching system based on the blue-green tree data structure of this invention also includes a matching algorithm optimization module: bit set optimization, dynamic scaling optimization based on node size, interval splitting point optimization based on load balancing of historical data, attribute filtering optimization, and multi-branch tree optimization.

[0121] Bitset optimization includes converting the counting arithmetic operations of the forward counting matching algorithm into logical operations and optimizing the performance of the reverse tagging matching algorithm using bitsets. The conversion includes the following: when inserting a predicate, using a bitset B for each attribute. null Subscriptions for which no predicate is defined on the attribute are recorded, where the length of the bit set is the total number of subscriptions, each bit represents a subscription, and a flag of 1 indicates that the subscription does not define a predicate on the attribute. During event matching, a result bit set B is first initialized, where the length of the result bit set is the total number of subscriptions, and initializing all flags to 1 indicates that all are matching subscriptions. For each event attribute, a copy of attribute B is first made. null The matching subscriptions found using a forward counting matching algorithm are marked in the copy bit set to obtain all matching subscriptions for the event attribute. A bitwise AND operation is then performed between the copy bit set and the result bit set B. For empty attributes with undefined event values, the B value of the empty attribute is directly set. null A bitwise AND operation is performed with B, and the bits in B that are 1 represent matching subscriptions. Performance optimizations include setting a threshold u and determining whether the number of predicates stored in a node is less than or equal to u. If so, the subscription ID is stored in a dynamic array; otherwise, a bit set of length equal to the total number of subscriptions is used to store the subscription ID. When handling nodes that store no matching predicates (nodes whose mismatches can be identified without comparison), if the node is stored as a bit set, a bitwise OR operation is performed between the bit set corresponding to the mismatched node and the resulting bit set.

[0122] Dynamic scaling optimization based on node size includes: a preset threshold V; when the node size is less than or equal to the threshold V, recursive splitting stops; when the node size is greater than V, dynamic splitting is performed. After deleting a predicate, if the size of the parent node of a leaf node is less than or equal to the threshold V, then the child nodes corresponding to the parent node are deleted, and the parent node is treated as a leaf node.

[0123] Attribute filtering optimization involves constructing two attribute layers—virtual and real—for event matching, including the following sub-modules: Module A: Treats each event attribute as a virtual attribute, retrieves matching predicates on the virtual attribute layer, and obtains a candidate matching subscription set. Module B: Treats each event attribute as a real attribute, obtains the matching relationship between the predicates on the real attribute and the event, and stores the matching results of each real attribute in a bit set. Module C: Iterates through the candidate matching subscriptions obtained from each virtual attribute layer, and checks whether the real attribute predicates of the candidate subscriptions are all matched based on the bit set of the results from the real attribute layer, thereby determining the final matching subscription.

[0124] Multi-branch tree optimization involves increasing the number of branches at each node by setting multiple split points on a single node, thereby widening and shortening the tree to form a multi-branch tree.

[0125] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0126] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An event matching method based on a blue-green tree data structure, characterized in that, include: Step S1: Construct a blue-green tree data structure, using blue and green nodes to store the predicates in the subscription; Step S2: Find matching and non-matching predicates using the blue-green tree data structure; Step S3: Based on the matching predicate and the non-matching predicate, complete the event matching using the corresponding forward counting event matching algorithm and reverse tagging event matching algorithm; The blue-green tree data structure is divided into three categories based on whether the interval formed by the high and low values ​​of the predicates defined on the attribute crosses the median of the attribute value range: the first category is when both the high and low values ​​are less than or equal to the median; the second category is when both the high and low values ​​are greater than the median; and the third category is when the high and low values ​​cross the median. The first category and the second category are represented by blue nodes in a blue-green tree data structure, and the third category is represented by green nodes in a blue-green tree data structure; The predicates are stored using two types of nodes: blue nodes and green nodes. A blue-green tree data structure is constructed by recursively splitting nodes. The high and low values ​​of the third category are stored separately, with low values ​​stored in low-green nodes and high values ​​stored in high-green nodes. The number of blue nodes in a layered blue-green tree is The number of green nodes was obtained by solving the problem. , the formula is as follows: in , express The number of all green nodes in the blue-green tree. Indicates the first The number of green nodes in the layer Indicates the first The number of blue nodes in the layer; In the positive counting event matching algorithm, there is only one blue node to be processed, and each predicate stored in the blue node needs to be compared and checked twice. One or more green nodes are complete matches or need to be compared and checked once. In the reverse-marking event matching algorithm, there are exactly h-1 blue nodes that are completely mismatched and need to be reverse-marked, and exactly one blue node that requires two comparisons to determine whether the predicate matches. There are one or more green nodes that are completely mismatched or require one comparison check, where h represents the number of layers in the blue-green tree data structure.

2. The event matching method based on the blue-green tree data structure according to claim 1, characterized in that, It also includes matching algorithm optimization steps: bit set optimization, dynamic scaling optimization based on node size, interval splitting point optimization based on historical data load balancing, attribute filtering optimization, and multi-way tree optimization.

3. The event matching method based on the blue-green tree data structure according to claim 2, characterized in that, The bit set optimization includes converting the counting arithmetic operation type of the forward counting matching algorithm into the logical operation type and optimizing the performance of the reverse marking matching algorithm through the bit set; The step of converting the counting arithmetic operation type of the forward counting matching algorithm to the logical operation type includes: Step S1.1: When inserting the predicate, use a bit set for each attribute. Record subscriptions for which no predicate is defined on the attribute, wherein the length of the bit set is the total number of subscriptions, each bit represents a subscription, and a mark of 1 indicates that the subscription does not define a predicate on the attribute; Step S1.2: When performing event matching, first initialize a result bit set. The length of the result bit set is the total number of subscriptions, and the initial full flag is set to 1 to indicate that all subscriptions are matching. Step S1.3: For each event attribute, first make a copy of the attribute. The matching subscriptions found using a forward counting matching algorithm are marked in the copy bit set, resulting in all matching subscriptions for that event attribute. The copy bit set is then compared with the result bit set. Execution and operation; for empty properties with undefined values ​​for events, directly apply the values ​​to the empty property. and Execution and computation, ultimately The bit that is 1 indicates a matching subscription; The performance optimization steps for the reverse-label matching algorithm include: Step S2.1: Set the threshold Determine whether the number of predicates stored in a node is less than or equal to If yes, then use a dynamic array to store the subscription ID; otherwise, use a bit set of length equal to the total number of subscriptions to store the subscription ID. Step S2.2: When processing nodes that store completely mismatched predicates, i.e., nodes that do not match can be known without comparison, if the storage method of the node is a bit set, then perform an OR operation between the bit set corresponding to the mismatched node and the result bit set.

4. The event matching method based on the blue-green tree data structure according to claim 2, characterized in that, Dynamic scaling optimization based on node size includes: A preset threshold V is set. When the node size is less than or equal to the threshold V, recursive splitting will stop. When the node size is greater than V, dynamic splitting will be performed. If, after deleting the predicate, the size of the parent node of the leaf node is less than or equal to the threshold V, then the child node corresponding to the parent node is deleted, and the parent node is used as the leaf node.

5. The event matching method based on the blue-green tree data structure according to claim 2, characterized in that, The attribute filtering optimization includes constructing two attribute layers, virtual and real, for event matching, including the following sub-steps: Step A: Treat each event attribute as a virtual attribute, retrieve the matching predicates on the virtual attribute layer, and obtain the candidate matching subscription set; Step B: Treat each event attribute as a real attribute, obtain the matching relationship between the predicate on the real attribute and the event, and store the matching result of each real attribute in the bit set; Step C: Traverse the candidate matching subscriptions obtained from each virtual attribute layer, and check whether the real attribute predicates of the candidate matching subscriptions are all matched based on the result bit set of the real attribute layer, thereby determining the final matching subscription.

6. The event matching method based on the blue-green tree data structure according to claim 2, characterized in that, The multi-branch tree optimization involves setting multiple split points on a single node to increase the branches of each node, thereby widening and shortening the tree to form a multi-branch tree.

7. An event matching system based on a blue-green tree data structure, characterized in that, include: Module M1: Constructs a blue-green tree data structure, using blue and green nodes to store predicates in the subscription; Module M2: Uses the blue-green tree data structure to find matching and non-matching predicates; Module M3: Based on the matching predicate and the non-matching predicate, it completes event matching through the corresponding positive counting type event matching algorithm and the reverse marking type event matching algorithm; The blue-green tree data structure is divided into three categories based on whether the interval formed by the high and low values ​​of the predicates defined on the attribute crosses the median of the attribute value range: the first category is when both the high and low values ​​are less than or equal to the median; the second category is when both the high and low values ​​are greater than the median; and the third category is when the high and low values ​​cross the median. The first category and the second category are represented by blue nodes in a blue-green tree data structure, and the third category is represented by green nodes in a blue-green tree data structure; The predicates are stored using two types of nodes: blue nodes and green nodes. A blue-green tree data structure is constructed by recursively splitting nodes. The high and low values ​​of the third category are stored separately, with low values ​​stored in low-green nodes and high values ​​stored in high-green nodes. The number of blue nodes in a layered blue-green tree is The number of green nodes was obtained by solving the problem. , the formula is as follows: in , express The number of all green nodes in the blue-green tree. Indicates the first The number of green nodes in the layer Indicates the first The number of blue nodes in the layer; In the positive counting event matching algorithm, there is only one blue node to be processed, and each predicate stored in the blue node needs to be compared and checked twice. One or more green nodes are complete matches or need to be compared and checked once. In the reverse-marking event matching algorithm, there are exactly h-1 blue nodes that are completely mismatched and need to be reverse-marked, and exactly one blue node that requires two comparisons to determine whether the predicate matches. There are one or more green nodes that are completely mismatched or require one comparison check, where h represents the number of layers in the blue-green tree data structure.

8. The event matching system based on the blue-green tree data structure according to claim 7, characterized in that, It also includes a matching algorithm optimization module: bit set optimization, dynamic scaling optimization based on node size, interval splitting point optimization based on historical data load balancing, attribute filtering optimization, and multi-way tree optimization.

9. The event matching system based on the blue-green tree data structure according to claim 8, characterized in that, The bit set optimization includes converting the counting arithmetic operation type of the forward counting matching algorithm into the logical operation type and optimizing the performance of the reverse marking matching algorithm through the bit set; The module that converts the counting arithmetic operation type of the forward counting matching algorithm into the logical operation type includes: Module M1.1: When inserting predicates, use a bit set for each attribute. Record subscriptions for which no predicate is defined on the attribute, wherein the length of the bit set is the total number of subscriptions, each bit represents a subscription, and a mark of 1 indicates that the subscription does not define a predicate on the attribute; Module M1.2: When performing event matching, first initialize a result bit set. The length of the result bit set is the total number of subscriptions, and the initial full flag is set to 1 to indicate that all subscriptions are matching. Module M1.3: For each event attribute, first make a copy of the attribute. The matching subscriptions found using a forward counting matching algorithm are marked in the copy bit set, resulting in all matching subscriptions for that event attribute. The copy bit set is then compared with the result bit set. Execution and operation; for empty properties with undefined values ​​for events, directly apply the values ​​to the empty property. and Execution and computation, ultimately The bit that is 1 indicates a matching subscription; The performance optimization module for the reverse tag-type matching algorithm includes: Module M2.1: Setting Thresholds Determine whether the number of predicates stored in a node is less than or equal to If yes, then use a dynamic array to store the subscription ID; otherwise, use a bit set of length equal to the total number of subscriptions to store the subscription ID. Module M2.2: When processing nodes that store completely mismatched predicates, i.e., nodes that do not match can be known without comparison, if the storage method of the node is a bit set, then perform an OR operation between the bit set corresponding to the mismatched node and the result bit set.

10. The event matching system based on the blue-green tree data structure according to claim 8, characterized in that, Dynamic scaling optimization based on node size includes: A preset threshold V is set. When the node size is less than or equal to the threshold V, recursive splitting will stop. When the node size is greater than V, dynamic splitting will be performed. If, after deleting the predicate, the size of the parent node of the leaf node is less than or equal to the threshold V, then the child node corresponding to the parent node is deleted, and the parent node is used as the leaf node.

Images