Optimization method and system for bitset-based counting event matching algorithm

Abstract

The application provides an optimization method and system of a bit set-based counting type event matching algorithm, comprising: setting a bit set for each attribute, and the size of the bit set is equal to the number of subscriptions; initializing the set to 0; performing original event matching logic through a forward counting type algorithm, marking the bit corresponding to the subscription ID of the attribute on the bit set as 1 every time a matched predicate is found, indicating that the corresponding subscription and event on the attribute are matched; performing logical AND operation on the bit set on each attribute, and then obtaining the subscription matched with all attributes on the event. The application can optimize any forward counting type event matching algorithm, and improve the matching performance by introducing a bit set without changing the original data structure and matching logic of the optimization object, converting all counting arithmetic operations into marking operations on the bit set and logical operations between the bit sets, improving the performance of the forward counting type matching algorithm and enhancing the parallelism of the algorithm.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and more specifically, to an optimization method and system for a bit-set-based counting event matching algorithm. Background Technology

[0002] Content-matching-based publish-subscribe systems are widely used in scenarios such as stock trading, online shopping, gaming, intelligent transportation, search and recommendation, the Internet of Things, and monitoring systems. Their main function is to quickly forward events to subscribers who meet the matching semantics. The event matching algorithm is one of the core components of a content-matching publish-subscribe system, determining the system's throughput and the event forwarding latency on the proxy server. Based on fine-grained matching semantics, the event matching algorithm quickly finds subscribers who match the event, aiming to reduce the event forwarding latency on the proxy server, meet subscribers' personalized data needs in real time, improve the overall throughput of the publish-subscribe system, and prevent the event matching operation from becoming a performance bottleneck.

[0003] The problem that event matching algorithms aim to solve is to quickly find all subscriptions that match an event within a set of subscriptions; essentially, it's a high-performance search problem. The matching semantics are typically a range matching that satisfies the conjunctive normal form. That is, a subscription can define range predicates as constraints on multiple attributes in its content space. For an event to be considered a match, the values ​​of the event on these attributes must all fall within the ranges defined by the subscription. In addition to the attributes defined by the subscription's predicates, an event can also contain other attributes. The values ​​of the event on these other attributes do not affect the matching relationship between the event and the subscription.

[0004] Over decades of research, scientists have proposed numerous data structures and event matching algorithms to meet a wide variety of application scenarios, reducing event matching time for millions of subscribers from seconds to milliseconds or even microseconds. However, current work rarely proposes a general optimization method to address the performance bottlenecks of existing matching algorithms, in order to fully leverage their academic and practical value and provide a reference direction for designing new algorithms. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the purpose of this invention is to provide an optimization method and system for a bit-set-based counting event matching algorithm.

[0006] An optimization method for a bit-set-based counting event matching algorithm provided by the present invention includes:

[0007] Step S1: Set a bit set for each attribute, the size of which is equal to the number of subscriptions;

[0008] Step S2: Initialize the set to 0;

[0009] Step S3: Execute the original event matching logic using a forward counting algorithm. For each matching predicate found, mark the bit corresponding to the subscription ID of the attribute in the bit set as 1, indicating that the subscription and event corresponding to the attribute are matched.

[0010] Step S4: Perform a logical AND operation on the bit set of each attribute to obtain subscriptions that match the event on all attributes.

[0011] Preferably, after finding and marking all matching predicates, a bitwise AND operation is performed once, without needing to maintain a separate bitwise set for each attribute. That is, only two bitwise sets need to be maintained during the matching process.

[0012] Preferably, when inserting a subscription, the bit corresponding to the subscription is marked as 1 in the bit set corresponding to the attribute for which the subscription does not define a predicate, and the bit set is denoted as:

[0013]

[0014] Where d represents the number of attributes, It fully records all subscriptions that satisfy the matching semantics on the empty attribute, and can be used as an intermediate result for bit set AND operation.

[0015] Preferably, the optimization formula for the bit-set-based forward counting event matching algorithm is as follows:

[0016]

[0017] Among them, B result Represents the result set. This represents the set of matching bits on event attribute i, that is, the set of bits marked with the subscription to which the found matching predicate belongs, and d represents the number of attributes in the content space;

[0018] When the value of the event on the current property is empty, It's a set of all zeros, which can be directly omitted. Perform an OR operation; when processing each event property i, first make a copy. Then, the subscriptions corresponding to the matching predicates are marked on the copied bit set, and the OR operation is omitted.

[0019] Preferably, based on the commonalities of forward and reverse event matching algorithms, it is possible to convert between any forward and reverse matching algorithms through bit sets;

[0020] The commonalities include reverse matching using bit set OR operation to find the union of the non-matching bit sets in each dimension, and forward matching using bit set AND operation to find the intersection of the matching bit sets in each dimension.

[0021] An optimization system for a bit-set-based counting event matching algorithm provided by the present invention includes:

[0022] Module M1: Sets a bit set for each attribute, the size of which is equal to the number of subscriptions;

[0023] Module M2: Initialize the set to 0;

[0024] Module M3: Executes the original event matching logic through a forward counting algorithm. For each matching predicate found, the bit corresponding to the subscription ID of the attribute in the bit set is marked as 1, indicating that the subscription and event corresponding to the attribute are matched.

[0025] Module M4: Performs a logical AND operation on the bit set of each attribute to obtain subscriptions that match the event on all attributes.

[0026] Preferably, after finding and marking all matching predicates, a bitwise AND operation is performed once, without needing to maintain a separate bitwise set for each attribute. That is, only two bitwise sets need to be maintained during the matching process.

[0027] Preferably, when inserting a subscription, the bit corresponding to the subscription is marked as 1 in the bit set corresponding to the attribute for which the subscription does not define a predicate, and the bit set is denoted as:

[0028]

[0029] Where d represents the number of attributes, It fully records all subscriptions that satisfy the matching semantics on the empty attribute, and can be used as an intermediate result for bit set AND operation.

[0030] Preferably, the optimization formula for the bit-set-based forward counting event matching algorithm is as follows:

[0031]

[0032] Among them, B result Represents the result set. This represents the set of matching bits on event attribute i, that is, the set of bits marked with the subscription to which the found matching predicate belongs, and d represents the number of attributes in the content space;

[0033] When the value of the event on the current property is empty. It's a set of all zeros, which can be directly omitted. Perform an OR operation; when processing each event property i, first make a copy. Then, the subscriptions corresponding to the matching predicates are marked on the copied bit set, and the OR operation is omitted.

[0034] Preferably, based on the commonalities of forward and reverse event matching algorithms, it is possible to convert between any forward and reverse matching algorithms through bit sets;

[0035] The commonalities include reverse matching using bit set OR operation to find the union of the non-matching bit sets in each dimension, and forward matching using bit set AND operation to find the intersection of the matching bit sets in each dimension.

[0036] Compared with the prior art, the present invention has the following beneficial effects:

[0037] 1. The optimization method of this invention can optimize any forward counting event matching algorithm and has universality. Without changing the original data structure and matching logic of the optimization object, it improves the matching performance by introducing bit sets, transforming all counting arithmetic operations into marking operations on bit sets and logical operations between bit sets. This not only improves the performance of the forward counting matching algorithm but also enhances the algorithm's parallelism.

[0038] 2. The three theories proposed in this invention all revolve around direction matching, emphasizing both theory and application. By summarizing and generalizing the essence and commonalities of forward and reverse event matching algorithms, it innovatively proposes a general forward counting optimization method, the conclusion that any forward and reverse matching algorithm can be mutually converted through bit sets, and a method for designing new algorithms based on existing data structures.

[0039] 3. This invention utilizes a bit set data structure to perform event matching, which is beneficial for maximizing hardware performance. Attached Figure Description

[0040] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0041] Figure 1 This is a schematic diagram of the workflow of the present invention.

[0042] Figure 2 This is a schematic diagram of the data structure of the forward matching fREIN algorithm proposed in this invention based on the reverse matching REIN algorithm. Detailed Implementation

[0043] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0044] The existing matching methods are categorized as follows:

[0045] Currently, no single matching algorithm can demonstrate optimal performance in all application scenarios; each has its limitations. Based on matching strategies, matching algorithms can be categorized into forward counting, reverse labeling, and filtering algorithms.

[0046] The forward counting matching algorithm maintains a counter for each subscription. During event matching, it searches for all matching predicates based on the data structure and increments the counter for subscriptions containing these matching predicates. It then compares the number of matching predicates recorded in the counter with the total number of predicates in the subscription. If they are equal, the subscription is considered a match; otherwise, it is a mismatch. The forward counting matching algorithm requires counting all matching predicates. When a mismatched subscription contains only one mismatched predicate, counting the other matching predicates becomes redundant and ineffective. Therefore, performance degradation is a drawback of the forward counting matching algorithm when the predicate width is wide and the number of predicates in a subscription is large.

[0047] The reverse-marking matching algorithm uses a bit set to record the matching status of each subscription. During event matching, it searches for all mismatched predicates on each attribute and marks the subscriptions corresponding to these mismatched predicates as mismatched in the bit set. The unmarked bits in the bit set represent matching subscriptions. The reverse-marking matching algorithm requires marking all mismatched predicates. When there are multiple mismatched predicates in a subscription, the same subscription must be marked multiple times; subsequent markings are redundant. Therefore, poor performance in high-dimensional spaces and scenarios with small predicate widths is an inherent drawback of the reverse-marking matching algorithm.

[0048] Filtering-based matching algorithms are generally based on tree data structures to classify and store subscriptions or predicates. During event matching, tree pruning filters out most mismatched subscriptions or predicates, ultimately requiring precise checking of only a small number of potentially matching candidate subscriptions. However, filtering-based matching algorithms struggle to guarantee a high filtering ratio in all situations, especially when the matching probability of subscriptions is high and there are few mismatched subscriptions. In such cases, they fail to achieve good filtering results and ultimately require individual checking of a large number of matching subscriptions. Therefore, filtering-based matching algorithms are unsuitable for applications requiring high matching ratios.

[0049] The common feature of forward counting event matching algorithms is that during the matching process, a counter is maintained for each subscription to record the number of matching predicates, and initialized to 0. When traversing each attribute value pair of an event, all predicates that match the event value for that attribute are found, and the counter for the subscription containing the matching predicate is incremented by one. This patent considers converting the numerous incrementing arithmetic operations into marking operations on the bit set to improve the performance of the forward counting matching algorithm.

[0050] Example 1

[0051] An optimization method for a bit-set-based counting event matching algorithm provided by the present invention, such as... Figure 1 As shown, it includes:

[0052] Step S1: Set a bit set for each attribute, the size of which is equal to the number of subscriptions.

[0053] Step S2: Initialize the set to 0.

[0054] Step S3: Execute the original event matching logic using a forward counting algorithm. For each matching predicate found, mark the bit corresponding to the subscription ID of the attribute in the bit set as 1, indicating that the subscription and event corresponding to the attribute are matched.

[0055] Step S4: Perform a logical AND operation on the bit set of each attribute to obtain subscriptions that match the event on all attributes.

[0056] Specifically, during actual matching, after processing each attribute (i.e., finding and marking all matching predicates on that attribute), a bitwise AND operation is performed with the resulting bit set. It is not necessary to maintain a separate bit set for each attribute; that is, only two bit sets need to be maintained during the matching process.

[0057] Furthermore, each attribute may contain subscriptions that do not define predicates, but these subscriptions are semantically matched. Therefore, when inserting a subscription, the bit corresponding to the subscription is marked as 1 in the bit set corresponding to the attribute where the subscription does not define a predicate. In other words, a bit set is maintained on each attribute to record subscriptions with empty predicates, denoted as:

[0058]

[0059] Where d represents the number of properties. For empty properties that the event does not contain, defining predicate subscriptions on these properties will definitely not match, so there is no need to try to find matching subscriptions on empty properties. The above fully records all subscriptions that satisfy the matching semantics on empty attributes, and can be used as intermediate matching results for bitwise AND operations. It should be noted that the number of attributes is also the dimension of the content space.

[0060] The optimization formula for the bit-set-based forward counting event matching algorithm is as follows:

[0061]

[0062] Among them, B result Represents the result set. This represents the set of matching bits for event attribute i, i.e., the set of bits marked with the subscription to which the found matching predicate belongs, and d represents the number of attributes in the content space. When the value of the event on the current attribute is empty, It's a set of all zeros, which can be directly omitted. Perform an OR operation; when processing each event property i, first make a copy. Then, the subscriptions corresponding to the matching predicates are marked on the copied bit set, and the OR operation is omitted.

[0063] After using the bitset-based optimization method, the increased space complexity is O(nd), where n is the number of subscriptions and d represents the number of attributes in the content space, i.e., maintaining a bitset for each attribute, and two more bitsets are needed during the matching process; the increased insertion time complexity is O(dk), where k is the subscription size, i.e., a marking operation needs to be performed for each subscription on empty attributes; during the matching process, the marking operation replaces the previous counting operation, the number of operations remains unchanged, and the added time overhead is d bitset AND operations.

[0064] The optimization method in this invention transforms all counting arithmetic operations into marking operations on bit sets and logical operations between bit sets, which not only improves the performance of the forward counting matching algorithm but also enhances its parallelism. Before optimization, the forward counting matching algorithm needed to maintain a counter for each subscription to record the number of matching predicates. Processing different attributes might require operating on the same counter, making it difficult to parallelize the counting of matching predicates. After using the bit set optimization method, predicate lookup and marking on each attribute can be performed simultaneously. Only one bit set recording partially matching subscriptions needs to be maintained in each parallel thread. Synchronization is only required when performing a bit set AND operation on the partial matching results of each thread. Therefore, the bit set-based optimization method enhances the parallelism of the forward counting matching algorithm.

[0065] The problem this invention aims to solve is to optimize the search performance of existing forward counting matching algorithms without changing their data structure and matching logic. Furthermore, based on this optimization method, this invention summarizes the essence and commonalities of forward and reverse matching algorithms, and proposes a fundamental theorem for directional matching and a design method for forward and reverse matching algorithms.

[0066] Specifically, based on the commonalities of forward and reverse event matching algorithms, any conversion between forward and reverse matching algorithms can be achieved through bit sets. These commonalities include: reverse matching using bit set OR operations to find the union of the non-matching bit sets in each dimension, and forward matching using bit set AND operations to find the intersection of the matching bit sets in each dimension. The specific analysis derived from this commonality is as follows: the optimization method for the bit set-based forward counting event matching algorithm utilizes an abstraction that forward counting matching requires finding the commonality of all matching predicates in each dimension. The optimization method achieves its generality based on this commonality. Naturally, since the forward counting matching algorithm has this commonality, the reverse tagging event matching algorithm should also have a corresponding commonality: finding non-matching subscriptions in each dimension. Therefore, the commonalities of matching algorithms employing these two different strategies can be summarized as follows:

[0067] (1) Reverse matching: In a bit set that is initially all 0, find all the mismatched bits, mark them as 1, and then perform an OR operation on the mismatched bit set obtained in each dimension. Finally, the bits that are 0 represent the matching subscriptions.

[0068] (2) Forward matching based on bit set optimization: In a bit set that is initially all 0, find all matching bits, mark them as 1, and then perform a bitwise AND operation on the bit set obtained in each dimension. Finally, the bit that is 1 indicates the subscription of the matching.

[0069] Furthermore, considering the short-circuit property of bitwise operations, we know that when performing a bitwise OR operation, the operands that result in 1 should appear on the left as much as possible, and when performing a bitwise AND operation, the operands that result in 0 should appear on the left as much as possible. Therefore, when there are fewer 1s and more 0s, a forward match can be achieved by replacing the OR operation with an AND operation without changing the original logic of the reverse matching. Similarly, when there are fewer 0s and more 1s, a reverse match can be achieved by replacing the AND operation with an OR operation without changing the original logic of the forward matching. Based on this, the commonality of reverse matching is finding all mismatched bits, which cannot be changed. Of the other four steps, the OR operation needs to be changed to an AND operation, leaving three steps that require modification. Since it needs to be changed to an AND operation, it means that in forward matching, the 1 recorded in the bit set of each attribute should represent a matching subscription, while the mismatched subscriptions found by reverse matching can only be marked as 0. Therefore, the bit set can be initialized to all 1s, indicating that all are matches by default. Following this line of thought, we can deduce the steps to transform reverse matching into forward matching that fully performs a logical AND operation: In a bit set that is initially all 1s, find all mismatched bits and mark them as 0; then perform an AND operation on the bit set of partial matching results obtained in each dimension, and finally, the bits that are 1s represent the subscription of the matching.

[0070] Before performing the AND operation, the 1s in the matching bit set of each attribute record all the matching subscriptions for that attribute. Performing the AND operation is equivalent to finding the subscriptions that match in each dimension, so this transformation is equivalent to transforming reverse matching into forward matching.

[0071] Similarly, for forward matching, the commonality is finding all matching bits, which cannot be changed. Of the other four steps, the AND operation needs to be changed to an OR operation, leaving three steps that require modification. Since it needs to be changed to an OR operation, it means that for reverse matching, the 1 recorded in the bit set of each attribute should represent a non-matching subscription. The matching subscriptions found by forward matching can only be marked as 0. Therefore, the bit set can be initialized to 1, indicating that by default, all are non-matches. Following this line of thought, we can deduce that forward matching is transformed into reverse matching by completely performing a logical OR operation: In a bit set initially set to all 1s, find all matching bits, mark them as 0, perform an OR operation on the non-matching bit set of each dimension, and finally, the bits that are 1 represent non-matching subscriptions.

[0072] Before performing the OR operation, the 1s in the partial matching result set for each attribute represent all non-matching subscriptions for that attribute. Performing the OR operation is equivalent to finding all subscriptions with non-matching dimensions, so this transformation is equivalent to transforming forward matching into reverse matching.

[0073] Based on the above analysis and demonstration, we can derive the fundamental theorem of matching strategy transformation: any forward and reverse matching algorithm can be mutually transformed through bit sets. Therefore, the design method for forward and reverse matching algorithms based on the same data structure is as follows: storing matching predicates and searching for matching predicates allows for the design of a forward matching algorithm; storing non-matching predicates and searching for non-matching predicates allows for the design of a reverse matching algorithm. This method can provide inspiration for designing new data structures and matching algorithms.

[0074] Furthermore, the feasibility, effectiveness, and universality of the optimization method and direction matching transformation strategy proposed in this invention are verified using the TAMA algorithm and the REIN algorithm, respectively, as described in detail below:

[0075] I. Verification based on the TAMA algorithm

[0076] 1. Application of forward and reverse matching algorithm design method to TAMA algorithm

[0077] Finding matching predicates using a data structure based on the TAMA algorithm is straightforward. Each event falls into exactly one bucket at each level, and retrieving these buckets yields all matching predicates. To find unmatched predicates, following the design method of forward and reverse matching algorithms, the unmatched intervals should first be stored. Let the interval predicate be [,h], and the attribute value range be [1,R]. Then the unmatched intervals are [1,l-1] and [h+1,R] (empty intervals when l=1 and h=R). Simply insert and store these two unmatched intervals using the original TAMA algorithm. During event matching, retrieving the bucket into which the event value falls yields all unmatched predicates. Mark the subscriptions corresponding to these unmatched predicates on the bit set. Finally, the subscriptions not marked on the bit set are the matching results.

[0078] For subscriptions to properties with undefined predicates, no additional processing is required. For empty properties in an event, subscriptions with defined predicates on these properties are all mismatched. A bitset can be maintained beforehand for each property to record subscriptions with defined predicates on that property. During matching, a logical OR operation can be used to obtain the mismatched subscriptions on empty properties in the event. The entire reverse matching process can be expressed by the formula:

[0079]

[0080] First item This indicates searching and marking non-matching subscriptions on m properties of an event with defined values, the second item. This indicates that dm logical OR operations are performed on dm empty properties in the event to merge mismatched subscriptions on these properties. Recorded in attribute a j The predicate subscription is defined above. Thus, TAMA can be transformed into a reverse event matching algorithm, bTAMA.

[0081] 2. An optimization method for the forward counting matching algorithm based on bit sets is applied to the TAMA algorithm.

[0082] TAMA is a forward counting event matching algorithm, which is a target object of optimization methods, and therefore bit sets can be used for optimization. According to formula (1), when inserting a subscription, the subscription is marked in the bit set corresponding to the attribute for which the subscription does not define a predicate. During matching, it is based on B. null The matching predicates found by the TAMA algorithm are marked on the event attributes. Then, a logical AND operation is performed on the matching bit sets on each attribute to obtain the subscriptions that match the event on each attribute. The TAMA algorithm optimized based on bit sets is called TAMA-C.

[0083] 3. The fundamental theorem of direction matching is applied to the TAMA algorithm.

[0084] Both bTAMA and TAMA-C use bit sets for event matching. According to the fundamental theorem of matching strategy transformation, the reverse matching algorithm bTAMA can be converted into the forward matching algorithm bTAMA-F, and the forward matching algorithm TAMA-C can be converted into the reverse matching algorithm TAMA-CB.

[0085] The steps to convert the reverse matching algorithm bTAMA to the forward algorithm bTAMA-F are as follows:

[0086] Step 1: Initialize a bit set B consisting entirely of 1s, indicating that the default subscriptions are all matching;

[0087] Step 2: Process each non-empty attribute of the event, find the mismatched subscriptions on these attributes according to the bTAMA algorithm, and unmark these subscriptions in B;

[0088] Step 3: Process each empty property of the event, and set the bit set B on the property. null Perform a bitwise AND operation with B, and store the result in B. Finally, bits in B with a value of 1 represent matched subscriptions.

[0089] This process of converting a reverse direction to a forward direction can be expressed by the following formula:

[0090]

[0091] That is, the formula (2) for the reverse matching of the logical OR operation is transformed into the formula (3) for the forward matching of the logical AND operation.

[0092] The steps to convert the forward matching algorithm TAMA-C to the reverse matching algorithm TAMA-CB are as follows:

[0093] Step 1: Initialize a bit set B consisting entirely of zeros, indicating that the default subscriptions are all matching;

[0094] Step 2: For each non-empty property of the event, based on B def Copy a bit set B′ that records the subscriptions that define predicate constraints on this attribute. def ;

[0095] Step 3: Find matching subscriptions on these attributes using the TAMA-C algorithm, and in B′ def Remove the flags for these subscriptions in the middle, and finally B′ def All non-matching subscriptions on this property are recorded, and an OR operation is performed between them and B;

[0096] Step 4: For each empty property of the event, set the B attribute on the property. def Perform a direct OR operation with B, and store the result in B. Finally, bits in B with a value of 0 represent matched subscriptions.

[0097] This process of converting a forward direction to a reverse direction can also be expressed by a formula:

[0098]

[0099] That is, the forward matching formula (1) is converted into the reverse matching formula (4) that performs the logical OR operation.

[0100] It should be noted that the OR operation in formula (1) and the AND operation in formula (4) are eliminated by bit set copying in actual operation. Therefore, the forward matching formula (1) only needs to perform the AND operation, and the reverse matching formula (4) only needs to perform the OR operation. The forward matching formula (3) and reverse matching formula (4) obtained by the transformation of the basic theorem of matching strategy are different from the original forward matching formula (1) and reverse matching formula (2) in terms of handling event attributes, so they are not equivalent.

[0101] II. Verification based on the REIN algorithm

[0102] 1. Application of forward and reverse matching algorithm design method to REIN algorithm

[0103] Based on the original REIN data structure, matching predicates can be found. Using forward and reverse matching algorithm design methods, a forward counting event matching algorithm (forward REIN, fREIN) can be designed, such as... Figure 2 As shown. Let the number of buckets be 16, meaning the value space of each attribute is equally divided into 16 subspaces. Suppose an event value falls into the 12th subspace. Using the REIN insertion algorithm... Figure 2 The dashed lines indicate the buckets into which the three predicates should be inserted at the low-value end (LVE) and high-value end (HVE).

[0104] For the native REIN, buckets 13-16 on LVE and buckets 1-11 on HVE store unmatched predicates. That is, the shaded area in the diagram needs to be traversed one by one and the predicates in the buckets need to be marked in reverse. The predicates in the two buckets 12 need to be compared with the event values ​​to determine whether they match.

[0105] Conversely, buckets 1-12 on LVE and buckets 12-16 on HVE contain all matching predicates. If we want to find matching predicates and perform a forward count, we can iterate through buckets 1-11 on LVE. The low values ​​of the predicates in these buckets already meet the requirement of being less than or equal to the event value; we only need to check if their corresponding high values ​​are greater than or equal to the event value. For easier detection, the entire interval of predicates can be stored in the buckets of LVE. The low values ​​are used to find the matching predicate in bucket 12 where the event falls, and the high values ​​are used to find the matching predicate in buckets 1-12. Since buckets 1-12 of LVE store the matching predicates for this attribute, the HVE bucket column does not need to be processed, and therefore, there is no need to set an HVE bucket column.

[0106] Similarly, matching predicates can be found from HVE, that is, checking the low values ​​of predicates in buckets 13-16 of HVE, and performing high-low value checks on predicates in bucket 12. This eliminates the need to process LVE. If LVE and HVE are combined, the end requiring fewer buckets can be chosen for forward counting. Figure 1 In the LVE, 12 buckets need to be checked, while in the HVE, 5 buckets need to be checked. Through this heuristic comparison, the matching predicate is ultimately selected to be found on the HVE. This optimization is called high-low value heuristic selection optimization.

[0107] Considering that the number of buckets traversed does not accurately represent the load size when the load is unbalanced, the load can be refined from the bucket granularity to the predicate granularity within the bucket: Maintain an array recording the total size of the prefix buckets and an array recording the total size of the suffix buckets on both LVE and HVE, and select the end with fewer predicates to be marked. This optimization is called predicate-granular load optimization.

[0108] Considering the potential large number of predicates within a bucket, and the need to check the low or high values ​​of predicates for each bucket, we can ensure that predicates are stored in order within the buckets during insertion: On the LVE, the bucket number to be inserted is determined by the low value of the predicate, and within each bucket, they are stored in ascending order by the high value of the predicate; on the HVE, the bucket number is determined by the high value of the predicate, and within each bucket, they are stored in ascending order by the low value of the predicate. During detection, the critical point to be marked can be quickly located within each bucket using binary search. This optimization is called the high-low value dual sorting optimization.

[0109] 2. An optimization method based on bit set forward counting matching is applied to the REIN algorithm.

[0110] fREIN searches for matching predicates in each dimension of the REIN native data structure and performs forward counting. Finally, it determines whether a subscription matches by comparing the number of matches for each subscription with the subscription size. This process shares common characteristics with forward counting event matching algorithms and can directly utilize optimization methods for bit-set-based forward counting matching algorithms. Similar to the optimization of the TAMA algorithm mentioned earlier, when inserting a subscription, it searches the bit set... null The above records the empty attributes of the subscription, and copies B are made first during matching. null Then, the matching predicates searched by fREIN logic are marked on the copied bit set to obtain all matching subscriptions for an attribute. A bitwise AND operation is performed on the bit set for each attribute, and the bits with a value of 1 in the final bit set represent matching subscriptions. The bit-set optimized fREIN is called fREIN-C.

[0111] 3. The fundamental theorem of matching strategy transformation is applied to the REIN algorithm.

[0112] According to the fundamental theorem of matching strategy transformation, the reverse matching algorithm REIN can be transformed into the forward matching algorithm REIN-F through bit sets, and the forward matching algorithm fREIN-C can be transformed into the reverse matching algorithm fREIN-CB through bit sets. This is similar to the transformation of the TAMA algorithm.

[0113] The reverse matching process of the REIN algorithm can be simply expressed by the following formula:

[0114]

[0115] The matching process after the reverse matching algorithm REIN is converted into the forward matching algorithm REIN-F can be represented by formula (3), or it can be simplified to:

[0116]

[0117] The conversion steps are as follows:

[0118] Step 1: When inserting a subscription, maintain a bit set B for each attribute. null This records subscriptions for which no predicate is defined on this property;

[0119] Step 2: During matching, first initialize a bit set B consisting entirely of 1s to store the final matching result;

[0120] Step 3: Traverse each non-empty attribute of the event, and mark all subscriptions corresponding to the mismatched predicates found based on the REIN algorithm as 0 in B;

[0121] Step 4: For each empty property of the event, the subscription of the predicate defined on the property is mismatched. At this point, B unmatch That is B def ,then B unmatc h is B null B directly null Performing a bitwise AND operation with B will yield subscriptions that match each attribute.

[0122] The process of converting the forward matching algorithm fREIN-C to the reverse matching algorithm fREIN-CB is similar to the process of converting the forward matching algorithm TAMA-C to the reverse matching algorithm, i.e., formula (4), except that B... match That is, the way to obtain the matching predicate has changed from the tree-like branching structure of TAMA to traversing and binary searching on the bucket column structure of REIN.

[0123] Example 2

[0124] The present invention also provides an optimization system for a bit-set-based counting event matching algorithm. Those skilled in the art can implement the optimization system by executing the steps of the optimization method for the bit-set-based counting event matching algorithm. That is, the optimization method for the bit-set-based counting event matching algorithm can be understood as a preferred embodiment of the optimization system for the bit-set-based counting event matching algorithm.

[0125] An optimization system for a bit-set-based counting event matching algorithm provided by the present invention includes:

[0126] Module M1: Sets a bit set for each attribute, the size of which is equal to the number of subscriptions.

[0127] Module M2: Initializes the set to 0.

[0128] Module M3: Executes the original event matching logic through a forward counting algorithm. For each matching predicate found, the bit corresponding to the subscription ID of the attribute in the bit set is marked as 1, indicating that the subscription and event corresponding to the attribute are matched.

[0129] Module M4: Performs a logical AND operation on the bit set of each attribute to obtain subscriptions that match the event on all attributes.

[0130] After finding and marking all matching predicates, a bitwise AND operation is performed once. There is no need to maintain a separate bitwise set for each attribute; that is, only two bitwise sets need to be maintained during the matching process.

[0131] When inserting a subscription, the bit corresponding to the subscription is marked as 1 in the bit set corresponding to the attribute for which no predicate is defined in the subscription. This bit set is denoted as:

[0132]

[0133] Where d represents the number of attributes, It fully records all subscriptions that satisfy the matching semantics on the empty attribute, and can be used as an intermediate result for bit set AND operation.

[0134] The optimization formula for the bit-set-based forward counting event matching algorithm is as follows:

[0135]

[0136] Among them, B result Represents the result set. This represents the set of matching bits for event attribute i, i.e., the set of bits marked with the subscription to which the found matching predicate belongs, and d represents the number of attributes in the content space. When the value of the event on the current attribute is empty, It's a set of all zeros, which can be directly omitted. Perform an OR operation; when processing each event property i, first make a copy. Then, the subscriptions corresponding to the matching predicates are marked on the copied bit set, and the OR operation is omitted.

[0137] Based on the commonalities of forward and reverse event matching algorithms, it is possible to convert between any forward and reverse matching algorithms using bit sets. These commonalities include: reverse matching using bit set OR operations to find the union of the non-matching bit sets in each dimension, and forward matching using bit set AND operations to find the intersection of the matching bit sets in each dimension.

[0138] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0139] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. An optimization method for a bit-set-based counting event matching algorithm, characterized in that, include: Step S1: Set a bit set for each attribute, the size of which is equal to the number of subscriptions; Step S2: Initialize the bit set to 0; Step S3: Execute the original event matching logic using a forward counting algorithm. For each matching predicate found, mark the bit corresponding to the subscription ID of the attribute in the bit set as 1, indicating that the subscription and event corresponding to the attribute are matched. Step S4: Perform a logical AND operation on the bit set of each attribute to obtain subscriptions that match the event on all attributes; When inserting a subscription, the bit corresponding to the subscription is marked as 1 in the bit set corresponding to the attribute for which no predicate is defined in the subscription. This bit set is denoted as: ； Where d represents the number of attributes, The above fully records all subscriptions that satisfy the matching semantics on the empty attribute, and can be used as intermediate matching results for bit set AND operations; The optimization formula for the bit-set-based forward counting event matching algorithm is as follows: in, Represents the result set. Indicates in event properties The matching bit set is the set of bits that are tagged with the subscriptions to which the found matching predicates belong. Indicates the number of attributes in the content space; When the value of the event on the current property is empty, It's a set of all zeros, which can be directly omitted. Execute or compute; when processing each event property First, make a copy. Then, the subscription corresponding to the matching predicate is marked on the copied bit set, and the OR operation is omitted.

2. The optimization method for the bit-set-based counting event matching algorithm according to claim 1, characterized in that, After finding and marking all matching predicates, a bitwise AND operation is performed once. There is no need to maintain a separate bitwise set for each attribute; that is, only two bitwise sets need to be maintained during the matching process.

3. The optimization method for the bit-set-based counting event matching algorithm according to claim 1, characterized in that, Based on the commonalities of forward and reverse event matching algorithms, it is possible to convert between any forward and reverse matching algorithms through bit sets; The commonalities include reverse matching using bit set OR operation to find the union of the non-matching bit sets in each dimension, and forward matching using bit set AND operation to find the intersection of the matching bit sets in each dimension.

4. An optimization system for a bit-set-based counting event matching algorithm, characterized in that, include: Module M1: Sets a bit set for each attribute, the size of which is equal to the number of subscriptions; Module M2: Initializes the bit set to 0; Module M3: Executes the original event matching logic through a forward counting algorithm. For each matching predicate found, the bit corresponding to the subscription ID of the attribute in the bit set is marked as 1, indicating that the subscription and event corresponding to the attribute are matched. Module M4: Performs a logical AND operation on the bit set of each attribute to obtain subscriptions that match the event on all attributes; When inserting a subscription, the bit corresponding to the subscription is marked as 1 in the bit set corresponding to the attribute for which no predicate is defined in the subscription. This bit set is denoted as: ； Where d represents the number of attributes, The above fully records all subscriptions that satisfy the matching semantics on the empty attribute, and can be used as intermediate matching results for bit set AND operations; The optimization formula for the bit-set-based forward counting event matching algorithm is as follows: in, Represents the result set. Indicates in event properties The matching bit set is the set of bits that are tagged with the subscriptions to which the found matching predicates belong. Indicates the number of attributes in the content space; When the value of the event on the current property is empty, It's a set of all zeros, which can be directly omitted. Execute or compute; when processing each event property First, make a copy. Then, the subscription corresponding to the matching predicate is marked on the copied bit set, and the OR operation is omitted.

5. The optimization system for the bit-set-based counting event matching algorithm according to claim 4, characterized in that, After finding and marking all matching predicates, a bitwise AND operation is performed once. There is no need to maintain a separate bitwise set for each attribute; that is, only two bitwise sets need to be maintained during the matching process.

6. The optimization system for the bit-set-based counting event matching algorithm according to claim 4, characterized in that, Based on the commonalities of forward and reverse event matching algorithms, it is possible to convert between any forward and reverse matching algorithms through bit sets; The commonalities include reverse matching using bit set OR operation to find the union of the non-matching bit sets in each dimension, and forward matching using bit set AND operation to find the intersection of the matching bit sets in each dimension.

Images