Interval event matching method and system for content-based publish / subscribe systems

Abstract

The application provides an interval event matching method and system of a content-based publish / subscribe system, through constructing a predicate disjoint set and an optimized partition model thereof, and integrating a learning index of a range query recursive model index in each partition, high-speed, stable and verifiable matching of events to a subscription set under multi-attribute interval predicates is realized; in particular, the traditional memory-intensive matching is reconstructed into a calculation-intensive process mainly based on model reasoning, while correctness and completeness are ensured, average latency is reduced and long tail delay is inhibited, and key modules such as partition and coverage optimization, model training and online reasoning, error boundary control and limited correction, and accurate back matching of residual set subscription are included.

Description

Technical Field

[0001] This invention relates to the technical fields of content-based publish / subscribe systems, event matching algorithms, machine learning and index optimization, and more specifically, to a method and system for interval event matching in a content-based publish / subscribe system. Background Technology

[0002] Content-based publish / subscribe (CPS) systems are crucial infrastructure for distributed data distribution. Their core task is to efficiently and accurately match continuously arriving events with a large and complex set of subscriptions. As the real-time and granular requirements of business scenarios continue to increase, the number of subscriptions, attribute dimensions, and predicate complexity grow exponentially, making event matching a major bottleneck for system throughput and stability. Typical subscriptions are given as conjunctions of multi-attribute interval predicates, exhibiting significant interval overlap and correlation, leading to large index structures, unstable memory access paths, and frequent cache misses. Under the combined effects of peak loads and skewed data distribution, average latency and standard deviation increase simultaneously, with long-tail latency issues being particularly prominent, severely impacting service quality and system predictability. Existing engineering practices often rely on a large amount of manual tuning and empirical structural optimization (such as tree partitioning, hash bucketing, bitmap compression, etc.) to reduce online computing costs. However, when subscriptions and events expand together, such "memory-intensive" indexes are difficult to reside in the cache. The frequent rollback to main memory access not only slows down the matching speed but also amplifies tail jitter, making it difficult to guarantee stability.

[0003] In recent years, the emerging learned indexing paradigm proposes to approximate data distribution or location mapping with lightweight models, replacing the traversal of large-scale pointer structures with a "prediction + limited correction" approach. It attempts to reconstruct traditional lookups from memory-access-driven to a "computationally intensive" process dominated by inference, thereby achieving comprehensive benefits in terms of latency and space consumption. While this paradigm has shown considerable potential in sorted indexes, hashing, and filtering scenarios, directly migrating it to CPS event matching is not easy: First, CPS semantics involve a "one-to-many" full match, requiring the output of all subscriptions satisfying the conjunction predicate, resulting in stricter constraints on correctness and completeness, far exceeding the limitations of "choose one" decisions. Second, the extensive overlap between multi-attribute intervals and wildcard predicates undermines the single-value mapping and separability assumptions commonly used in learned models, leading to label conflicts, model capacity expansion, and difficulty in characterizing error bounds. Third, online scenarios impose stringent requirements on stability, low latency, and cache friendliness; learned indexes lacking clear error upper bounds and verifiable correction mechanisms struggle to meet industrial-grade availability requirements. The aforementioned differences make "how to maintain completeness while supporting high throughput matching with steady-state model inference" a key technical issue.

[0004] From the perspective of load characteristics and engineering constraints, CPS event streams often exhibit characteristics such as the coexistence of burstiness and hot attributes, and dynamic evolution of subscription sets, further exacerbating the contradiction between index residency and update maintenance. Traditional tree, hash, and bitmap structures are prone to index bloat and path jitter in high-dimensional overlapping situations, are extremely sensitive to cache levels, and struggle to maintain low-tail latency stably under ultra-large-scale conditions. Existing explorations of using learned indexes for range matching have shown that hierarchical models and piecewise linear approximations can provide analytical error control for one-dimensional interval positioning. However, under the CPS semantics of multi-domain, multi-attribute conjunction, and full-match output, the following issues still need to be addressed: how to construct learnable and mutually exclusive intra-attribute partitions to alleviate overlap; how to provide computable error boundaries and implement small window correction in range inference to ensure no missed detections; and how to operate stably under peak and skewed loads using a pipelined and parallelized system. Furthermore, in subscription areas with low diversity and high overlap, learning indexes often struggle to establish robust mapping relationships, inevitably introducing fallback paths. Without a collaborative design with the exact matcher, the fallback ratio and cost will be uncontrollable, thus affecting the convergence of overall latency and variance.

[0005] Patent application CN110413927A discloses an optimization method and system based on real-time matching in a publish-subscribe system. This method calculates the matching probability between subscriptions and events, differentiates subscriptions based on the matching probability to form subscription group categories, and establishes a structural index for the data structure based on these categories. The method then adjusts the subscription group category and its position in the structural index in real time according to changes in the subscription matching probability to improve matching efficiency. However, this patent cannot completely solve the existing technical problems, nor can it meet the needs of this invention. Summary of the Invention

[0006] In view of the deficiencies in the prior art, the purpose of this invention is to provide a method and system for interval event matching in a content-based publish / subscribe system.

[0007] The interval event matching method for a content-based publish / subscribe system provided by the present invention includes:

[0008] Step 1: Receive event streams from external data sources through the system's input interface. The events are structured data with multiple attribute value pairs.

[0009] Step 2: By optimizing the predicate disjoint set partitioning OPDS engine, the subscription set stored in the system memory is divided into k predicate disjoint subsets PDS according to one or more selected main attributes. In each of the predicate disjoint subsets, the interval predicates of any two subscriptions on the main attribute do not overlap. The value of k is determined by solving a constraint optimization model with the goal of minimizing the total matching delay.

[0010] Step 3: For each predicate disjoint subset, call its corresponding pre-trained learned interval indexing model RQ-RMI, use the attribute value of the event as input for forward inference, and output the index position prediction value of the candidate subscription; based on the prediction value and the pre-calculated maximum prediction error ε, determine a correction window in the index position sequence; perform local retrieval and predicate verification within the correction window to obtain the first matching subscription set;

[0011] Step 4: For the subscription remainder set not covered by any of the predicate disjoint subsets, call the exact backoff matching engine OEM, and by searching the pre-computed bit set cache, perform bitwise logic operations to filter out the subscriptions that match the event, and obtain the second matching subscription set;

[0012] Step 5: Merge the first matching subscription set and the second matching subscription set using the result aggregator to generate the final full matching subscription set, and distribute it to the corresponding subscribers through the system's output interface.

[0013] Preferably, the partitioning of the OPDS engine is based on a coverage model, wherein for attributes... The probability that two subscription intervals do not overlap is:

[0014]

[0015] in, The cumulative distribution function represents the standard normal distribution; It is the expected width of the predicate interval; It is the variance of the predicate interval width; It is the variance of the predicate interval center;

[0016] The coverage model includes the maximum PDS coverage of a single attribute. The calculation is as follows:

[0017]

[0018] in, It is an attribute Subscription count; It is the probability of paired predicate overlap; It is a quantity of asymptotically lower order than 1;

[0019] For cases with multiple attributes, attribute subsets The combined coverage on the above satisfies:

[0020]

[0021] in, It is the total number of PDS. It is the index sequence of PDS. It is an attribute The number of PDS on the device It is an attribute The probability of paired predicate overlap.

[0022] Preferably, the OPDS engine determines the optimal number of PDS k, attribute subset J, and subscription allocation by solving an optimization model. The objective function of the optimization model is:

[0023]

[0024]

[0025]

[0026]

[0027] in, It is the total matching delay. This refers to the inference time per iteration of the LIEM model. It is the validation time for a single attribute. N is the number of attributes participating in the validation, and N is the total number of subscriptions. It's about coverage. It is a rollback to the matching unit time. It is the error boundary. It is for finding the single-step time. This refers to the memory usage of a single LIEM model. It is the memory overhead of the remainder set. It's the memory limit. It is the target coverage rate. It is the minimum subscription allocation;

[0028] The optimization model is solved using dynamic programming combined with an approximate heuristic method, which utilizes the submodularity and monotonicity of the covering function to search for an approximate optimal solution.

[0029] Preferably, the learning-type interval index RQ-RMI includes a three-layer fully connected neural network with a ReLU activation function, wherein the first layer input is the event attribute value, the output is truncated and mapped to the range [0,1] as the selection input for the next layer sub-model, and the leaf layer outputs the predicted subscription position;

[0030] The RQ-RMI calculates the maximum prediction error at the leaf layer. To ensure that the actual subscription location is within the range Within this range, a secondary retrieval and verification are performed through finite correction, where prediction is the direct output of the model's inference.

[0031] Preferably, the precise fallback matching OEM uses a bit set caching mechanism (SCM), in which the attribute value domain is discretized into multiple units and grouped, the invalid subscription bit set corresponding to each group is pre-calculated offline, and during online matching, the bit sets are merged by bitwise OR operation and the uncovered units are supplemented with markers to obtain the candidate matching bit set;

[0032] The bitset operation of the OEM supports single-instruction multiple-data (SIMD) parallel processing for efficient matching of remainder subscriptions.

[0033] The interval event matching system for a content-based publish / subscribe system provided by the present invention includes:

[0034] Module M1: Receives event streams from external data sources through the system's input interface. The events are structured data with multiple attribute value pairs.

[0035] Module M2: By optimizing the predicate disjoint set partitioning OPDS engine, the subscription set stored in the system memory is divided into k predicate disjoint subsets PDS according to one or more selected main attributes. In each of the predicate disjoint subsets, the interval predicates of any two subscriptions on the main attribute do not overlap. The value of k is determined by solving a constraint optimization model with the goal of minimizing the total matching delay.

[0036] Module M3: For each predicate disjoint subset, call its corresponding pre-trained learned interval indexing model RQ-RMI, use the attribute values ​​of the event as input for forward inference, and output the predicted index position value of the candidate subscription; based on the predicted value and the pre-calculated maximum prediction error ε, determine a correction window in the index position sequence; perform local retrieval and predicate verification within the correction window to obtain the first matching subscription set;

[0037] Module M4: For the subscription remainder set not covered by any of the predicate disjoint subsets, call the exact backoff matching engine OEM, and by searching the pre-computed bit set cache, perform bitwise logic operations to filter out subscriptions that match the event, and obtain the second matching subscription set;

[0038] Module M5: Merges the first matching subscription set and the second matching subscription set through the result aggregator to generate the final full matching subscription set, and distributes it to the corresponding subscribers through the system's output interface.

[0039] Preferably, the partitioning of the OPDS engine is based on a coverage model, wherein for attributes... The probability that two subscription intervals do not overlap is:

[0040]

[0041] in, The cumulative distribution function representing the standard normal distribution; It is the expected width of the predicate interval; It is the variance of the predicate interval width; It is the variance of the predicate interval center;

[0042] The coverage model includes the maximum PDS coverage of a single attribute. The calculation is as follows:

[0043]

[0044] in, It is an attribute Subscription count; It is the probability of paired predicate overlap; It is a quantity of asymptotically lower order than 1;

[0045] For cases with multiple attributes, attribute subsets The combined coverage on the above satisfies:

[0046]

[0047] in, It is the total number of PDS. It is the index sequence of PDS. It is an attribute The number of PDS on the device It is an attribute The probability of paired predicate overlap.

[0048] Preferably, the OPDS engine determines the optimal number of PDS k, attribute subset J, and subscription allocation by solving an optimization model. The objective function of the optimization model is:

[0049]

[0050]

[0051]

[0052]

[0053] in, It is the total matching delay. This refers to the inference time per iteration of the LIEM model. It is the validation time for a single attribute. N is the number of attributes participating in the validation, and N is the total number of subscriptions. It's about coverage. It is a rollback to the matching unit time. It is the error boundary. It is for finding the single-step time. This refers to the memory usage of a single LIEM model. It is the memory overhead of the remainder set. It is the memory limit. It is the target coverage rate. It is the minimum subscription allocation;

[0054] The optimization model is solved using dynamic programming combined with an approximate heuristic method, which utilizes the submodularity and monotonicity of the covering function to search for an approximate optimal solution.

[0055] Preferably, the learning-type interval index RQ-RMI includes a three-layer fully connected neural network with a ReLU activation function, wherein the first layer input is the event attribute value, the output is truncated and mapped to the range [0,1] as the selection input for the next layer sub-model, and the leaf layer outputs the predicted subscription position;

[0056] The RQ-RMI calculates the maximum prediction error at the leaf layer. To ensure that the actual subscription location is within the range Within this range, a secondary retrieval and verification are performed through finite correction, where prediction is the direct output of the model's inference.

[0057] Preferably, the precise fallback matching OEM uses a bit set caching mechanism (SCM), in which the attribute value domain is discretized into multiple units and grouped, the invalid subscription bit set corresponding to each group is pre-calculated offline, and during online matching, the bit sets are merged by bitwise OR operation and the uncovered units are supplemented with markers to obtain the candidate matching bit set;

[0058] The bitset operation of the OEM supports single-instruction multiple-data (SIMD) parallel processing for efficient matching of remainder subscriptions.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] (1) By reconstructing the core process of event matching from the traditional "memory access-dominated" to "model inference computation-dominated", and utilizing a lightweight model resident cache, memory access latency is significantly reduced. Tests on typical workloads and real datasets show that, compared with existing methods such as REIN, Ada-REIN, WPA-REIN, OEM, and TAMA, the average matching latency of this method (LIEM) is reduced by approximately 88.51%, 86.05%, 83.30%, 39.71%, and 83.93%, respectively.

[0061] (2) By optimizing partitioning and a verifiable inference correction mechanism, the execution time of the matching path is made more certain; experimental results show that the standard deviation of the matching time decreased by about 49.15%, effectively converging the long-tail delay and improving the service quality and predictability of the system under peak load and data skew conditions.

[0062] (3) By calculating the maximum prediction error ε in the learned index (RQ-RMI) and performing finite window correction, the missed detection of subscriptions is avoided from the mechanism. At the same time, exact backoff matching (OEM) is used for the uncovered residual set subscriptions to ensure the 100% correctness and completeness of the final matching results.

[0063] (4) By using the digital coverage function and resource constraint optimization model in OPDS partitioning, it is possible to accurately find the balance between "too few partitions leading to a large number of rollbacks" and "too many partitions leading to model and verification overload", thereby achieving near-optimal system configuration under a given cache and latency budget;

[0064] (5) The lightweight model parameters and bit set structure adopted can be resident in the high-speed cache. Its core operations (model inference, bitwise operation) are easy to implement in vectorized parallel processing on modern general-purpose CPUs, thus providing support for the stable operation of the system in high-throughput scenarios. It is suitable for fields with high real-time requirements such as Internet of Things, financial distribution, online recommendation and advertising. Attached Figure Description

[0065] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0066] Figure 1 This is a schematic diagram of the overall system framework of the present invention: showing the OPDS partition, RQ-RMI inference of each partition, OEM rollback and result aggregation pipeline;

[0067] Figure 2 For the OPDS solution algorithm;

[0068] Figure 3 This refers to the LIEM algorithm. Detailed Implementation

[0069] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0070] Example 1

[0071] This invention proposes a high-throughput, stable, and low-latency event matching method for content publishing / subscription systems. It restructures event matching from a "memory access-driven" to a "reasoning computation-driven" approach at the paradigm level, significantly reducing average and long-tail latency while ensuring completeness and correctness. Figure 1 As shown, the system receives events (attribute-value pairs). First, the OPDS engine divides the subscription set into k PDSs based on the primary attribute according to the optimal partitioning scheme. Then, the RQ-RMI models within each PDS infer in parallel, predicting candidate subscription positions and correcting and validating them within a small window. For uncovered remainder sets, the OEM performs bit set marking. Finally, the PDS matching the OEM results are aggregated to output a fully matched subscription set. The above process implements a "two-level divide-and-conquer" approach from top to bottom: macroscopic "attribute-aligned non-overlapping partitioning" and microscopic "recursive assignment at the model level".

[0072] The overall architecture adopts a two-level divide-and-conquer and hybrid matching approach: Optimized Predicate Disjoint Partition (OPDS) + Learned Range Index (RQ-RMI) + Exact Matching of Remainder Sets (OEM).

[0073] 1) OPDS partitioning strategy and cost optimization: Based on the main attribute, the subscription set is divided into several predicate-disjoint sets (PDS) according to the principle of non-overlapping intervals. The OPDS cost model is used to solve the optimal number of PDS and attribute allocation among "model coverage - inference / verification overhead - cache usage", thereby decomposing the high-dimensional one-to-many matching into several parallel and learnable one-dimensional subproblems.

[0074] 2) Learned Interval Indexing (RQ-RMI): A set of lightweight neural sub-models is trained for each PDS to directly learn the deterministic mapping of "event attribute value → subscription ordered position". The sub-model output is truncated and quantized in [0,1] for use in the next layer selection. The leaf layer evaluates the maximum prediction error for trigger points and boundary points. To ensure that the actual position falls within [prediction- ,prediction+ The provable boundary of ] is combined with local finite correction to achieve efficient range matching.

[0075] 3) Exact Matching Engine (OEM): For subscription remainders that cannot form large-scale disjoint sets (high overlap / low diversity), the Inverse Exact Matching Engine (OEM) is used. OEM pre-computes and caches bitset groups based on the Subscription Caching Mechanism (SCM), and quickly eliminates mismatched subscriptions online with constant bitwise OR operations (which can be accelerated by SIMD), ensuring end-to-end completeness and stability.

[0076] To quantitatively guide PDS scaling, this invention establishes an analytical model of coverage and latency, and solves for the optimal number of PDS (k), attribute subset (J), and attribute allocation ({n_j}) using constrained optimization. First, the following conclusions are given regarding the non-overlapping probability of one-dimensional intervals:

[0077] Lemma 1 (Interval Non-overlapping Probability): Suppose that in attribute Two subscriptions on and The predicate intervals are respectively and , Predicate interval The minimum and maximum values, Predicate interval The minimum and maximum values. Parameterize the interval with its center and width. , They are independent and follow a normal distribution Interval width , They are independent and follow a normal distribution. ,and , Small enough to guarantee with high probability , .

[0078] in, , They represent predicates respectively. The interval center and predicate The center of the interval; These represent the expected value and variance of the population distribution parameters of the predicate interval center, respectively. These represent the expected value and variance of the overall distribution parameters for the predicate interval width, respectively.

[0079] The probability that the two intervals do not overlap is:

[0080]

[0081] in, The cumulative distribution function (CDF) represents the standard normal distribution.

[0082] Proof: To derive the interval and The probability of non-overlapping, note that non-overlapping if and only if Completely located The left side ( ), or the opposite (i.e. ).because and The distributions of the two events are independent and identically distributed, and the two events are symmetric about the distribution. Therefore, we have:

[0083]

[0084] in, Representing attributes The probability that the two predicate intervals do not intersect; express Completely located The probability on the left;

[0085] Will and Substituting, we get:

[0086]

[0087] make W= Then the probability we are looking for is reduced to its difference. Obey ,therefore:

[0088]

[0089] in, The directed distance (center difference) between the centers of the two intervals is represented by W; W represents the average of the two widths.

[0090] Lemma 2 (Maximum PDS Expected Size): Suppose that in the attribute There is a set of predicates. Each predicate Independently generated, its center ,width Let the probability of pairwise overlap between predicates be . This probability is derived from Lemma 1. Assume the predicate parameter ( If the predicates are independent of each other, and the overlap relationships between predicates are also independent, then the expected size of the largest disjoint predicate set on the predicate graph satisfies:

[0091]

[0092] in, Indicates the maximum PDS; Representing attributes The number of predicates on; It represents a quantity of asymptotic order lower than n;

[0093] Proof: Each predicate Define an undirected graph Given a vertex, there exists an edge in the overlapping region. The probability of overlap is:

[0094]

[0095] make Let k be the number of disjoint sets of size k. Then:

[0096]

[0097] in, This represents the expected number of PDS of size k;

[0098] maximize get:

[0099]

[0100] in, Represents the largest disjoint subset of predicates;

[0101] Therefore, we get:

[0102]

[0103] Theorem 1 (Maximum PDS Coverage for a Single Attribute) Considering attributes n subscriptions on the platform, where each predicate is an independent sampling center. and width .make Let represent the pairwise overlap probability derived from Lemma 1. Under the assumptions of mutual independence and sparsity, the expected coverage of the maximally disjoint predicate set (PDS) satisfies:

[0104]

[0105] Proof: Overlapping graph Let subscriptions be represented as nodes and overlaps as edges, where each edge exists independently with a probability of:

[0106]

[0107] By Lemma 1 and classical random graph theory, the maximum PDS is:

[0108]

[0109] Therefore, the coverage rate is:

[0110]

[0111] Theorem 2 Let for On each attribute Each subscription set. For each attribute ,make For the relevant number of subscribers, Let be the probability of paired predicate overlap. Assume . distribute The total number of disjoint predicate sets (PDS) is 1. And their attributes are independent of each other. Therefore, subsets The combined coverage on the above satisfies:

[0112]

[0113] Proof: For a single attribute The expected probability of a PDS is Therefore, a subscription was not... The probability of each independent PDS being covered is:

[0114]

[0115] Assuming independence between attributes, subsets The combined coverage is the product of the individual coverage rates:

[0116]

[0117] Therefore, when Then we have:

[0118]

[0119] This formula is consistent with the asymptotic behavior of independent sets in sparse random graphs.

[0120] Building upon the previous discussion of Theorem 2 of the coverage model and the subscription partitioning strategy, this paper further focuses on determining the optimal configuration parameters of the learning index to achieve a balance between performance and resources. In learning-based event matching systems, a core challenge lies in how to strategically partition the subscription set. Each predicate disjoint set (PDS) is independently managed by its respective learning index to balance inference overhead with coverage efficiency. Increasing k can improve the overall coverage of the aggregation model and reduce the dependence of uncovered subscriptions on slower fallback algorithms (such as OEM), but it also introduces four interdependent complexities: First, inference latency and verification costs increase approximately linearly because each event requires querying the entire dataset. First, the model is tested and its output is verified; second, the repeated storage of model parameters reduces memory overhead compared to... The following are some of the key factors affecting the performance of the matching algorithm: First, the coverage gains increase proportionally; second, the marginal effect of coverage gains diminishes, meaning the number of additional subscriptions that can be covered by newly added PDS decreases exponentially; and third, as the remaining uncovered load shrinks, the performance optimization space available for the fallback matching algorithm also decreases accordingly. To maximize system throughput and ensure the stability of the matching process within a limited memory and latency budget, this paper formally proposes a cost-benefit optimization framework. This framework analytically models the aforementioned nonlinear factors (especially the tension between the saturation of coverage gain and the increase in computational cost), thereby deriving the optimal number of PDS, k, that minimizes end-to-end matching latency. The mathematical expression of this optimization model is shown below:

[0121]

[0122]

[0123]

[0124]

[0125] To facilitate understanding, the objective function and constraints will be further explained below. First, The average end-to-end matching latency is measured and includes three latency overhead components. Firstly, This represents the latency of the "inference-verification" phase in the parallel pipeline: each event requires parallel execution of LIEM model inference on k PDS (the unit cost of each inference is...). ), and complete based on Dependency-aware verification of each attribute (the cost of checking each attribute is...) Therefore, this part follows approximately linearly. and Growth. Secondly, This is used to characterize the processing latency when there are remaining unmatched subscriptions entering the fallback matcher (e.g., the OEM algorithm), where Total number of subscribers, To adopt The percentage of subscriptions covered by the learning index (coverage) per PDS, therefore This is approximately the size of the subscriptions that need to be rolled back. This refers to the unit time cost of performing candidate retrieval or set matching for each uncovered subscription. Thirdly, This represents the compensation delay caused by the bounded lookup of the model's prediction error; since the learning index may contain some prediction error, it is necessary to... Each candidate subscription is used to perform an adaptive binary search to locate the truly matching subscription, with each search step taking [time]. Overall, it shows logarithmic growth.

[0126] Secondly, the constraints reflect the system's resource and performance requirements. Memory budget constraints. Require The total memory usage of each LIEM model plus the storage space required for the remaining uncovered subscription portion must not exceed the available memory limit. Coverage constraint The system is required to pass Each PDS must achieve at least the preset target coverage rate. This ensures that the vast majority of subscriptions are directly hit by the learning index, thereby significantly reducing the load and tail latency of the fallback path; subscription allocation constraints For all The requirement is to give the total Subscriptions are appropriately distributed to selected subsets of attributes. Above, and ensure that at least one is assigned to each attribute involved in modeling. Subscriptions are provided to offer sufficient training samples to support the statistical validity and generalization ability of the model's learning.

[0127] To ensure clarity in the use of symbols, key quantities in the model are now uniformly defined. Indicates the number of PDS. For the subset of attributes participating in the construction of the learning index, The subscription allocation for each attribute, where N is the total number of subscriptions. The total number of attributes. This represents the number of attributes involved in the validation. This represents the time cost of a single inference iteration of the LIEM model (typically measured to be approximately...). ), The time taken to validate a single attribute predicate (approximately) ), The time taken for a single step of the binary search (in typical cases, the total time taken for the error correction phase can be approximately fixed at 1) ), For fallback matching (such as candidate retrieval or OEM algorithms), the unit processing time for a single uncovered subscription is approximately ). The memory usage of index parameters for a single LIEM model. In a given The memory overhead used to maintain the remaining uncovered subscription data structures (such as candidate indexes, auxiliary bitmaps, or caches) is as follows. This represents the maximum available memory. To adopt Overall coverage of the learning index per PDS This is the upper bound on the number of candidate subscriptions to be considered in the error-bounded search, where It is the standard deviation of the prediction error of the learning model based on k PDS, by limiting... The maximum value of 64 ensures that the worst-case cost of this search phase is subject to a strict upper bound.

[0128] In summary, the optimization model described above aims to minimize end-to-end matching latency. This is reflected in the non-linear trade-off between the saturation of coverage benefits and the rising computation / memory costs: on the one hand, increasing... It can improve coverage On the one hand, it reduces the matching overhead of the remaining load; on the other hand, it linearly increases the time cost of model inference and attribute verification as well as the model memory usage, while the logarithmic search cost introduced by error correction and the memory overhead of the remaining matching structure. Also with Closely related and intertwined. Specifically, decision variables Attribute subsets and the number of subscriptions allocated to each attribute Together they determine the system's latency and coverage characteristics.

[0129] Because the covering function exhibits submodularity and nonlinearity, this optimization problem is inherently nonconvex and computationally challenging. Therefore, this invention employs a dynamic programming approach combined with approximate heuristics, leveraging the submodularity and monotonicity of the covering function to efficiently search for an approximate optimal solution. The overall process is described below. Figure 2 The process includes: calculation The feasible bound is determined, the coverage probability is evaluated by attribute, and the optimal subscription allocation scheme on the selected attributes is systematically determined through dynamic programming. Finally, the optimal solution is obtained by solving the problem in this model. Attribute subsets and subscription distribution scheme This approach can minimize end-to-end matching latency under given resource constraints and coverage targets, thereby improving the overall system throughput and stability. Following the modeling and analysis, subsequent work will present a solution strategy and implementation algorithm for LIEM system parameter configuration based on this optimization framework, and thereby transform the theoretically optimal solution into a deployable system design.

[0130] To ensure completeness and robustness, the remainder set uses OEM as a fallback mechanism: it is based on bitset / block-level parallelism, performs batch labeling through constant bitwise OR operations, and is accelerated on SIMD, making it suitable for handling difficult example subscriptions with high overlap and complex predicate combinations. This hybrid system is scheduled by OPDS: the learning index inference and lightweight verification of the coverage part are performed in parallel within each PDS, and OEM is called for the remainder set, finally aggregating and outputting the complete matching set.

[0131] This invention constructs a Predicate-Disjoint Set (PDS) on the main attributes and uses an Optimized Partition Model (OPDS) to quantitatively balance inference coverage, candidate verification, and cache usage, achieving mutual exclusion and learnability of the subscription set. Within each PDS, a Range-Query Recursive Model Index (RQ-RMI) is introduced, using a lightweight neural network to quickly locate the event attribute value to the subscription index segment. The maximum prediction error is evaluated at trigger points and boundaries, and limited corrections ensure no missed detections and verifiability. Simultaneously, for "remainder set" subscriptions that are difficult to mutually exclude, bit-set-driven precise backoff matching (such as OEM) is integrated, achieving efficient aggregation with constant bitwise operations. Thus, while ensuring correctness and completeness, the main matching time consumption is transformed from main memory access to vectorizable inference computation, significantly reducing average latency and suppressing long-tail latency, meeting the engineering stability requirements of large-scale, high-dimensional, and real-time CPS scenarios.

[0132] Example 2

[0133] This invention provides a learning-based interval event matching method for content-based publish / subscribe systems, comprising:

[0134] I. Matching Semantics

[0135] Content space consists of attribute sets Composition, event ,subscription For the conjunction of interval predicates on an attribute, the predicate ,width The matching relationship is defined as follows:

[0136]

[0137] If a PDS is constructed on a certain main attribute, then any two subscription intervals on that attribute are mutually exclusive, ensuring that the one-dimensional mapping can be learned as a single-valued function and eliminating training label conflicts.

[0138] II. OPDS: Predicate Disjointness Partitioning and Coverage Model

[0139] Based on Lemma 1, Lemma 2, and Theorem 1, we derive the maximum PDS coverage for a single attribute. And obtain the joint coverage under the condition of multi-attribute independence:

[0140]

[0141] It can be proven about The overhead is monotonically increasing, and the marginal benefit of adding new attributes diminishes. Considering the linear growth of multi-model inference and validation costs, the diminishing marginal benefit of coverage, and the geometric decrease in the size of the remainder set, this invention constructs the following optimization: The mathematical expression of this optimization model is shown below:

[0142]

[0143]

[0144]

[0145]

[0146] Algorithm Implementation: Due to the submodularity and nonlinearity of the covering function, this optimization problem is inherently nonconvex and computationally challenging. Therefore, this paper employs a dynamic programming approach combined with an approximate heuristic, leveraging the submodularity and monotonicity of the covering function to efficiently search for an approximate optimal solution. The overall process is described in the algorithm. Figure 2 The process includes: calculation The feasible bound is determined, the coverage probability is evaluated by attribute, and the optimal subscription allocation scheme on the selected attributes is systematically determined through dynamic programming. Finally, the optimal solution is obtained by solving the problem in this model. Attribute subsets and subscription distribution scheme This approach can minimize end-to-end matching latency under given resource constraints and coverage targets, thereby improving the overall system throughput and stability. Following the modeling and analysis in this section, subsequent sections will present a solution strategy and implementation algorithm for LIEM system parameter configuration based on this optimization framework, and thereby transform the theoretical optimal solution into a deployable system design.

[0147] III. Learning-Scope Index RQ-RMI: Learning-Scope Index within PDS

[0148] 1) Sub-model structure: Three-layer fully connected network (1-16-1), ReLU activation; the original output is truncated and mapped to [0,1] as the input for the next sub-model selection; each layer has a small number of parameters and low latency, suitable for L2 / L3 resident.

[0149] 2) Boundary analysis and responsibility domain division: The trigger point (segment boundary) and stage switching point (quantization switching boundary) are calculated analytically using the piecewise linear property of the sub-model. Evaluation is only performed at a limited number of points, which can accurately determine the responsibility domain of the next layer model and avoid full domain scanning.

[0150] 3) Error bound and finite correction: Calculate the maximum prediction error ε at the leaf layer based on the trigger point and subscription boundary to ensure that the true position is located in [prediction-ε, prediction+ε]. Online, only a secondary search and verification of this small window is needed to achieve the verifiable range matching of "prediction + finite correction".

[0151] IV. OEM: Precise rollback matching of the remainder

[0152] 1) Bitset caching mechanism (SCM): Discretizes the attribute value range into units and groups them. Offline, pre-computes the subscription bitset for each group that will be "invalidated by a certain range of event values". Online matching only requires constant bitwise OR operations and comparison with a small number of supplementary operations to complete the large-scale elimination.

[0153] 2) Parallelism and SIMD friendly: Bit set operations are naturally adapted to vector instructions, with low synchronization overhead between threads, making it suitable as a steady-state "backup engine" for LIEM.

[0154] V. Matching Process and Pseudocode ( Figure 3 )

[0155] Step S1 (Parallel Learning Index Query): Perform RQ-RMI forward inference in parallel for each PDS to obtain candidate positions and... Perform local search and verification within the window, and add the matched results to the candidate result set;

[0156] Step S2 (OEM Marking): Find the corresponding low / high boundary bucket cells for each attribute value of the event, merge the pre-computed bit sets bit by bit and add the marked uncovered cells to obtain the candidate matching bit set;

[0157] Step S3 (Result Aggregation): Merge the matching subscriptions of S1 and S2, and output the complete matching set. The above process can be easily implemented in a pipelined parallel environment.

[0158] VI. Implementation Key Points and Complexity

[0159] 1) Cache residency and parameter layout: RQ-RMI sub-models and OEM bit sets are preferentially laid out to L2 / L3 to reduce DRAM access; model parameters are organized according to contiguous memory blocks and alignment strategies to improve vectorization efficiency.

[0160] 2) Time complexity: Learning index lookup and finite correction are in constant time; OEM bit set operations are O(n / W) (W is the word length), and the final linear scan bit set is O(n). Under typical coverage, the fallback size is small, and the overall performance is scalable with low constant time complexity.

[0161] VII. Effect Verification

[0162] The experimental platform used Ubuntu 18.04, with an AMD R9 5900X CPU and 32GB of memory. The algorithm was implemented in C++ (-O3). The workload included synthetic and real datasets, systematically examining factors such as the number of subscriptions n, the number of attributes d, the subscription length k, the predicate width w, the event size m, and the Zipf skew parameter α. Experimental results show that LIEM significantly outperforms REIN, Ada-REIN, WPA-REIN, OEM, and TAMA under different scales and distributions: the highest reduction in average matching latency was 88.51%, 86.05%, 83.30%, 39.71%, and 83.93%, respectively; the standard deviation of matching time was reduced by approximately 49.15%, effectively suppressing the long-tail effect, demonstrating the steady-state advantages and engineering deployability of "model inference-driven + OEM fallback".

[0163] In summary, this invention, through the collaborative design of "OPDS optimized partitioning + RQ-RMI learning index + OEM precise rollback", significantly reduces average latency and variance while ensuring full matching correctness and verifiability, suppresses long-tail latency, and achieves cache-friendly and high-throughput event matching in general CPU scenarios with a quantifiable coverage-cost trade-off. It is suitable for real-time CPS scenarios such as IoT, financial distribution, online recommendation and advertising.

[0164] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0165] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for interval event matching in a content-based publish / subscribe system, characterized in that, include: Step 1: Receive event streams from external data sources through the system's input interface. The events are structured data with multiple attribute value pairs. Step 2: By optimizing the predicate disjoint set partitioning OPDS engine, the subscription set stored in the system memory is divided into k predicate disjoint subsets PDS according to one or more selected main attributes. In each of the predicate disjoint subsets, the interval predicates of any two subscriptions on the main attribute do not overlap. The value of k is determined by solving a constraint optimization model with the goal of minimizing the total matching delay. Step 3: For each predicate disjoint subset, call its corresponding pre-trained learned interval indexing model RQ-RMI, use the attribute value of the event as input for forward inference, and output the index position prediction value of the candidate subscription; based on the prediction value and the pre-calculated maximum prediction error ε, determine a correction window in the index position sequence; perform local retrieval and predicate verification within the correction window to obtain the first matching subscription set; Step 4: For the subscription remainder set not covered by any of the predicate disjoint subsets, call the exact backoff matching engine OEM, and by searching the pre-computed bit set cache, perform bitwise logic operations to filter out the subscriptions that match the event, and obtain the second matching subscription set; Step 5: Merge the first matching subscription set and the second matching subscription set using the result aggregator to generate the final full matching subscription set, and distribute it to the corresponding subscribers through the system's output interface; The partitioning of the OPDS engine is based on a coverage model, where for attributes... The probability that two subscription intervals do not overlap is: in, The cumulative distribution function representing the standard normal distribution; It is the expected width of the predicate interval; It is the variance of the predicate interval width; It is the variance of the predicate interval center; The coverage model includes the maximum PDS coverage of a single attribute. The calculation is as follows: in, It is an attribute Subscription count; It is the probability of paired predicate overlap; It is a quantity of asymptotically lower order than 1; For cases with multiple attributes, attribute subsets The combined coverage on the above satisfies: in, It is the total number of PDS. It is the index sequence of PDS. It is an attribute The number of PDS on the device It is an attribute The probability of pairwise predicate overlap; The OPDS engine determines the optimal number of PDS k, attribute subset J, and subscription allocation by solving an optimization model. The objective function of the optimization model is: in, It is the total matching delay. This refers to the inference time per iteration of the LIEM model. It is the validation time for a single attribute. N is the number of attributes participating in the validation, and N is the total number of subscriptions. It's about coverage. It is a rollback to the matching unit time. It is the error boundary. It is for finding the single-step time. This refers to the memory usage of a single LIEM model. It is the memory overhead of the remainder set. It is the memory limit. It is the target coverage rate. It is the minimum subscription allocation; The optimization model is solved using dynamic programming combined with an approximate heuristic method, which utilizes the submodularity and monotonicity of the covering function to search for an approximate optimal solution.

2. The interval event matching method for a content-based publish / subscribe system according to claim 1, characterized in that, The learned interval index RQ-RMI includes a three-layer fully connected neural network with a ReLU activation function. The first layer takes event attribute values ​​as input and its output is truncated and mapped to the range [0,1] as the input for the next sub-model selection. The leaf layer outputs the predicted subscription position. The RQ-RMI calculates the maximum prediction error at the leaf layer. To ensure that the actual subscription location is within the range Within this range, a secondary retrieval and verification are performed through finite correction, where prediction is the direct output of the model's inference.

3. The interval event matching method for a content-based publish / subscribe system according to claim 1, characterized in that, The precise backoff matching OEM uses a bit set caching mechanism (SCM), in which the attribute value domain is discretized into multiple units and grouped. The invalid subscription bit set corresponding to each group is pre-calculated offline. During online matching, the bit sets are merged by bitwise OR operation and the uncovered units are supplemented with markers to obtain the candidate matching bit set. The bitset operation of the OEM supports single-instruction multiple-data (SIMD) parallel processing for efficient matching of remainder subscriptions.

4. A range event matching system for a content-based publish / subscribe system, characterized in that, include: Module M1: Receives event streams from external data sources through the system's input interface. The events are structured data with multiple attribute value pairs. Module M2: By optimizing the predicate disjoint set partitioning OPDS engine, the subscription set stored in the system memory is divided into k predicate disjoint subsets PDS according to one or more selected main attributes. In each of the predicate disjoint subsets, the interval predicates of any two subscriptions on the main attribute do not overlap. The value of k is determined by solving a constraint optimization model with the goal of minimizing the total matching delay. Module M3: For each predicate disjoint subset, call its corresponding pre-trained learned interval indexing model RQ-RMI, use the attribute values ​​of the event as input for forward inference, and output the predicted index position value of the candidate subscription; based on the predicted value and the pre-calculated maximum prediction error ε, determine a correction window in the index position sequence; perform local retrieval and predicate verification within the correction window to obtain the first matching subscription set; Module M4: For subscription remainder sets not covered by any of the predicate disjoint subsets, the exact backoff matching engine OEM is invoked. By searching the pre-computed bit set cache, bitwise logic operations are performed to filter out subscriptions that match the event, thus obtaining the second matching subscription set. Module M5: Merges the first matching subscription set and the second matching subscription set through the result aggregator to generate the final full matching subscription set, and distributes it to the corresponding subscribers through the system's output interface; The partitioning of the OPDS engine is based on a coverage model, where for attributes... The probability that two subscription intervals do not overlap is: in, The cumulative distribution function representing the standard normal distribution; It is the expected width of the predicate interval; It is the variance of the predicate interval width; It is the variance of the predicate interval center; The coverage model includes the maximum PDS coverage of a single attribute. The calculation is as follows: in, It is an attribute Subscription count; It is the probability of paired predicate overlap; It is a quantity of asymptotically lower order than 1; For cases with multiple attributes, attribute subsets The combined coverage on the above satisfies: in, It is the total number of PDS. It is the index sequence of PDS. It is an attribute The number of PDS on the device It is an attribute The probability of pairwise predicate overlap; The OPDS engine determines the optimal number of PDS k, attribute subset J, and subscription allocation by solving an optimization model. The objective function of the optimization model is: in, It is the total matching delay. This refers to the inference time per iteration of the LIEM model. It is the validation time for a single attribute. N is the number of attributes participating in the validation, and N is the total number of subscriptions. It's about coverage. It is a rollback to the matching unit time. It is the error boundary. It is for finding the single-step time. This refers to the memory usage of a single LIEM model. It is the memory overhead of the remainder set. It's the memory limit. It is the target coverage rate. It is the minimum subscription allocation; The optimization model is solved using dynamic programming combined with an approximate heuristic method, which utilizes the submodularity and monotonicity of the covering function to search for an approximate optimal solution.

5. The interval event matching system for a content-based publish / subscribe system according to claim 4, characterized in that, The learned interval index RQ-RMI includes a three-layer fully connected neural network with a ReLU activation function. The first layer takes event attribute values ​​as input and its output is truncated and mapped to the range [0,1] as the input for the next sub-model selection. The leaf layer outputs the predicted subscription position. The RQ-RMI calculates the maximum prediction error at the leaf layer. To ensure that the actual subscription location is within the range Within this range, a secondary retrieval and verification are performed through finite correction, where prediction is the direct output of the model's inference.

6. The interval event matching system for a content-based publish / subscribe system according to claim 4, characterized in that, The precise backoff matching OEM uses a bit set caching mechanism (SCM), in which the attribute value domain is discretized into multiple units and grouped. The invalid subscription bit set corresponding to each group is pre-calculated offline. During online matching, the bit sets are merged by bitwise OR operation and the uncovered units are supplemented with markers to obtain the candidate matching bit set. The bitset operation of the OEM supports single-instruction multiple-data (SIMD) parallel processing for efficient matching of remainder subscriptions.

Images