A distributed complex event processing system using hyper-heuristic algorithm

Abstract

The application discloses a kind of distributed complex event processing systems using super heuristic algorithm, belong to distributed computer system technical field.The system includes cluster control component, distributed database, message middleware, event collection and analysis middleware, event distribution middleware and CEP processing node.It is in at core, by event collection and analysis middleware call super heuristic algorithm, CEP rule and historical data are trained, and optimal event fragment division method and Gower distance model based on attribute weight are automatically obtained;At runtime, event distribution middleware is based on the distance model to calculate the similarity of new event and existing partition, and in combination with the real-time load state of CEP processing node, dynamic, reasonable load distribution and balance are realized.The application realizes the intelligent adaptation and zero business invasion of CEP processing distribution, significantly improves the adaptability, universality and resource utilization of system in dynamic scene.

Description

Technical Field

[0001] This invention belongs to the field of distributed computer system technology, specifically relating to a distributed complex event processing system employing a hyperheuristic algorithm. Background Technology

[0002] Complex Event Processing (CEP) is a technology for real-time analysis of massive heterogeneous event streams and identification of related complex events, widely used in fields such as financial risk control. Current distributed CEP implementations fall into three categories, all of which have significant limitations and are ill-suited to the efficient processing requirements of high-frequency, heterogeneous, and dynamic scenarios, as detailed below: CEP engines based on streaming processing frameworks, such as Flink CEP, rely on the Flink streaming processing framework. They track matching progress through an NFA state machine, and their distributed operation and fault tolerance depend on Flink's KeyedState state management and Checkpoint mechanism. Multi-node collaboration relies on Flink's master-slave architecture. This approach suffers from poor versatility, weak load balancing, and the potential for cross-machine transmission to become a performance bottleneck.

[0003] Distributed CEP systems based on the publish / subscribe model, such as patents CN104572859B and CN121255928A, respectively rely on the JMS specification and ActiveMQ, adapting to scenarios through topics or dynamic contract mechanisms to achieve event distribution, matching, and processing. Their limitations lie in their reliance on manual pre-configuration, low level of automation, imperfect load balancing and fault tolerance mechanisms, and difficulty in adapting to high-frequency fluctuation scenarios.

[0004] The processing method based on distributed databases and transaction queries involves storing event data in distributed databases such as HBase and MongoDB, relying on transaction retrieval and matching, and simulating a distributed effect through multiple query initiating nodes. HBase faces design difficulties due to its reliance on RowKey storage for queries, while MongoDB has limited support for join queries. Both suffer from high transmission and storage costs, insufficient real-time performance, difficulty in ensuring consistency, and strong transaction intrusion, resulting in poor adaptability.

[0005] In summary, these three technologies share common bottlenecks that restrict their large-scale application in complex and dynamic scenarios. All three lack dynamic adaptive capabilities, relying on manual pre-configuration, making them ill-suited to handle changes in events, nodes, and traffic; resource scheduling optimization is insufficient, with coarse load balancing strategies leading to low resource utilization and unstable processing latency; the contradiction between versatility and domain dependence is prominent, with high customization requirements and high costs for cross-scenario migration; and they lack intelligent optimization capabilities, making it difficult to improve operational performance at the core level. Summary of the Invention

[0006] This invention addresses the aforementioned problems and overcomes the shortcomings of existing technologies by providing a distributed complex event processing system employing a hyperheuristic algorithm. It aims to solve the problems of poor dynamic adaptability, unreasonable resource scheduling, low versatility, high resource requirements, and strong business intrusion in existing distributed CEP technologies. The system is suitable for IoT and large-scale industrial control systems, achieving real-time, high-concurrency, highly automated, and low-resource-consumption distributed complex event processing.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] A distributed complex event processing system employing a hyperheuristic algorithm, deployed on a containerized cluster, includes: a cluster control component, a distributed database, a message middleware, an event acquisition and analysis middleware, an event distribution middleware, and CEP processing nodes; The event acquisition and analysis middleware, in response to the creation of new CEP rules, executes the following training process: parsing the rules to obtain event attributes; collecting raw event data; calling a hyperheuristic algorithm to segment the raw event data to determine the subset of key attributes used to distinguish the segments to which events belong and the segmentation results; and calling a hyperheuristic algorithm to perform partition combination solution to obtain a weighted Gower distance model based on event attributes for calculating the similarity between events. The distributed database is used to store the fragment segmentation results, attribute clustering results, and the weighted Gower distance model; The event distribution middleware, deployed when the rules are online, includes a Leader node elected through election. The Leader node is used to: identify the segment to which a newly arrived original event belongs based on the segmentation results; calculate the distance between the segment and existing core segments of the partition based on the weighted Gower distance model; and perform dynamic load allocation based on the distance calculation results and the real-time load information of each CEP processing node obtained from the distributed database to determine the target CEP processing node for processing the event. The CEP processing node integrates a CEP engine instance and an event receiving middleware; the event receiving middleware is used to proxy the CEP engine instance to receive events and send complex events, and to report the resource usage status of this node as the real-time load information to the distributed database.

[0009] Specifically, the hyperheuristic algorithm, based on the events, attributes, and actual data collected by CEP rule parsing, controls heuristic operators to solve the problem of segment partitioning and partition combination, thereby obtaining a method for event segment partitioning and partition combination.

[0010] As a preferred embodiment of the present invention, the event acquisition and analysis middleware specifically includes: The rule analysis module is used to parse CEP rule scripts and extract event names, attribute lists, and rule backbone content. The data acquisition module is used to collect a specified amount or duration of raw event data from the message middleware; The segmentation module is used to iteratively solve the problem by calling a hyperheuristic algorithm based on the associated attributes in the core content of the rules and the original event data, so as to select the optimal attribute subset for segmentation, and use an objective function to quantitatively evaluate the accuracy of the segmentation results, the number of segments and the computational cost. The partition combination module is used to combine the original event data, call the hyperheuristic algorithm to perform iterative solution to obtain the optimal attribute weights in the weighted Gower distance model, and use the objective function to quantitatively evaluate the correctness of the partition combination result, the number of partitions, the partition cohesion and the computational cost.

[0011] Specifically, in the fragment partitioning problem, the hyperheuristic algorithm reduces the dimensionality of event attributes, extracts the optimal set of attributes for determining the fragment to which an event belongs, evaluates the accuracy, number of fragments, and computational cost by the objective function, and uses the results for the partitioning and combination problem.

[0012] Specifically, in the partitioning and combining problem, a hyperheuristic algorithm adjusts the event attribute weights and combines them by fusing multi-dimensional features of the fragments using the Gower distance of these weights. This ensures that the complex events triggered by the combined partitions through CEP rule matching are consistent with those before partitioning. The objective function evaluates accuracy, number of partitions, partition cohesion, and computational cost to obtain the optimal attribute weights for load balancing. The distance-based load balancing method first determines the fragment to which an event belongs based on the fragment partitioning results, then calculates its distance to existing partitions using the attribute weights obtained from partitioning and combining, thereby determining the load distribution nodes. New partitions are preferentially allocated to nodes with sufficient resources and holding similar partitions to maximize the aggregation of related events and reduce redundant transmission. When node resources are insufficient, they are allocated to other nodes or expansion is triggered to avoid single-point overload and ensure processing stability.

[0013] As another preferred embodiment of the present invention, the dynamic load balancing performed by the Leader node of the event distribution middleware includes the following three logics: If the nearest existing partition's corresponding CEP processing node has sufficient resources, the event will be assigned to that node; If the nearest existing partition has insufficient CEP processing node resources, other nodes with sufficient resources will be searched in order of distance from nearest to farthest as the target node. If all CEP processing nodes are insufficient in resources, a request for expansion is made to the cluster control component, and new partitions and CEP processing nodes are created.

[0014] As another preferred embodiment of the present invention, the Leader node of the event distribution middleware is also used to perform load redistribution, including: Dynamic reallocation: Periodically check the load of each CEP processing node, calculate the load difference, and reallocate some event fragments of high-load nodes according to their distance from the core fragments of low-load nodes. Complete reallocation: Based on the fragment partitioning results and the current number of CEP processing nodes, reinitialize all partition allocation relationships.

[0015] As another preferred embodiment of the present invention, the event receiving middleware in the CEP processing node modifies the source annotations and sink annotations in the CEP rule script to HTTP types pointing to the local port to proxy the communication between the CEP engine instance and the message middleware, thereby achieving zero-intrusion integration of CEP business processing logic.

[0016] As another preferred embodiment of the present invention, the distance calculation in the weighted Gower distance model is performed by pairing attributes of different event types based on the attribute clustering results and assigning weights to each attribute pair; the distance value is calculated and weighted according to the attribute data type using Hamming distance, normalized Euclidean distance or string edit distance.

[0017] As another preferred embodiment of the present invention, the hyperheuristic algorithm used in the segmentation module is based on the genetic algorithm framework, integrates crossover and mutation mechanisms, and calls various heuristic operators, including simulated annealing, genetic algorithm, tabu search, differential evolution, artificial bee colony, ant colony optimization, particle swarm optimization, gray wolf optimization and whale optimization, for iterative optimization.

[0018] As another preferred embodiment of the present invention, the hyperheuristic algorithm used in the partitioning combination module is based on a genetic algorithm framework, integrates crossover and mutation mechanisms, and calls various heuristic operators, including tabu search, ant colony optimization, particle swarm optimization, differential evolution, artificial bee colony, gray wolf optimization, whale optimization, sine and cosine algorithm, butterfly optimization algorithm and sparrow search algorithm, for iterative optimization; the correctness of its solution is verified by co-training independent CEP engine instances, and evaluated by comprehensively considering the trigger matching rate, trigger coverage rate and false trigger rate.

[0019] As another preferred embodiment of the present invention, the cluster control component is used to manage the entire lifecycle of CEP rules, the lifecycle including: creation, collection and analysis, online, operation, offline and deletion stages; the cluster control component controls the deployment and destruction of corresponding middleware according to the lifecycle stage.

[0020] As another preferred embodiment of the present invention, the containerized cluster is a Kubernetes cluster; the distributed database is Redis; the message middleware is RabbitMQ; and the CEP engine instance is a Siddhi instance.

[0021] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention provides a distributed complex event processing system employing a hyperheuristic algorithm. It analyzes the load distribution of CEP rules using a "training-first, analysis-second" approach, combined with a load balancing method based on weighted Gower distance calculation. Deploying a distributed CEP system on a Kubernetes cluster, the system's components collaboratively manage and run CEP rules throughout their entire lifecycle. The system uses a hyperheuristic algorithm to identify the event segment; it determines the partition to which the segment belongs and performs load balancing based on the partition; it uses a weighted Gower distance calculation method to achieve dynamic load allocation and rebalancing; and it uses a proxy CEP middleware for event sending and receiving, enabling a distributed transformation of the CEP process with zero business intrusion. Specifically, the system offers the following advantages: (1) Significantly improved dynamic adaptability and versatility: Relying on intelligent algorithms, it can autonomously adapt to changes in business rules and operating environment without human intervention, solving the pain point of low automation and adapting to complex dynamic scenarios; (2) Significantly improved usability: Automatically adapts to various CEP rules through fitting training, reducing the user's requirements for mastering business details and data distribution, and adapting to the needs of multiple fields; (3) Significantly reduced core resource requirements: Intelligent distribution based on partition distance reduces redundant transmission, and aggregated allocation of associated events reduces redundant storage across nodes, resulting in significant optimization of bandwidth and memory overhead; (4) The load balancing mechanism is more reasonable: load distribution and redistribution are based on partition distance, replacing round-robin or hashing methods, reducing the probability of unreasonable distribution and transfer; (5) CEP business is non-intrusive and flexible: the distributed function is independent of the CEP middleware, and can be compatible with various mature middlewares without modification, realizing business decoupling and reducing development difficulty.

[0022] (6) The present invention realizes intelligent adaptation and zero business intrusion of distributed CEP processing, which significantly improves the system’s adaptability, versatility and resource utilization in dynamic scenarios. Attached Figure Description

[0023] Figure 1 This is a schematic diagram of the overall architecture of the distributed complex event processing system of the present invention; Figure 2 This is a flowchart of the rule lifecycle management process in this invention; Figure 3 This is a schematic diagram illustrating the rule creation, collection, and analysis stages in this invention; Figure 4 This is a flowchart illustrating the training process of the event acquisition and analysis middleware of this invention. Figure 5 This is a schematic diagram of the online and operational phases of the rules in this invention; Figure 6 This is a schematic diagram of the load balancing mechanism based on partition distance according to the present invention; Figure 7 This is a data flow diagram for the distributed complex event processing system of the present invention during actual operation. Detailed Implementation

[0024] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0025] like Figure 1 As shown, the distributed complex event processing system provided in this embodiment of the invention is deployed on a Kubernetes cluster, with various middleware and CEP processing nodes deployed in a containerized manner, connecting the original event input end and the complex event output end, including a cluster control component, a distributed NoSQL database, a message middleware, an event collection and analysis middleware, an event distribution middleware, and CEP processing nodes.

[0026] Cluster control component: Allows users to perform CEP rule creation, deletion, modification and query, start and stop the data collection and analysis process, rule online and offline and operation monitoring, and deploy or destroy middleware instances. It includes rule management, operation status monitoring and cluster basic information management modules.

[0027] Distributed NoSQL database: To ensure the consistency of component state synchronization, Redis is selected in this embodiment of the invention.

[0028] Message middleware: Enables internal and external communication and message passing between components. In this embodiment of the invention, RabbitMQ is selected, with 3 switches, which are respectively connected to the original event input end, the complex event output end, and the event distribution and receiving middleware message passing.

[0029] Event collection and analysis middleware: One-to-one correspondence with rules, through each module to perform rule parsing, data collection, fragment segmentation, attribute clustering, partition combination operations, generate core results to support high-concurrency processing.

[0030] Event distribution middleware: One rule corresponds to one group, containing one Leader and several Workers. The Leader allocates event fragments to CEP processing nodes according to the distance between the segments, and requests expansion when necessary; all nodes receive RabbitMQ events and deliver them to the corresponding CEP processing nodes according to the allocation results.

[0031] CEP processing nodes: Several nodes correspond to one rule. Each node contains one Siddhi instance and one event receiving middleware. The Siddhi instance is responsible for CEP rule matching.

[0032] Event receiving middleware: proxies the sending and receiving of events for Siddhi instances, records throughput and trigger count, and periodically synchronizes node resource usage to Redis for use by other components.

[0033] Figure 2 The rule lifecycle is shown in six stages and the transformation logic: After the rule is created, it enters the collection and analysis stage, and then goes online; after going online, the components are deployed and started to run, and load monitoring and redistribution are performed synchronously; after running, the rule is taken offline, and the offline rule can be re-brought online or deleted. After deletion, all data is cleaned up.

[0034] Figure 3 Display rule creation and data collection / analysis process: During the creation phase, the cluster control component saves the rule information and returns the results. During the data collection and analysis phase, the cluster control component deploys an event collection and distribution middleware instance, which sequentially performs rule analysis, data collection, fragment partitioning, attribute clustering, training to solve the partition combination problem, saves the results, and returns them. See the detailed process below. Figure 4 .

[0035] Figure 4 To refine the training process of the event acquisition and analysis middleware and improve the two-layer optimization logic of "segment division - partition combination": Semantic analysis parses key information of rules: The rule analysis module parses the script, matches "@Source" and "@Sink" annotations to extract the source / destination of events, matches "define stream" statements to extract the event name and attribute list, matches "from** select ** insert into **" statements to extract the core content, and stores the results.

[0036] Receive and save raw message training data: Based on the parsing results, the data acquisition module reads raw event messages from the specified RabbitMQ queue, collects them until the duration or data volume requirements are met, and temporarily stores them in a local file after JSON serialization.

[0037] Based on all associated attributes, obtain the correct segment results: extract all associated attributes of the core content, divide and temporarily store the original events, group those with the same value in the entire set of associated attributes into the same segment, and record the correct results.

[0038] The hyperheuristic algorithm performs fragment partitioning: the fragment partitioning module performs fragment partitioning based on the association attributes of the rule backbone content and collected data. The core objective is to obtain the optimal attribute subset and fragment partitioning result. The hyperheuristic algorithm is based on a genetic algorithm framework, incorporating crossover and mutation mechanisms to iteratively optimize three dimensions: solution selection method, operator selection, and operator iteration time. It calls various heuristic operators to generate an initial population, iteratively optimizes the attribute subset, calculates the solution quality using the objective function, and solves the fragment partitioning problem. The algorithms used as operators include: simulated annealing (with added population mechanism), genetics, tabu search, differential evolution, artificial bee colony, ant colony optimization (with added dynamic distance matrix), particle swarm optimization (with added attribute approximation mechanism), gray wolf optimization, and whale optimization.

[0039] Quantitative Evaluation of the Fragment Partition Objective Function: The objective function used in this problem simultaneously examines three dimensions: fragment partitioning accuracy, fragment quantity, and computational cost. The calculation formula is as follows:

[0040] In the formula: GTF This represents the correct segmentation result; D It is a collection of event attributes of the rule; phi This is the current partitioning method, which uses D Several attributes; n It represents the number of segments divided by φ; p i It is a fragment i The precision; r i It is a fragment i The recall rate; m yes D The length of the dimension list, i.e., how many dimensions are in the dimension list; omega ( d j )yes D In a certain dimension j The computational cost is calculated using the following formula;

[0041] α yes phi The weighting coefficients for the correctness of the partitioning results β yes phi The weighting coefficients for the number of segments obtained from the partitioning. gamma yesphi The weighting coefficients for calculating the cost, and the values ​​of these three coefficients. α > β > gamma In this embodiment, we take α =9, β= 2, gamma =1, such a hyperparameter choice can ensure correctness while making phi It results in more segments using fewer, less costly dimensions. The objective function ranges from (1, +∞), with smaller values ​​yielding better results.

[0042] Attribute clustering is performed: Based on the statistical characteristics of the collected data, event attributes are grouped by data type, excluding pairs with the same event attribute, and similarity is calculated. Attributes of different types are considered dissimilar; Boolean similarity is calculated by combining absolute error, relative error, and JS similarity; normally distributed numerical data are tested using the Welch-T test, and non-normal data are tested using the Mann-Whitney-U test; string similar data are clustered using a Bloom filter, combining overlap rate, fill rate, and multiple similarity coefficients. Clusters are generated using DBSCAN, and the results are saved to Redis.

[0043] The hyperheuristic algorithm performs partition combinatorial optimization: The partition combinatorial module uses a hyperheuristic algorithm to solve the partition combinatorial problem and obtain the optimal partition combination result. The hyperheuristic algorithm itself is based on a genetic algorithm framework, integrating crossover and mutation mechanisms, and iterative optimization across three dimensions: solution selection method, operator selection, and operator iteration time. It calls various heuristic operators to generate an initial population, iteratively optimizes attribute subsets, calculates solution quality using an objective function, and solves the partition combinatorial problem. The algorithms used in this problem include: tabu search, ant colony optimization, particle swarm optimization, differential evolution, artificial bee colony optimization, gray wolf optimization, whale optimization, sine and cosine optimization, butterfly optimization, and sparrow search.

[0044] The correctness of the current solution is obtained using the training component: the partition combination module works with the training Siddhi instance component to evaluate the correctness quality of the obtained solution. The correctness comprehensively considers the trigger matching rate, trigger coverage rate, and false trigger rate. In the final correctness, the weights of the three are set to 0.5, 0.25, and 0.25, respectively.

[0045] Quantitative Evaluation of the Partition Combination Objective Function: When evaluating the solution quality in the partition combination module, the objective function simultaneously considers four dimensions: combination correctness, number of partitions, partition cohesion, and computational cost. The quantitative evaluation is performed using the following formula:

[0046] In the formula: CTF This represents the correct result of a complex event triggering; D It is a collection of event attributes of the rule;π This is the current combined solution, which is the list of event attribute weights used to calculate the distance in the combination method; Outcome To determine the correctness of the current solution; n yes π The number of partitions resulting from the combination; Intra Group Similarity The mean similarity within a group represents the cohesion of the partitions, and its calculation formula is as follows:

[0047] D jk ( P , W (segment) j and fragments k The weighted Gower distance between them (which can also represent the original events) j and the original event k The distance between them is calculated using the following formula:

[0048] p for j and k The list of all attribute pairs that can be paired with the event type in the attribute clustering results. It is the set of all weights for all properties of all types of events. p A total of n For pairable attributes, retrieve The weights of each pair of attributes are averaged, and this... n After normalizing the weights, we get n The weights of attribute groups, where the attribute group αβ The weight is ,property α and β The actual value is a and b , d ab It is the distance between these two values. d ab The calculation method is as follows: α and β If both are Boolean types, then Hamming distance is used. α and β If both are numeric types, normalized Euclidean distance is used; otherwise, if both are string types, string edit distance is used.

[0049] omega ( d K The cost of an attribute is calculated using the following formula;

[0050] α , β , gamma and epsilon These are weighted coefficients for combinational correctness, number of partitions, partition cohesion, and computational cost, respectively. The values ​​of these three coefficients are... α > β > gamma > epsilon In this embodiment, we take α =100, β= 10, gamma =5, epsilon =1 (prioritizing the correctness of partitioning). The objective function takes values ​​in the range (1, +∞), with smaller values ​​yielding better results.

[0051] Store relevant information and return results: Obtain the optimal partition combination solution (including attribute list and attribute weight), store it in Redis, destroy all deployed Siddhi instances, the event collection and analysis middleware notifies the cluster control component that the rule is ready to go online, and ends the execution of the event collection and analysis middleware.

[0052] Figure 5 The process is shown in the rule deployment phase and the rule execution phase, including: Rule Deployment: This phase is initiated by the user. The cluster control component deploys event distribution middleware instances (the number is specified by the user) and CEP processing node instances (the initial number specified by the user), and then performs the following operations: After the event distribution middleware starts, a Leader is elected using a simplified version of the Raft algorithm (removing log replication and state machine synchronization mechanisms, relying on Redis to ensure consistency). The Leader selects core fragments of the partition based on the fragment partitioning, attribute clustering, and partition combination results in Redis, combined with the number of CEP nodes, and performs hierarchical sampling. It calculates the distance between all fragments and the core fragment (the calculation method is the same as the distance calculation method in the partition combination problem) and assigns them to the partitions. After associating the node number, it stores the information in Redis.

[0053] After the CEP processing node starts, the event receiving middleware opens port 10000 to receive Siddhi output, deploys the modified rule script for the Siddhi instance in the node (replacing @Source and @Sink with local HTTP type, and the port numbers are the corresponding port for the event and 10000 respectively), reads messages from RabbitMQ according to the node number and forwards them to the local port, completing the proxy sending and receiving.

[0054] Rule Execution: This phase begins after the deployment phase is completed. During execution, CEP processing nodes periodically upload their resource heartbeats to Redis. These heartbeats include CPU and memory usage, used for load balancing and redistribution. They also periodically update the number of raw events received and complex events triggered by the node in Redis to monitor the rule execution status. The event distribution middleware synchronizes the resource heartbeats of the CEP processing nodes in Redis as a basis during load balancing and redistribution. The cluster control component periodically collects the number of raw events received and complex events triggered by each node stored in Redis for display.

[0055] During rule execution, the three operations of load distribution, scaling up CEP processing nodes, and load redistribution rely on a partition distance-based load balancing mechanism, which is... Figure 6 Demonstration; Data flow during the runtime phase is from Figure 7 exhibit.

[0056] Figure 6 This fully demonstrates a load balancing mechanism based on partition distance, primarily focusing on load distribution functionality and the scaling logic for CEP processing nodes. It's a detailed extension of the load balancing mechanism shown in Figure 5 during the operational phase. All three states in the figure begin with the event distribution middleware reading the original event, identifying it as a new partition, and reporting it to the Leader node. The Leader's processing logic differs. Specifically: During normal operation, the CEP processing node corresponding to the partition closest to the new fragment has sufficient resources. After calculating the distance between the newly discovered fragment and the core fragments of all current partitions and sorting them from smallest to largest, the Leader finds the core fragment of the partition closest to the newly discovered fragment. If the CPU utilization and memory utilization of the CEP processing node corresponding to this partition are both below the user-defined high watermark, then the fragment is directly allocated to this partition, which becomes the target partition. In other words, the fragment is sent to the CEP processing node corresponding to this partition, and that node becomes the target node.

[0057] When no expansion is needed, it means that the CEP processing node corresponding to the partition closest to the new fragment is insufficient, but other CEP processing nodes have sufficient resources. After the aforementioned distance calculation and sorting, the Leader checks the CPU and memory status of the nodes corresponding to the partitions in the order of the list until it finds a partition with sufficient resources. This partition is the target partition, and the corresponding node is the target node.

[0058] When expansion is needed, it means that the resources of all CEP processing nodes belonging to this rule are insufficient. A new partition must be created, and the CEP processing nodes must be expanded. The new partition corresponds to the new node. After the aforementioned distance calculation and node resource check, the Leader fails to find the target partition and target node, and requests the cluster control component to expand the CEP processing nodes, creating a new partition. This partition is the target partition, the current segment is the core segment of the partition, and the newly created node after expansion is the target node.

[0059] After the Leader determines the target partition and node, it stores the results in Redis. All event distribution middleware then delivers events according to the results. If the user has set a maximum number of node instances, only normal operating logic will be executed after the limit is reached.

[0060] The load redistribution function addresses the issue that after a period of rule operation, the cumulative allocation of newly added unknown segments may lead to partition imbalance, resulting in load imbalance. This embodiment designs two load redistribution mechanisms, namely: Dynamic reallocation requires the user to enable this switch. The Leader will periodically check the CPU and memory usage of each CEP processing node, identify nodes with usage 20% above and below the average, sort them by load in descending and ascending order respectively, and then take the head node of each list as the migration-out node. h i and the migration node l i Calculate the load difference rate ; Sort the fragments of the migrating node from largest to smallest distance from the cluster center of the migrating node (total number of fragments is...). F ), calculate the number of migrations M =( D × F ) / 100, determine the list of migration segments, write the migration results to Redis, and the dynamic redistribution is completed.

[0061] A complete reassignment requires manual triggering by the user. The Leader reassigns partitions according to the initial allocation logic at startup, writes the allocation results to Redis, and the complete reassignment is finished.

[0062] Figure 7 This demonstrates the data flow during the actual operation of a distributed complex event processing system, covering the entire process from initial event input to complex event output, including load synchronization. The raw message is received, and the raw message data source writes the data to RabbitMQ switch 1. The original event stream is generated by the event distribution middleware, which retrieves packets from switch 1 and uses the snowflake algorithm to generate unique numbers. Partition allocation and raw event stream: The Leader node determines the fragment partitioning result, and each Worker forwards the message to the queue of the corresponding CEP processing node through RabbitMQ switch 3. The CEP node's event receiving middleware receives messages from the switch's 3 queue and forwards them to the Siddhi instance within the container via a local HTTP request. When a complex event is triggered, the Siddhi instance processes the event according to the rule script, generates a complex event message, and sends it back to the event receiving middleware via HTTP. Complex event streams: The event receiving middleware sends complex event messages to RabbitMQ switch 2 according to the defined rules. Complex event output: The complex event output end of the result output layer provides it to other business systems through the queue of switch 2. Node resource heartbeat packets: The event receiving middleware of each CEP node periodically writes the resource usage status to Redis. Load status synchronization: When the Leader node allocates a new fragment, it reads the load data of each CEP node in Redis as the allocation basis.

[0063] Rule offline: After user initiation, the cluster control component will scale down the CEP processing node and event distribution middleware to 0, while retaining the Redis rule information and RabbitMQ binding relationship for easy quick re-uploading; during re-uploading, the scale will be expanded to the number before offline, and operation will be resumed directly after Leader election.

[0064] Rule deletion: After a user initiates a deletion, the cluster control component deletes the corresponding StatefulSet, RabbitMQ queue, and Redis key-value pairs related to the rule, thus ending the rule's lifecycle.

[0065] Among them, basic components are replaceable: for example, Kubernetes can be replaced by Docker Compose or OpenStack, RabbitMQ can be replaced by any message middleware, Redis can be replaced by NoSQL databases such as Etcd or MongoDB, and Siddhi can be replaced by CEP processing middleware such as Esper.

[0066] It is understood that, although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A distributed complex event processing system employing a hyperheuristic algorithm, deployed on a containerized cluster, characterized in that, include: Cluster control components, distributed database, message middleware, event acquisition and analysis middleware, event distribution middleware, and CEP processing nodes; The event acquisition and analysis middleware is used to execute the following training process in response to the creation of new CEP rules: parsing the rules to obtain event attributes; and acquiring raw event data. The original event data is segmented using a hyperheuristic algorithm to determine the subset of key attributes used to distinguish the segments to which events belong and the segmentation results. Additionally, a hyperheuristic algorithm is invoked to solve for partitioning and combining, in order to obtain a weighted Gower distance model based on event attributes for calculating the similarity between events; The distributed database is used to store the fragment segmentation results, attribute clustering results, and the weighted Gower distance model; The event distribution middleware, deployed when the rules are online, includes a Leader node elected through election. The Leader node is used to: identify the segment to which a newly arrived original event belongs based on the segmentation results; calculate the distance between the segment and existing core segments of the partition based on the weighted Gower distance model; and perform dynamic load allocation based on the distance calculation results and the real-time load information of each CEP processing node obtained from the distributed database to determine the target CEP processing node for processing the event. The CEP processing node integrates a CEP engine instance and an event receiving middleware; the event receiving middleware is used to proxy the CEP engine instance to receive events and send complex events, and to report the resource usage status of this node as the real-time load information to the distributed database.

2. The distributed complex event processing system employing a hyperheuristic algorithm according to claim 1, characterized in that, The event acquisition and analysis middleware specifically includes: The rule analysis module is used to parse CEP rule scripts and extract event names, attribute lists, and rule backbone content. The data acquisition module is used to collect a specified amount or duration of raw event data from the message middleware; The segmentation module is used to iteratively solve the problem by calling a hyperheuristic algorithm based on the associated attributes in the core content of the rules and the original event data, so as to select the optimal attribute subset for segmentation, and use an objective function to quantitatively evaluate the accuracy of the segmentation results, the number of segments and the computational cost. The partition combination module is used to combine the original event data, call the hyperheuristic algorithm to perform iterative solution to obtain the optimal attribute weights in the weighted Gower distance model, and use the objective function to quantitatively evaluate the correctness of the partition combination result, the number of partitions, the partition cohesion and the computational cost.

3. A distributed complex event processing system employing a hyperheuristic algorithm as described in claim 1, characterized in that, The dynamic load balancing performed by the Leader node of the event distribution middleware includes the following three logics: If the nearest existing partition's corresponding CEP processing node has sufficient resources, the event will be assigned to that node; If the nearest existing partition has insufficient CEP processing node resources, other nodes with sufficient resources will be searched in order of distance from nearest to farthest as the target node. If all CEP processing nodes are insufficient in resources, a request for expansion is made to the cluster control component, and new partitions and CEP processing nodes are created.

4. A distributed complex event processing system employing a hyperheuristic algorithm as described in claim 1, characterized in that, The Leader node of the event distribution middleware is also used to perform load redistribution, including: Dynamic reallocation: Periodically check the load of each CEP processing node, calculate the load difference, and reallocate some event fragments of high-load nodes according to their distance from the core fragments of low-load nodes. Complete reallocation: Based on the fragment partitioning results and the current number of CEP processing nodes, reinitialize all partition allocation relationships.

5. A distributed complex event processing system employing a hyperheuristic algorithm according to claim 1, characterized in that, The event receiving middleware in the CEP processing node proxies the communication between the CEP engine instance and the message middleware by modifying the source and aggregate annotations in the CEP rule script to HTTP types pointing to the local port, thereby achieving zero-intrusion integration of CEP business processing logic.

6. A distributed complex event processing system employing a hyperheuristic algorithm according to claim 1, characterized in that, The distance calculation in the weighted Gower distance model is based on the attribute clustering results, which pair attributes of different event types and assign weights to each attribute pair. The distance value is calculated and weighted based on the attribute data type using Hamming distance, normalized Euclidean distance, or string edit distance.

7. A distributed complex event processing system employing a hyperheuristic algorithm according to claim 2, characterized in that, The hyperheuristic algorithm used in the segmentation module is based on a genetic algorithm framework, incorporating crossover and mutation mechanisms, and calling various heuristic operators, including simulated annealing, genetic algorithm, tabu search, differential evolution, artificial bee colony, ant colony optimization, particle swarm optimization, gray wolf optimization, and whale optimization, for iterative optimization.

8. A distributed complex event processing system employing a hyperheuristic algorithm according to claim 2, characterized in that, The hyperheuristic algorithm used in the partitioning combination module is based on a genetic algorithm framework, integrating crossover and mutation mechanisms, and calling various heuristic operators, including tabu search, ant colony optimization, particle swarm optimization, differential evolution, artificial bee colony, gray wolf optimization, whale optimization, sine and cosine algorithm, butterfly optimization algorithm, and sparrow search algorithm, for iterative optimization. The correctness of its solution is verified by co-training independent CEP engine instances, and evaluated by comprehensively considering the trigger matching rate, trigger coverage rate, and false trigger rate.

9. A distributed complex event processing system employing a hyperheuristic algorithm according to claim 1, characterized in that, The cluster control component is used to manage the entire lifecycle of CEP rules, which includes the stages of creation, collection and analysis, deployment, operation, decommissioning and deletion. The cluster control component controls the deployment and destruction of the corresponding middleware according to the lifecycle stage.

10. A distributed complex event processing system employing a hyperheuristic algorithm according to claim 1, characterized in that, The containerized cluster is a Kubernetes cluster; the distributed database is Redis; the message middleware is RabbitMQ; and the CEP engine instance is a Siddhi instance.

Images