A complex event processing unit deployment system
Patent Information
- Application Number
- CN202310392787.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2022-11-22
- Filing Date
- 2023-04-11
- Publication Date
- 2026-10-23
- Estimated Expiration
- 2043-04-11
AI Technical Summary
[0003]发明目的：针对现有技术中存在的问题与不足，本发明提供一种基于Kubernetes的复杂事件处理单元部署系统，借助容器集群管理平台Kubernetes，有效解决了复杂事件处理单元部署问题(Operator Placement Problem)相关领域的研究人员在进行研究时对部署算法验证与比较的需求，实现了对复杂事件处理模型的简易部署
[0019]本发明具备以下有益效果：可以帮助解决复杂事件处理单元部署问题(OperatorPlacement Problem)相关领域的研究人员在进行研究时对部署算法验证与比较的需求。相较于以往，用户只需要编写处理单元Operator、编排处理单元图Graph、实现自己的算法、配置基本参数即可实现对复杂事件处理模型的简易部署。同时还具备链路等效时延模拟、输入率等效负载等级等功能，支持多样化的自定义和拓展。相比于从头编写分布式程序、实现各种底层技术细节，该系统能帮助用户更专注于部署问题本身；相比于单机运行的网络仿真环境，该系统能在真实的物理机集群上部署，使得实验效果更加精确、真实。
Smart Images
Figure CN116400930B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to a complex event processing unit deployment system based on Kubernetes, belonging to the field of computer software technology. Background Technology
[0002] Complex Event Processing (CEP) is a real-time stream processing and analysis technique for events in dynamic environments. Events typically refer to state changes that are meaningful to the system or environment. The CEP model generally consists of a directed acyclic graph (DAG) composed of a series of operators, known as an operator graph. Using this technique, users define rules for the operators based on their understanding of the relationships between events, enabling real-time filtering, correlation, and aggregation of events to obtain an analysis of the event stream. The goal is to extract valuable complex event sequences or information to inform responses. This technique is widely used in crime prevention, risk identification, and marketing decision-making. In distributed computing models, how to deploy complex event processing operators across various computing nodes to meet user expectations for system response time and load balancing is academically categorized as the operator placement problem. In existing technologies, research on the deployment of processing units typically employs software simulation environments, such as OMNeT++ or NS3. However, compared to a real machine environment, software simulation has certain limitations in simulating link latency, congestion control, and other aspects, requiring further verification of the researched algorithms or content in a real machine environment. Summary of the Invention
[0003] Purpose of the Invention: Addressing the problems and shortcomings of existing technologies, this invention provides a Kubernetes-based deployment system for complex event processing units. Leveraging the Kubernetes container cluster management platform, it effectively solves the need for researchers in the field of the complex event processing unit deployment problem (Operator Placement Problem) to verify and compare deployment algorithms, enabling simplified deployment of complex event processing models. Users can use this system to easily deploy processing units in a real-device environment, efficiently completing the verification and comparison of deployment algorithms.
[0004] Technical solution: A Kubernetes-based complex event processing unit deployment system, which mainly consists of four parts: Cluster abstraction layer, Models complex event processing unit models, Methods algorithm module, and Core deployment module.
[0005] The Cluster abstraction layer, implemented through the KubeTools module, virtualizes the Kubernetes cluster, shielding it from many configuration details from the hardware nodes themselves or Kubernetes. The KubeTools module primarily encapsulates the Kubernetes API and also includes the DelayRules module for simulating link latency.
[0006] The Complex Event Processing Unit Models module is a container image that packages all user-written processing units (Operators), the orchestrated processing unit graph (Graph), the event description class (Event), the data generation module (Data), and the starter (Starter). The Complex Event Processing Unit Models module will be packaged by Docker and uploaded to the image repository according to the instructions in the user-written Dockerfile. The Dockerfile is the configuration file used by Docker during packaging; Docker is an open-source application container engine.
[0007] The Methods module primarily consists of an abstract algorithm base class and several actual algorithm examples that inherit from this base class. The abstract algorithm base class defines the input and output of the deployment algorithm, and users can implement their desired deployment algorithms by inheriting from it. The deployment algorithm obtains state information from the cluster, generates a deployment strategy, and the result is ultimately called by the Core module for deployment.
[0008] The deployment module, Core, primarily consists of the configuration module Config for deployment experiments, the cluster initialization module Initialize, the Deploy module responsible for executing deployments, and the DelayCalculate module for calculating response times. The Config module extracts and centralizes various parameters from the system. The Initialize module uses KubeTools to deploy an Nginx container for each node in the cluster to measure inter-node link latency. The Deploy module is responsible for executing the deployment on the cluster using the deployment strategy returned by the user-selected deployment algorithm.
[0009] The Cluster abstraction layer, through the KubeTools module, virtualizes the Kubernetes cluster, shielding it from many configuration details from the hardware nodes themselves or Kubernetes. Each node in the Kubernetes cluster deploys a containerized Nginx server, primarily used to extend functionality not provided by the Kubernetes API. When needing to obtain the response latency between nodes, the Kubernetes API can be used to enter the container and execute the Curl command.
[0010] The Cluster abstraction layer defines the amplification factor of the link latency between Kubernetes cluster nodes through the DelayRules module in the KubeTools module. These amplification factors are called in the processing unit Operator in the Models complex event processing unit model and used to implement the equivalent link latency simulation.
[0011] In the Complex Event Processing Unit Models, all Operators and the orchestrated processing unit graph are inherited from their respective abstract base classes. The abstract base class for Operators implements functionalities such as communication models between processing units, equivalent link latency simulation, and equivalent load levels based on input rates. Users only need to inherit from the base class and override the methods for handling events to define their required Operators. The abstract base class for the processing unit graph implements various methods for deployment and algorithm use, and provides subclasses with a method that needs to be overridden and is called by the constructor; this is called the override call method. This override call method returns the types, names, and topological descriptions of several Operators to initialize the orchestrated processing unit graph. Users only need to override this override call method when inheriting the abstract base class to orchestrate the required processing unit graph.
[0012] The Complex Event Processing Unit (CEP) Models are packaged and uploaded to the image repository by Docker according to the instructions in the user-defined Dockerfile. During deployment, the Deploy module in the Core module instructs KubeTools to send different instructions to populate the ENTRYPOINT (equivalent to the run instructions) of the CEP container and deploy the containers to their respective nodes. Finally, the Starter in the CEP Models instantiates the corresponding processing unit Operator according to the instructions, ultimately achieving the effect of using the same container image on different nodes but running different processing unit instances.
[0013] The Methods module contains several examples of practically deployed algorithms that inherit from the abstract algorithm base class, including the Greedy algorithm, LoadBalance algorithm, ResponseTimeAware algorithm, and ReinforcementLearning algorithm.
[0014] The DelayCalculate module in the Deployment Core calculates the response time of the deployment model under different deployment algorithms by analyzing Kubernetes logs. Furthermore, the modules within the Deployment Core are loosely coupled, allowing users to define their own modules similar to DelayCalculate for extension and to obtain more comprehensive analysis of deployment experiments.
[0015] The abstract base class of the processing unit Operator implements the communication model between processing units Operators. This communication model mainly relies on the principles of Kubernetes inter-Pod communication and TCP network programming. In Kubernetes, all containers are packaged in Pods, and Pods deployed on different nodes cannot communicate directly because they reside in different virtual address spaces. However, Kubernetes provides an API object called Service. Essentially, a Service acts like a routing table, exposing an access address within the cluster and forwarding all data to Pods with the same label. By creating a dedicated Service for each Pod, communication between Pods can be achieved.
[0016] The abstract base class of the processing unit Operator implements the link equivalent latency simulation. Its main principle is to assume that there are N nodes in the current cluster, and define M... ij M is the factor (M) by which the delay between the i-th node and the j-th node is amplified. ij ≥1). First, allow users to configure the latency amplification factor between all nodes in the cluster (in the DelayRules module). When the system starts running, record the transmission time T in the data packet. s And send node i, and then record the receiving time T again at the receiving end. r And the current node j, let the receiving thread wait for a period of length (T) s –T r )×(M ij -1) time, and then hand the data packet to the method responsible for event handling, thereby achieving the effect of simulating link latency.
[0017] The abstract base class of the processing unit Operator implements an input rate equivalent load level. In real-world environments, the event stream input rate faced by complex event processing models is often dynamically changing, making it impossible to fully simulate the effect of external data sources dynamically generating events within the cluster. Specialized processing units are needed to act as both data sources and data consumers. However, a situation arises where the input rate has an upper limit, preventing the simulation of high-pressure states. Therefore, we define the system state value as the ratio of the event input rate to the cluster computing power, considering system states with the same value as equivalent states. This allows us to reduce cluster computing power by extending the event processing time, thus maintaining a constant event input rate while decreasing cluster computing power, effectively simulating a state where cluster computing power remains constant but the event input rate increases. Assume the range of event input rate variation to be simulated is R. L To R H This range can then be divided into I (I≥0) intervals, and the changes in the event input rate can be replaced by changes in I+1 equivalent load levels: 0, 1, 2, ..., I. The larger I is, the more refined the equivalent result. Assume that the time taken for an operator to process the current event is T. process Then, the thread method responsible for event handling waits for T after processing each event. wait If a certain length of time is sufficient to achieve an equivalent reduction in cluster computing power, then at the equivalent load level of level i (i = 0, 1, 2, ..., I), the waiting time T wait The calculation formula is:
[0018]
[0019] This invention offers the following advantages: it helps researchers in the field of complex event processing unit deployment problems (Operator Placement Problem) to verify and compare deployment algorithms during their research. Compared to previous methods, users only need to write the processing unit Operator, arrange the processing unit graph, implement their own algorithm, and configure basic parameters to easily deploy complex event processing models. It also features functions such as link equivalent latency simulation and input rate equivalent load level, supporting diverse customization and expansion. Compared to writing distributed programs from scratch and implementing various low-level technical details, this system allows users to focus more on the deployment problem itself; compared to a single-machine network simulation environment, this system can be deployed on a real physical machine cluster, making the experimental results more accurate and realistic. Attached Figure Description
[0020] Figure 1 A schematic diagram illustrating the communication model of complex event processing units in a Kubernetes cluster;
[0021] Figure 2 A schematic diagram illustrating the principle of equivalent latency simulation in a Kubernetes cluster;
[0022] Figure 3 Deployment system architecture diagram for complex event processing units based on Kubernetes. Detailed Implementation
[0023] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.
[0024] like Figure 3 As shown, the Kubernetes-based complex event processing unit deployment system mainly consists of four parts: Cluster (the cluster abstraction layer), Models (the complex event processing unit models), Methods (the algorithm module), and Core (the deployment module).
[0025] The Cluster abstraction layer, implemented through the KubeTools module, virtualizes the Kubernetes cluster, shielding it from many configuration details from the hardware nodes themselves or Kubernetes itself. The KubeTools module primarily encapsulates the Kubernetes API. Each node in the Kubernetes cluster deploys a containerized Nginx server, mainly used to extend functionality not provided by the Kubernetes API. When needing to obtain inter-node response latency, the Kubernetes API can be used to enter the container and execute the Curl command. KubeTools also includes the DelayRules module for simulating link latency. The DelayRules module defines the amplification factor for inter-node link latency in the Kubernetes cluster. These amplification factors are invoked in the Operator processing units within the Models complex event handling unit model to simulate equivalent link latency.
[0026] The Complex Event Processing Unit Models module is a container image that packages all user-written processing units (Operators), an orchestrated processing unit graph (Graph), event description classes (Event), a data generation module (Data), and a starter. The Complex Event Processing Unit Models module is packaged by Docker and uploaded to the image repository according to the instructions in the user-written Dockerfile. During deployment, the Deploy module in the Core module instructs KubeTools to send different instructions to populate the ENTRYPOINT (equivalent to the run instructions) of the Complex Event Processing Unit Models container and deploy the container to its corresponding node. Finally, the starter in the Complex Event Processing Unit Models module instantiates the corresponding processing unit (Operator) according to the instructions, ultimately achieving the effect of using the same container image on different nodes but running different instances of processing units. For C / C++ and Java, the Dockerfile used during container packaging should include compilation-related configurations in addition to the base image; this is not necessary for interpreted languages like Python. The Event description class describes the event format, and the Data generation module is responsible for providing the data. Furthermore, all Operators and the orchestrated Graph of complex event processing units (Models) inherit from their respective abstract base classes. The abstract base class for Operators implements functions such as communication models between processing units, equivalent link latency simulation, and equivalent load levels based on input rates. Users only need to inherit from the base class and override the methods for handling events to define their required Operators. The abstract base class for the Graph implements various methods for deployment and algorithm use, and provides a method to subclasses that needs to be overridden and called by the constructor; this is called the override call method. This override call method returns the types, names, and topological descriptions of several Operators to initialize the orchestrated Graph. Users only need to override this override call method when inheriting the abstract base class to orchestrate the required Graph.
[0027] The abstract base class of the processing unit Operator implements the communication model between processing units Operators. This communication model mainly relies on the principles of inter-Pod communication in Kubernetes and TCP network programming. In Kubernetes, all containers are packaged in Pods, and Pods deployed on different nodes cannot communicate directly because they reside in different virtual address spaces. However, Kubernetes provides an API object called Service. Essentially, a Service acts like a routing table, exposing an access address within the cluster and forwarding all data to Pods with the same label. Figure 1 As shown, communication between Pods can be achieved simply by creating a dedicated Service for each Pod.
[0028] The abstract base class of the processing unit Operator implements the simulation of link equivalent delay, such as Figure 2 As shown, its main principle is that, assuming there are N nodes in the cluster, M is defined as follows: ij M is the factor (M) by which the delay between the i-th node and the j-th node is amplified. ij ≥1). First, allow users to configure the latency amplification factor between all nodes in the cluster (in the DelayRules module). When the system starts running, record the transmission time T in the data packet. s And send node i, and then record the receiving time T again at the receiving end. r And the current node j, let the receiving thread wait for a period of length (T) s –T r )×(M ij -1) time, and then hand the data packet to the method responsible for event handling, thereby achieving the effect of simulating link latency.
[0029] The abstract base class of the processing unit Operator implements an input rate equivalent load level. In real-world environments, the event stream input rate faced by complex event processing models is often dynamically changing, making it impossible to fully simulate the effect of external data sources dynamically generating events within the cluster. Specialized processing units are needed to act as both data sources and data consumers. However, a situation arises where the input rate has an upper limit, preventing the simulation of high-pressure states. Therefore, we define the system state value as the ratio of the event input rate to the cluster computing power, considering system states with the same value as equivalent states. This allows us to reduce cluster computing power by extending the event processing time, thus maintaining a constant event input rate while decreasing cluster computing power, effectively simulating a state where cluster computing power remains constant but the event input rate increases. Assume the range of event input rate variation to be simulated is R. L To R HThis range can then be divided into I (I≥0) intervals, and the changes in the event input rate can be replaced by changes in I+1 equivalent load levels: 0, 1, 2, ..., I. The larger I is, the more refined the equivalent result. Assume that the time taken for an operator to process the current event is T. process Then, the thread method responsible for event handling waits for T after processing each event. wait If a certain length of time is sufficient to achieve an equivalent reduction in cluster computing power, then at the equivalent load level of level i (i = 0, 1, 2, ..., I), the waiting time T wait The calculation formula is:
[0030]
[0031] The `Methods` module primarily consists of an abstract algorithm base class and several actual algorithm examples that inherit from this base class. These include the Greedy algorithm, LoadBalance algorithm, ResponseTimeAware algorithm, and ReinforcementLearning algorithm. The abstract algorithm base class defines the input and output of the deployment algorithm, and users can implement their desired deployment algorithms by inheriting from it. The deployment algorithm obtains state information from the cluster, generates a deployment strategy, and the result is ultimately called by the `Core` module for deployment.
[0032] The deployment module Core consists of three main components: the configuration module Config for deployment experiments, the cluster initialization module Initialize, the Deploy module responsible for executing deployments, and the DelayCalculate module for calculating response times. The Config module extracts and aggregates numerous system parameters, such as the key file used for connecting to Kubernetes, various default ports, image repository addresses, and load parameters. The Initialize module uses KubeTools to deploy an Nginx container for each node in the cluster to measure inter-node latency. The Deploy module executes deployments based on the deployment strategy returned by the user-selected deployment algorithm. The DelayCalculate module calculates the response time of the deployment model under different deployment algorithms by analyzing Kubernetes logs. Furthermore, the modules within the Core module are loosely coupled, allowing users to define their own modules similar to DelayCalculate for expansion and to obtain more comprehensive analysis of deployment experiments.
Claims
1. A complex event processing unit deployment system based on Kubernetes, characterized in that, It includes the Cluster abstraction layer, Models (complex event processing unit models), Methods (algorithm modules), and Core (deployment module); The Cluster abstraction layer, through the KubeTools module, virtualizes the Kubernetes cluster. The KubeTools module is a wrapper for the Kubernetes API and also includes the DelayRules module for simulating link latency. The Complex Event Processing Unit Models module is a container image that packages all user-written processing units (Operators), orchestrated processing unit graphs (Graphs), event description classes (Events), data generation modules (Data), and starters (Starters). The Complex Event Processing Unit Models module is packaged by Docker and uploaded to the image repository according to the instructions in the user-written Dockerfile. Dockerfile is the configuration file used during Docker packaging; Docker is an open-source application container engine. The Methods module consists of an abstract base class for algorithms and several actual algorithm examples that inherit from that base class. The abstract algorithm base class defines the input and output of the deployment algorithm. Users can implement their desired deployment algorithm by inheriting from the abstract algorithm base class. The deployment algorithm obtains state information from the cluster, generates a deployment strategy, and the result is finally called by the deployment module Core and applied to the deployment. The deployment module Core mainly consists of the configuration module Config for deployment experiments, the cluster initialization module Initialize, the Deploy module responsible for executing deployments, and the DelayCalculate module for calculating response time. The configuration module Config extracts and centralizes various parameters from the system. The cluster initialization module Initialize uses KubeTools to deploy an Nginx container for each node in the cluster to measure inter-node link latency. The Deploy module is responsible for executing the deployment on the cluster based on the deployment strategy returned by the user-selected deployment algorithm. The input rate equivalent load level is implemented in the abstract base class of the processing unit Operator; The system state value is defined as the ratio of the event input rate to the cluster computing power. System states with the same value are considered equivalent states. Let the range of the event input rate to be simulated be... arrive Divide this range into Each interval Using 0, 1, 2, ... common The change in the equivalent load level is used to replace the change in the event input rate. Let the time taken for an Operator to process the current event be... Then, the thread method responsible for event handling waits after processing each event. A time period of sufficient duration can achieve an equivalent reduction in cluster computing power. Therefore, at the equivalent load level of the th level, Waiting time The calculation formula is: Official 1.
2. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The Cluster abstraction layer implements virtualization of the Kubernetes cluster through the KubeTools module. Each node of the Kubernetes cluster deploys an Nginx server in the form of a container to extend some functions not provided by the Kubernetes API. When it is necessary to obtain the response latency between nodes, the Kubernetes API is used to enter the container and execute the Curl command to obtain it.
3. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The Cluster abstraction layer defines the amplification factor of the link latency between Kubernetes cluster nodes through the DelayRules module in the KubeTools module. These amplification factors are called in the processing unit Operator in the Models complex event processing unit model and used to implement the equivalent link latency simulation.
4. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, All processing units (Operators) and orchestrated processing unit graphs in the Complex Event Processing Unit Models are inherited from their respective abstract base classes. The abstract base class of the processing unit Operator implements the communication model between processing units, the equivalent latency simulation of links, and the equivalent load level of input rate. Users only need to inherit the base class and override the methods used to handle events to define the processing unit Operator they need. The abstract base class of the processing unit graph implements various methods for deployment and algorithm use, and provides a method that needs to be overridden by the constructor for subclasses. This method is called the overridden call method. The overridden call method returns the types, names, and topological descriptions of several processing unit Operators to complete the initialization of the orchestrated processing unit graph. Users only need to override this overridden call method when inheriting the abstract base class to orchestrate the required processing unit graph.
5. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The Complex Event Processing Unit (CEP) Models are packaged and uploaded to the image repository by Docker according to the instructions in the user-written Dockerfile. During deployment, the Deploy module in the Core module instructs KubeTools to send different instructions to populate the ENTRYPOINT of the CEP container, which is equivalent to running instructions, and deploys the container to its corresponding node. Finally, the Starter in the CEP model instantiates the corresponding processing unit Operator according to the instructions, thus achieving the effect of using the same container image on different nodes but running different processing unit instances.
6. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The Methods module contains several examples of practically deployed algorithms that inherit from the abstract algorithm base class, including the Greedy algorithm, LoadBalance algorithm, ResponseTimeAware algorithm, and ReinforcementLearning algorithm.
7. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The DelayCalculate module in the Deployment Core calculates the response time of the deployment model under different deployment algorithms by analyzing Kubernetes logs. At the same time, the modules within the Deployment Core are loosely coupled, and users can define their own modules similar to DelayCalculate to extend the scope and obtain more dimensions of analysis for deployment experiments.
8. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The abstract base class of the processing unit Operator implements the communication model between processing units Operator. This communication model is implemented by leveraging the inter-Pod communication principle of Kubernetes and TCP network programming.
9. The Kubernetes-based complex event processing unit deployment system according to claim 1, characterized in that, The abstract base class of the processing unit Operator implements the simulation of link equivalent latency. The principle is as follows: Suppose there are N nodes in the current cluster, define M... ij This is the factor by which the delay between the i-th node and the j-th node should be amplified. In the DelayRules module, users first configure the latency amplification factor between all nodes in the cluster; when the system starts running, the transmission time T is recorded in the data packets. s And send node i, and then record the receiving time T again at the receiving end. r And the current node j, so that the receiving thread waits for a period of length . The data packet is then handed over to the event handling method after a certain time, thus simulating link latency.