Method and System for Implementing Complex Event Processing in a Distributed Streaming Data Processing Engine

Abstract

The present invention provides a method and system for implementing complex event processing in a distributed stream data processing engine. First, obtain a pre-defined event sequence pattern to be matched and data processing logic, then embed a CEP operator into the data processing logic, and issue an execution instruction to the distributed stream data processing engine; obtain the events entering the CEP operator, and distribute the events to the event queue corresponding to a certain key in the partition according to the key-taking logic; finally, determine whether the time concept in the event is the actual processing time or the event occurrence time, and perform corresponding event processing according to the judgment result. The present invention provides an effective solution for implementing CEP on an existing distributed stream data processing engine. While implementing a fully functional CEP module, this solution also integrates it with the existing distributed stream data processing engine, enabling it to perfectly inherit many excellent characteristics of the processing engine and being able to horizontally scale under a large amount of data.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed computing, and particularly to a method and system for implementing complex event processing in a distributed stream data processing engine. Background Art

[0002] With the further in-depth informatization of enterprises and governments, all walks of life continuously generate a large amount of data with implicit value, such as transaction data in the securities market, detection data in the power industry, etc. It has become powerless to process these real-time data with relational databases. More importantly, business and decision-making personnel need to timely discover and process some meaningful information patterns from these data. For example, abnormal operations with a high suspicion of violation in the securities market transactions need to be further tracked and observed, and then manual intervention is required. For such requirements, if traditional data processing methods are used to solve them, such as importing all the data into a relational database and then uniformly analyzing and processing it with analysis tools, although some meaningful data can also be obtained, it cannot meet the real-time requirements, resulting in unnecessary losses.

[0003] Since CEP is a method for discovering specific event patterns in an event sequence. An event refers to something that happens in a business, including business activities, process states, network conditions, etc. Many events can be processed as long as they can be identified, usually accompanied by actions such as alarm sending. When it is impossible to reliably discover anomalies by processing a single event and it is necessary to analyze in context, CEP will be used. In almost all scenarios, CEP analysis requires events to be associated in real time and requires a certain format with accurate timestamps. CEP is one of the typical applications of stream processing, and its application scenarios include financial services, automatic goods management, click stream analysis, electronic health systems and other fields that require identification, understanding and quick response. CEP implemented based on a distributed stream data processing engine naturally inherits the advantages of the engine such as high throughput and low latency, and can meet the needs of most actual scenarios. Therefore, the present invention proposes a technology for implementing CEP using an existing distributed stream data processing engine. Summary of the Invention

[0004] In view of the above-mentioned disadvantages of the prior art, the purpose of the present invention is to provide a method and system for implementing complex event processing in a distributed stream data processing engine to solve the problems existing in the prior art.

[0005] To achieve the above purpose and other related purposes, the present invention provides a system for implementing complex event processing in a distributed stream data processing engine. The system includes a distributed stream data processing engine, and a CEP module and CEP operators are embedded in the distributed stream data processing engine; the CEP module includes: an event pattern definition sub-module, an NFA sub-module, and an event pattern processing sub-module;

[0006] The event pattern definition sub-module is used to define the event sequence pattern to be matched and the processing logic for the successfully matched event sequence;

[0007] The event pattern processing sub-module

[0008] The NFA sub-module is used to receive an event sequence and update the state and matching information according to the event itself information and the current state information;

[0009] The CEP operator is used to process event sequences under different temporal semantics and send the event sequences with correct semantics to an independent NFA sub-module for processing.

[0010] Optionally, the NFA sub-module includes a shared buffer component, an NFA compiler component, a post-matching policy component, and an NFA component;

[0011] The shared buffer component is used to store the state information during the event sequence matching process; the state information includes at least two key data structures, one of which is an event set, and the other is a buffer node set; the event set is used to save the mapping relationship between the event number and the real event, and the buffer node set is used to save the mapping relationship between the buffer node number and the specific node information;

[0012] The NFA compiler component is used to compile the matched event pattern into an NFA sub-module in the initial state;

[0013] The post-matching policy component is used to filter the preset repeated matching sequences that appear during the matching process;

[0014] The NFA component is used to receive an event sequence and update the state and matching information according to the event itself information and the current state information; and when a new match occurs, determine whether to retain the new match according to the post-matching policy.

[0015] Optionally, the process of the CEP operator processing event sequences under different temporal semantics includes:

[0016] Judge the event according to the specified temporal semantics to determine whether the event is the actual processing time or the event occurrence time;

[0017] If it is the actual processing time, continue to judge whether it is necessary to process the events with the same timestamp in a preset order; and when there is no need to process, update the time information, directly hand the event to the NFA sub-module for processing, and record the updated state of the NFA sub-module; otherwise, when it is necessary to process, cache the event until after the actual processing time increases, and uniformly process all cached events at the corresponding timestamp;

[0018] If it is the event occurrence time, continue to determine whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program; if it does not exceed, cache the event until after the actual processing time increases, and uniformly process all cached events at the corresponding timestamp; if it exceeds, determine that the event is a late event and perform corresponding processing.

[0019] Optionally, the process in which the CEP operator uniformly processes all cached events at the corresponding timestamp after the actual processing time increases includes:

[0020] When the actual processing time or the minimum event time being processed increases, retrieve all events with the previous unit timestamp;

[0021] Sort all the retrieved events according to a preset rule, and sequentially submit the sorted times to the NFA sub-module for processing;

[0022] Update the time information and record the state of the updated NFA sub-module.

[0023] Optionally, the CEP operator includes one or more CEP partitions, and each CEP partition includes one or more keys, and each key is used to map all currently received events, the current state of the NFA sub-module, and all currently matched partial event sequences.

[0024] Optionally, the system further includes an input module for obtaining the original event information source of the CEP service.

[0025] The present invention also provides a method for implementing complex event processing in a distributed stream data processing engine, and the method includes the following steps:

[0026] Obtain a pre-defined event sequence pattern to be matched and data processing logic, and the data processing logic at least includes: the processing logic for the successfully matched event sequence;

[0027] Embed the CEP operator into the data processing logic and issue an execution instruction to the distributed stream data processing engine;

[0028] Obtain the events entering the CEP operator and allocate the events to the event queue corresponding to a certain key in the partition according to the key-taking logic; wherein, the CEP operator includes one or more CEP partitions, and each CEP partition includes one or more keys;

[0029] Judge whether the time concept in the event is the actual processing time or the event occurrence time, and perform corresponding event processing according to the judgment result.

[0030] Optionally, the process of determining whether the time concept in the event is the actual processing time or the event occurrence time and performing corresponding event processing according to the determination result includes:

[0031] Judge the event according to the specified time semantics to determine whether the event is the actual processing time or the event occurrence time;

[0032] If it is the actual processing time, continue to judge whether it is necessary to process the events with the same timestamp in the preset order; and when no processing is required, update the time information, directly hand the event to the NFA sub-module for processing, and record the updated state of the NFA sub-module; otherwise, when processing is required, cache the event until after the actual processing time increases, and uniformly process all cached events at the corresponding timestamp;

[0033] If it is the event occurrence time, continue to judge whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program; if not, cache the event until after the actual processing time increases, and uniformly process all cached events at the corresponding timestamp; if it exceeds, determine that the event is a late event and perform corresponding processing.

[0034] Optionally, the process of uniformly processing all cached events at the corresponding timestamp after the actual processing time increases includes:

[0035] When the actual processing time or the minimum event time being processed increases, take out all events with the previous unit timestamp;

[0036] Sort all the taken-out events according to the preset rules, and hand the sorted times to the NFA sub-module for processing in turn;

[0037] Update the time information and record the updated state of the NFA sub-module.

[0038] Optionally, the process of directly handing the event to the NFA sub-module for processing and recording the updated state of the NFA sub-module includes:

[0039] Obtain the event sequence being matched and the event to be processed;

[0040] According to the matching state diagram corresponding to the event sequence being matched, and according to the matching state diagram, enumerate the states of the new matching state diagram that can be reached after consuming the event to be processed;

[0041] Among them, the method of enumerating the states of the new matching state diagram that can be reached after consuming the event to be processed includes: enumerating all conversion actions caused by the corresponding event; all conversion actions at least include: ignore action and receive action.

[0042] As described above, the present invention provides a method and system for implementing complex event processing in a distributed stream data processing engine, which has the following beneficial effects:

[0043] The present invention first obtains a pre-defined event sequence pattern and data processing logic to be matched, then embeds a CEP operator into the data processing logic, and issues an execution instruction to the distributed stream data processing engine; obtains events entering the CEP operator, and distributes the events to an event queue corresponding to a key in a partition according to the key-taking logic; finally, determines whether the time concept in the event is the actual processing time or the event occurrence time, and performs corresponding event processing according to the judgment result. The present invention provides an effective solution for implementing CEP on an existing distributed stream data processing engine. While implementing a functionally complete CEP module, this solution also integrates it with the existing distributed stream data processing engine, enabling it to perfectly inherit many excellent characteristics of the processing engine and being able to horizontally expand under a large amount of data. The present invention adopts the architectural concept that the basic functions of the CEP module and the stream data processing engine are independent of each other, enabling both the CEP module and the stream data processing engine to be tested and optimized separately, and ensuring good maintainability. For a specific CEP service, users can choose to send events to the same partition or different partitions for processing according to business needs, which is not only convenient for local functional debugging but also suitable for deployment in an actual production environment, thus reflecting the good usability of the present invention from the user's perspective. In addition, the present invention enables users to add a CEP function with a complete mechanism on the basis of an existing distributed stream data processing engine. Compared with a separate CEP module, the CEP function implemented on the basis of the distributed stream data processing engine in the present invention can perfectly inherit the distributed processing advantages of the engine, expanding from single-machine processing to cluster operation. At the same time, the CEP operator can be further decomposed into two-layer functional subunits, which can better solve the problem of data skew. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a schematic diagram of the hardware structure of a system for implementing complex event processing in a distributed stream data processing engine provided in an embodiment;

[0045] Figure 2 It is a schematic flowchart of a method for implementing complex event processing in a distributed stream data processing engine provided in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0046] The following describes the embodiments of the present invention through specific examples. Those skilled in the art can easily understand the other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments. Various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, without conflict, the following embodiments and the features in the embodiments can be combined with each other.

[0047] It should be noted that the diagrams provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner. Therefore, only the components related to the present invention are shown in the diagrams, rather than being drawn according to the number, shape, and size of the components in actual implementation. The type, quantity, and proportion of each component in actual implementation can be arbitrarily changed, and the component layout type may also be more complex.

[0048] In the shared buffer, an event set is a mapping table of event numbers and real events, and a node set is a mapping table of node numbers and buffer nodes; in addition, it also includes a mapping table of timestamps and the number of all events arriving at a certain timestamp, a cache of an event set, and a cache of a node set. The latter two are used to improve the throughput of calculations. This shared buffer includes a variety of callable methods. When the system time advances by one unit, the timestamp mapping table will clear all event counts before the current system time. When the system discovers a new event, the event needs to be registered in the shared buffer, the count value corresponding to the timestamp where the event is located in the timestamp mapping table is incremented by one, and the event is added to the event combination buffer. When the system needs to add a node event to the node set, it first needs to create or obtain a node, then establish a connection between the node and the state node before consuming the event, and then add the node to the set.

[0049] The present invention provides a system for implementing complex event processing in a distributed stream data processing engine. The system includes a distributed stream data processing engine, in which a CEP module and CEP operators are embedded; the CEP module includes: an event pattern definition sub-module, an NFA sub-module, and an event pattern processing sub-module;

[0050] The event pattern definition sub-module is used to define the event sequence pattern to be matched, and to define the processing logic for the event sequence with successful matching;

[0051] The event pattern processing sub-module

[0052] The NFA sub-module is used to receive the event sequence and update the state and matching information according to the event itself information and the current state information;

[0053] The CEP operator is used to process event sequences under different time semantics and send the semantically correct event sequences to an independent NFA sub-module for processing. Among them, the CEP operator includes one or more CEP partitions, and each CEP partition includes one or more keys, and each key is used to map all currently received events, the current state of the NFA sub-module, and all partially matched event sequences currently matched.

[0054] According to the above description, in an exemplary embodiment, the NFA sub-module in this embodiment includes a shared buffer component, an NFA compiler component, a post-matching policy component, and an NFA component;

[0055] The shared buffer component is used to store the state information during the event sequence matching process; the state information includes at least two key data structures, one of which is an event set, and the other is a buffer node set; the event set is used to save the mapping relationship between event numbers and real events, and the buffer node set is used to save the mapping relationship between buffer node numbers and specific node information;

[0056] The NFA compiler component is used to compile the matched event pattern into an NFA sub-module in an initial state;

[0057] The post-matching policy component is used to filter the preset repeated matching sequences that appear during the matching process;

[0058] The NFA component is used to receive event sequences, update the state and matching information according to the event itself information and the current state information; and when a new match occurs, determine whether to retain the new match according to the post-matching policy.

[0059] According to the above description, in an exemplary embodiment, the process of the CEP operator processing event sequences under different time semantics includes:

[0060] Judge the event according to the specified time semantics to determine whether the event is the actual processing time or the event occurrence time;

[0061] If it is the actual processing time, continue to judge whether it is necessary to process the events with the same timestamp in a preset order; and when there is no need to process, update the time information, and directly hand over the event to the NFA sub-module for processing, and record the updated state of the NFA sub-module; otherwise, when it is necessary to process, cache the event until after the actual processing time increases, and uniformly process all cached events corresponding to the timestamp;

[0062] If it is the event occurrence time, continue to determine whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program; if it does not exceed, cache the event until after the actual processing time increases, and uniformly process all cached events under the corresponding time stamp; if it exceeds, determine that the event is a late event and perform corresponding processing.

[0063] Specifically, the process in which the CEP operator uniformly processes all cached events under the corresponding time stamp after the actual processing time increases includes:

[0064] When the actual processing time or the minimum event time being processed increases, retrieve all events of the previous unit time stamp;

[0065] Sort all the retrieved events according to a preset rule, and sequentially submit the sorted times to the NFA sub-module for processing;

[0066] Update the time information and record the status of the updated NFA sub-module.

[0067] In an exemplary embodiment, the system further includes an input module for obtaining the source of the original event information of the CEP service.

[0068] In a specific embodiment, this embodiment also provides a system for gradually implementing the CEP function on an existing distributed stream data processing engine, including: a CEP module and a CEP operator. The CEP module specifically includes: an event pattern definition sub-module, an NFA (Non-deterministic Finite Automaton) sub-module, and an event pattern processing sub-module; add a suitable CEP operator to the distributed stream data processing engine, which is divided into two processing modes according to whether the event distinguishes the time concept, and the processing mode of distinguishing the time concept is further considered separately according to two situations: according to the actual processing time and according to the event occurrence time; embed the above two into the distributed stream data processing engine, and detect a certain or certain event sequences by defining a suitable event stream pattern, and perform corresponding processing on the matching events.

[0069] Specifically, the NFA sub-module in this embodiment includes multiple components such as a shared buffer, an NFA compiler, a post-matching strategy, and the NFA itself, where:

[0070] The shared buffer component is used to store the status information during the event sequence matching process, including two key data structures, namely the event set and the buffer node set. The former is used to save the mapping relationship between the event number and the real event, and the latter is used to save the mapping relationship between the buffer node number and the specific node information;

[0071] The NFA compiler component is used to compile the event pattern specified by the user for matching into the NFA sub-module in the initial state;

[0072] The post-matching policy component is used to filter out preset repeated matching sequences that occur during the matching process;

[0073] The NFA component is used to receive an event sequence and update the state according to the event itself information and the current state information while updating the matching information. If a new match appears, it is determined whether to retain the match according to the post-matching policy.

[0074] According to the above description, the CEP operator in this embodiment is used to process event sequences under different time semantics and send the event sequences with correct semantics to an independent NFA sub-module for processing, including the following steps:

[0075] i) According to the time semantics specified by the user, if it is the actual processing time, continue to determine whether it is necessary to process the events with the same timestamp in a preset order. If not, update the time information, directly send the event to the NFA sub-module for processing, and record the updated state of the NFA sub-module; otherwise, it is necessary to cache the event until the actual processing time in the system increases and then uniformly process all the cached events at this timestamp, as described in iii). If it is the event occurrence time, jump to ii);

[0076] ii) First, determine whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program. If not, it is necessary to cache the event until the minimum event time processed in the system increases and then uniformly process all the cached events at this timestamp, as described in iii). Otherwise, it is necessary to declare the event as a late event and perform corresponding processing;

[0077] iii) When the actual processing time or the minimum event time processed in the system increases, first retrieve all the events of the previous unit timestamp. If sorting is required according to certain rules, perform a global sort and send them to the NFA sub-module for processing in sequence. Otherwise, directly send them to the NFA sub-module for processing in the order in which the events arrive. Then update the time information and record the updated state of the NFA sub-module.

[0078] According to the above description, the system in this embodiment may further have the following characteristics: For CEP services that need to distinguish different keys, a CEP operator can be decomposed into multiple CEP partitions, and each Cep partition holds multiple keys. Each key maps all the events currently received by the key, the current NFA sub-module state of the key, and all the partially matched event sequences of the key. In this way, large-scale CEP services can be horizontally distributed across multiple computers. The same machine can contain one or more CEP partitions, the same partition can contain one or more keys, and the number of events contained in different keys can be more or less, enabling reasonable load balancing according to the actual business situation.

[0079] As can be seen from the above, this embodiment enables users to add a CEP function with a complete mechanism on the basis of the existing distributed stream data processing engine. Further, compared with a separate CEP module, the CEP function implemented on the basis of the distributed stream data processing engine can perfectly inherit the distributed processing advantages of the engine, expanding from single-machine processing level to cluster operation. At the same time, the CEP operator can continue to be decomposed into two layers of functional subunits, which can better solve the problem of data skew.

[0080] In another exemplary embodiment, as Figure 1 shown, the CEP framework diagram of this embodiment includes the following related modules: an input module, other operator modules of the stream data engine, a CEP operator module, and an NFA sub-module. Among them, the input module is the source of the original event information for the CEP service, other operator modules of the stream data engine are the preprocessing and postprocessing stages of the CEP, the CEP operator module is the module that actually invokes the CEP function, and the NFA sub-module is the core dependency for the execution of the CEP function.

[0081] In summary, the present system provides an effective solution for implementing CEP on the existing distributed stream data processing engine. While implementing a CEP module with complete functions, this solution also integrates it with the existing distributed stream data processing engine, enabling it to perfectly inherit many excellent characteristics of the processing engine and be horizontally scalable under a large amount of data. The present system adopts the architectural concept that the basic functions of the CEP module are independent of the stream data processing engine, enabling both the CEP module and the stream data processing engine to be tested and optimized separately, and ensuring good maintainability. For a specific CEP service, users can choose to send events to the same partition or different partitions for processing according to business needs, which is not only convenient for local function debugging but also suitable for deployment in the actual production environment, thus reflecting the good usability of the present system from the user's perspective.

[0082] The present invention also provides a method for implementing complex event processing in a distributed stream data processing engine, the method comprising the following steps:

[0083] Obtain a pre-defined event sequence pattern to be matched and data processing logic, the data processing logic at least including: processing logic for the successfully matched event sequence;

[0084] Embed a CEP operator into the data processing logic, and issue an execution instruction to the distributed stream data processing engine;

[0085] Obtain events entering the CEP operator, and allocate the events to an event queue corresponding to a key in a partition according to the key-taking logic; wherein, the CEP operator includes one or more CEP partitions, and each CEP partition includes one or more keys;

[0086] Judge whether the time concept in the event is the actual processing time or the event occurrence time, and perform corresponding event processing according to the judgment result.

[0087] According to the above description, in an exemplary embodiment, the process of judging whether the time concept in the event is the actual processing time or the event occurrence time, and performing corresponding event processing according to the judgment result includes:

[0088] Judge the event according to the specified time semantics to determine whether the event is the actual processing time or the event occurrence time;

[0089] If it is the actual processing time, continue to judge whether it is necessary to process the events with the same timestamp in a preset order; and when there is no need to process, update the time information, and directly hand over the event to the NFA sub-module for processing, and record the updated state of the NFA sub-module; otherwise, when it is necessary to process, cache the event until after the actual processing time increases, and uniformly process all cached events at the corresponding timestamp;

[0090] If it is the event occurrence time, continue to judge whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program; if it does not exceed, cache the event until after the actual processing time increases, and uniformly process all cached events at the corresponding timestamp; if it exceeds, determine that the event is a late event, and perform corresponding processing.

[0091] According to the above description, specifically, the process of uniformly processing all cached events at the corresponding timestamp after the actual processing time increases includes:

[0092] When the actual processing time or the minimum event time for processing increases, take out all events with the previous unit timestamp;

[0093] Sort all the retrieved events according to the preset rules, and sequentially submit the sorted events to the NFA sub-module for processing;

[0094] Update the time information and record the status of the NFA sub-module after the update.

[0095] Specifically, the process of directly submitting the event to the NFA sub-module for processing and recording the status of the NFA sub-module after the update includes:

[0096] Obtain the event sequence being matched and the event to be processed;

[0097] According to the matching state diagram corresponding to the event sequence being matched, and after enumerating and consuming the event to be processed according to the matching state diagram, enumerate the states of the new matching state diagram that can be reached;

[0098] Among them, the method of enumerating the states of the new matching state diagram that can be reached after consuming the event to be processed includes: enumerating all the transition actions caused by the corresponding event; all the transition actions at least include: ignore action and receive action.

[0099] As Figure 2 shown, in a specific embodiment, a method for implementing complex event processing in a distributed stream data processing engine is further provided, and the method includes the following steps:

[0100] Step 110, the user customizes the event sequence pattern to be matched, and needs to declare many conditions including filtering conditions in the start stage, filtering conditions in multiple intermediate stages, and connection conditions between each stage, etc.;

[0101] Step 120, the user also needs to customize the processing logic for the successfully matched event sequence, such as outputting to the console or sending an alarm message, etc.;

[0102] Step 130, the user defines other data processing logics, places the CEP operator in one of the links, and issues an execution instruction to the data processing engine;

[0103] Step 140, for an event entering the CEP operator, first allocate it to the event queue corresponding to a certain key in a certain partition according to the key-taking logic;

[0104] Step 150, determine whether the time concept in the system is the actual processing time or the event occurrence time. If it is the former, jump to step 151; otherwise, jump to step 160;

[0105] Step 151, continue to determine whether it is necessary to process the events with the same time stamp in a preset order. If not, jump to step 152; otherwise, jump to step 153;

[0106] Step 152: Update the time information, directly hand over the event to the NFA sub-module for processing, jump to Step 180, and record the updated status of the NFA sub-module;

[0107] Step 153: Cache the event until the actual processing time in the system increases, and then uniformly process all the cached events at this time stamp, and jump to Step 170;

[0108] Step 160: First, determine whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program. If not, jump to Step 161; otherwise, jump to Step 162;

[0109] Step 161: Cache the event until the minimum event time processed in the system increases, and then uniformly process all the cached events at this time stamp, and jump to Step 170;

[0110] Step 162: Declare the event as a late event and perform corresponding processing if necessary;

[0111] Step 170: When the actual processing time or the minimum event time processed in the system increases, first retrieve all the events of the previous unit time stamp. If sorting is required according to certain rules, jump to Step 171; otherwise, jump to Step 180. After completion, the time information needs to be updated, and the updated status of the NFA sub-module needs to be recorded;

[0112] Step 171: Globally sort the events at this time stamp, hand them over to the NFA sub-module for processing in sequence, and jump to Step 180;

[0113] Step 180: Try to put the event into all the currently matching event sequences recorded by the current NFA and update the matching status. The specific process is shown in Step 190. Then, determine whether there are new complete matching sequences. According to the matching strategy, decide whether to retain these new complete matching sequences. If retained, continue with subsequent processing; otherwise, discard them directly;

[0114] Step 190: For a currently matching event sequence and an event to be processed, according to the matching status diagram corresponding to the matching event sequence, enumerate the possible new states of the matching status diagram after consuming this event for this event. The method is to enumerate all the possible transition actions caused by this event. According to the type of transition action, it includes the following two cases:

[0115] i) Ignoring the action, it is further divided into three sub - cases according to the matching status. In the first case, if the current matching status is the start status, no operation is performed. In the second case, if the next matching status is the same as the current status, the version number of the current node is incremented. In the third case, skipping this matching status and entering the next matching status, the version number of the current node is updated and a new status column is added.

[0116] ii) For the accepting action, first, a new node needs to be added to the shared buffer, and the version number of this node is the version number of the current matching status. Secondly, a new status column is added, and the increment of the version number of this status column is the sum of the number of accepting actions caused by this event. Finally, if the next matching status can be directly skipped to the end status, it needs to be taken out and returned to the user as a complete set of matching event sequences.

[0117] The present invention provides a method for implementing complex event processing in a distributed stream data processing engine. First, obtain a pre - defined event sequence pattern to be matched and data processing logic, then embed the CEP operator into the data processing logic, and issue an execution instruction to the distributed stream data processing engine; obtain the events entering the CEP operator, and distribute the events to the event queue corresponding to a certain key in the partition according to the key - taking logic; finally, determine whether the time concept in the events is the actual processing time or the event occurrence time, and perform corresponding event processing according to the judgment result. The present invention provides an effective solution for implementing CEP on an existing distributed stream data processing engine. While implementing a fully functional CEP module, it also integrates it with the existing distributed stream data processing engine, enabling it to perfectly inherit many excellent characteristics of the processing engine and being able to horizontally expand under a large amount of data. The present invention adopts the architectural concept that the basic functions of the CEP module and the stream data processing engine are independent of each other, enabling both the CEP module and the stream data processing engine to be tested and optimized separately, and ensuring good maintainability. For a specific CEP service, users can choose to send events to the same partition or different partitions according to business needs, which is not only convenient for local functional debugging but also suitable for deployment in the actual production environment, thus reflecting the good usability of the present invention from the user's perspective. In addition, the present invention enables users to add a CEP function with a complete mechanism on the basis of the existing distributed stream data processing engine. Compared with a separate CEP module, the CEP function implemented on the basis of the distributed stream data processing engine in the present invention can perfectly inherit the distributed processing advantages of the engine, expand from single - machine processing to cluster operation, and at the same time, the CEP operator can be further decomposed into two - layer functional sub - units, which can better solve the problem of data skew.

[0118] The above embodiments are only illustrative of the principles and effects of the present invention and are not intended to limit the present invention. Any person familiar with this technology can modify or change the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by those with ordinary knowledge in the technical field without departing from the spirit and technical idea disclosed by the present invention should still be covered by the claims of the present invention.

Claims

1. A system for implementing complex event processing in a distributed stream data processing engine, characterized in that, The system includes a distributed stream data processing engine, in which a CEP module and CEP operators are embedded; the CEP module includes: an event pattern definition sub-module, an NFA sub-module, and an event pattern processing sub-module; The event pattern definition sub-module is used to define the event sequence pattern to be matched and the processing logic for the successfully matched event sequence; The event pattern processing sub-module, The NFA sub-module is used to receive the event sequence and update the state and matching information according to the event itself information and the current state information; The CEP operator is used to process the event sequence under different time semantics and send the event sequence with correct semantics to an independent NFA sub-module for processing; wherein, the process of the CEP operator processing the event sequence under different time semantics includes: judging the event according to the specified time semantics to determine whether the event is the actual processing time or the event occurrence time; if it is the actual processing time, continue to judge whether it is necessary to process the events with the same timestamp in a preset order; and when there is no need to process, update the time information, directly hand the event to the NFA sub-module for processing, and record the updated state of the NFA sub-module; otherwise, when it is necessary to process, cache the event until after the actual processing time increases, and uniformly process all the cached events corresponding to the timestamp; if it is the event occurrence time, continue to judge whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program; if it does not exceed, cache the event until after the actual processing time increases, and uniformly process all the cached events corresponding to the timestamp; if it exceeds, determine that the event is a late event and perform corresponding processing.

2. The system for implementing complex event processing in a distributed stream data processing engine according to claim 1, wherein The NFA sub-module includes a shared buffer component, an NFA compiler component, a post-matching policy component, and an NFA component; The shared buffer component is used to store the state information during the event sequence matching process; the state information includes at least two key data structures, one of which is an event set, and the other is a buffer node set; the event set is used to save the mapping relationship between the event number and the real event, and the buffer node set is used to save the mapping relationship between the buffer node number and the specific node information; The NFA compiler component is used to compile the matched event pattern into the NFA sub-module in the initial state; The post-matching policy component is used to filter the preset repeated matching sequences that appear during the matching process; The NFA component is used to receive the event sequence and update the state and matching information according to the event itself information and the current state information; And when a new match appears, determine whether to retain the new match according to the post-matching policy.

3. The system for implementing complex event processing in a distributed stream data processing engine according to claim 1 or 2, characterized in that, The process of the CEP operator uniformly processing all the cached events corresponding to the timestamp after the actual processing time increases includes: When the actual processing time or the minimum event time to be processed increases, take out all the events of the previous unit timestamp; Sort all the retrieved events according to the preset rules, and sequentially submit the sorted events to the NFA sub-module for processing; Update the time information and record the state of the updated NFA sub-module.

4. The system for implementing complex event processing in a distributed stream data processing engine according to claim 1 or 2, wherein The CEP operator includes one or more CEP partitions, and each CEP partition includes one or more keys, and each key is used to map all the currently received events, the current state of the NFA sub-module, and all the partially matched event sequences.

5. The system for implementing complex event processing in a distributed stream data processing engine according to claim 1 or 2, characterized in that, The system further includes an input module for obtaining the source of the original event information of the CEP service.

6. A method for implementing complex event processing in a distributed stream data processing engine, characterized in that, The method includes the following steps: Obtain the predefined event sequence pattern to be matched and the data processing logic, and the data processing logic at least includes: the processing logic for the successfully matched event sequence; Embed the CEP operator into the data processing logic and issue an execution instruction to the distributed stream data processing engine; Obtain the events entering the CEP operator and allocate the events to the event queue corresponding to a certain key in the partition according to the key-taking logic; wherein, the CEP operator includes one or more CEP partitions, and each CEP partition includes one or more keys; Judge whether the time concept in the event is the actual processing time or the event occurrence time, and perform corresponding event processing according to the judgment result, including: judging the event according to the specified time semantics to determine whether the event is the actual processing time or the event occurrence time; if it is the actual processing time, continue to judge whether it is necessary to process the events with the same timestamp in the preset order; and when there is no need to process, update the time information, and directly submit the event to the NFA sub-module for processing and record the state of the updated NFA sub-module; otherwise, when it is necessary to process, cache the event until after the actual processing time increases, and uniformly process all the cached events at the corresponding timestamp; if it is the event occurrence time, continue to judge whether the actual occurrence time included in the event exceeds the minimum event time processed by the current program; if it does not exceed, cache the event until after the actual processing time increases, and uniformly process all the cached events at the corresponding timestamp; if it exceeds, determine that the event is a late event and perform corresponding processing.

7. The method for implementing complex event processing in a distributed stream data processing engine according to claim 6, wherein The process of uniformly processing all the cached events at the corresponding timestamp after the actual processing time increases includes: When the actual processing time or the minimum event time being processed increases, retrieve all the events with the previous unit timestamp; Sort all the retrieved events according to the preset rules, and sequentially submit the sorted events to the NFA sub-module for processing; Update the time information and record the state of the updated NFA sub-module.

8. The method for implementing complex event processing in a distributed stream data processing engine according to claim 6, wherein The process of directly submitting the event to the NFA sub-module for processing and recording the state of the updated NFA sub-module includes: Obtain the event sequence being matched and the event to be processed; According to the matching state diagram corresponding to the event sequence being matched, and after enumerating and consuming the event to be processed according to the matching state diagram, obtain the state of the new matching state diagram that can be reached; Among them, after enumerating the events to be processed for consumption, the ways to reach the states of the new matching state diagram that can be reached include: enumerating all the transition actions that the corresponding events will cause; all the transition actions at least include: Ignore action and receive action.

Images