Event stream management method, system, device and medium

Abstract

The application discloses an event stream management method, system, device and medium, and belongs to the technical field of event management. The technical problem to be solved by the application is that a traditional event management system has a large load, is not timely in processing, has high delay and poor real-time performance when a large number of events are generated. The technical solution is as follows: event rule configuration: defining an event management rule; wherein the event management rule comprises an event source to be monitored, an event type, an event monitoring time period, an event pushing role, an event processing rule and an event linkage action; event aggregation: collecting and temporarily storing generated events in a message queue; event analysis: after subscribing to receive an event, performing storage analysis according to the event rule; and event processing: processing the event.

Description

Technical Field

[0001] This invention relates to the field of event management technology, specifically to an event streaming management method, system, device, and medium. Background Technology

[0002] Incident management is a systematic approach to effectively manage and control various incidents occurring both within and outside an organization. In the IT field, incident management specifically refers to a series of processes that link changes in the status of events with personnel responses in order to achieve specific business objectives. It involves multiple stages, including the collection, analysis, classification, prioritization, response, processing, and follow-up of incident information.

[0003] In the process of data interaction between multiple systems or services, event management systems are often used to decouple business logic and improve system flexibility. However, most event management systems process events in batches, which can lead to problems such as high load, untimely processing, high latency, and poor real-time performance when a large number of events are generated. Summary of the Invention

[0004] The technical objective of this invention is to provide an event streaming management method, system, device, and medium to solve the problems of high load, untimely processing, high latency, and poor real-time performance that occur in traditional event management systems when a large number of events are generated.

[0005] The technical objective of this invention is achieved in the following manner: an event streaming management method, the specific details of which are as follows:

[0006] Event rule configuration: Define event management rules; where event management rules include the event source to be monitored, event type, event listening time period, event push role, event processing rules, and event linkage actions;

[0007] Event aggregation: Collects and temporarily stores generated events in a message queue;

[0008] Event analysis: After subscribing to and receiving events, the event data is stored and analyzed according to the event rules;

[0009] Event handling: Handling events.

[0010] As a preferred approach, after the event rules are configured, the event rule extension message is sent to the processing hub based on the eKuiper middleware, and the event rules are used as the flow rules for the corresponding event stream. Then, the event rule configuration clarifies the rules for the collection, transmission, processing, and linkage of the event stream, providing parameter dependencies for subsequent functions.

[0011] As preferred, in the event aggregation process, the data of the RabbitMQ open source component is accessed into the data stream as a data source according to the eKuiper source plug-in, and the same rule event supports multiplexing an event stream;

[0012] In the process of event analysis, the event stream is subscribed to the relevant topic according to the business rule or the event terminal, the event raw data is parsed, the data is cleaned, the business event processing extension information is bound with the original information to form the complete event information, and then the eKuiper sink plug-in is used to import the event into the event stream, and finally the event is sent back to the final event topic of the RabbitMQ open source component.

[0013] After the event aggregation, the event needs to be shunted according to the event rule configuration, the shunting depends on the eKuiper open source component, and the suitable data source plug-in needs to be developed according to the business demand.

[0014] Or,

[0015] The event stream channel multiplexing function is developed to improve the data processing efficiency.

[0016] In the process of event analysis, the event stream is subscribed to the relevant topic according to the business rule or the event terminal, the event raw data is parsed, the data is cleaned, the business event processing extension information is bound with the original information to form the complete event information, and then the eKuiper sink plug-in is used to import the event into the event stream, and finally the event is sent back to the final event topic of the RabbitMQ open source component.

[0017] As preferred, after the event source is configured, the flow rule configured for the corresponding event source is queried and applied in the process of querying the event rule configuration.

[0018] The flow concept of event analysis is bound with the specific business process ID or the specified event monitoring device ID, and supports accessing multiple protocol data sources to obtain higher scalability and flexibility; wherein the data sources include MySQL, RabbitMQ, MQTT, Redis and HTTP.

[0019] When subscribing to access the data in the RabbitMQ open source component, the event analysis technology and business process are explained, after the event stream data is accessed, the sink plug-in of the eKuiper component is used to clean the data, the data is analyzed according to the custom rule information in the business system, the concerned data is left and is cut and assembled, and the cutting and assembling specifically means that the trigger action and linkage information corresponding to the event are bound, and finally the event data containing the business extension information is formed.

[0020] For the event data containing the event raw information and the corresponding processing rule extension information, the event analysis sends the event data back to the exchange of different topics in the RabbitMQ open source component for event processing subscription.

[0021] More preferably, in the event processing process, the event data after subscribing to the event analysis specification is parsed to obtain specific event sources and corresponding processing rules, and the processing request is further sent to specific business processing systems such as video recording systems, screenshot systems, notification systems, and the like according to the event rules to perform event recording, event screenshot, email and short message notification, so that a complete event flow from event occurrence to event processing feedback is continuously generated until the termination of the event flow.

[0022] For the analysis and judgment of abnormal events, positive feedback is formed for the device management personnel, the probability of event occurrence is reduced, and the logical closed loop of the event management system is completed.

[0023] An event flow management system, the system comprises:

[0024] An event rule configuration module for defining event management rules; wherein the event management rules include event sources, event types, event listening time periods, event pushing roles, event processing rules, and event linkage actions that need to be monitored;

[0025] An event aggregation module for accessing data from RabbitMQ open source components as data sources into a data stream according to a self-developed eKuiper source plug-in, and supporting reuse of an event stream for events of the same rule;

[0026] An event analysis module for combining event source raw event information with business rules to aggregate complete event stream data and send it back to the RabbitMQ open source component;

[0027] An event processing module for subscribing to final event data, sending relevant business systems for event resolution processing after format processing, and analyzing and judging event data to feed back to management personnel to complete the closed loop of the event management system.

[0028] As a preferred, the event rule configuration module sends event rule expansion messages to the processing hub based on the eKuiper middleware after completing the event rule configuration, and uses the event rules as stream rules of corresponding event streams; and then collects, transmits, processes, and links rules of the event stream through the event rule configuration to provide parameter dependencies for subsequent functions;

[0029] The event aggregation module uses RabbitMQ open source components as message queues, and according to the event stream rules configured by the event rules, the message queue stores system topic events, business topic events, and intelligent analysis topic events in the exchange corresponding to the multiple topic types configured by the event rules; after the event aggregation, the events need to be shunted according to the event rule configuration, and the shunting depends on the eKuiper open source component, and suitable data source plug-ins need to be developed according to business requirements;

[0030] The flow concept of the event analysis module is bound with a specific business process ID or a specified event monitoring device ID, and supports access to data sources of various protocols, so that higher expansibility and flexibility are obtained, wherein the data sources include MySQL, RabbitMQ, MQTT, Redis and HTTP.

[0031] When subscribing to access data in the RabbitMQ open source component, the technology and business process of event analysis are explained, after the event stream data is accessed, the sink plug-in mode of the eKuiper component is written to clean the data, the data is analyzed according to the custom rule information in the business system, the concerned data is left and cut and assembled, and the cutting and assembling are specifically: the trigger action and linkage information corresponding to the event are bound, and finally the event data containing business expansion information is formed; for the event data containing event original information and corresponding processing rule expansion information, the event analysis sends the event data back to the exchange of different topics in the RabbitMQ open source component for event processing subscription;

[0032] The event processing module subscribes to the event data after the event analysis specification, analyzes the specific event source and corresponding processing rule, and further sends the processing request to the specific business processing system according to the event rule, such as video recording system for event recording, screenshot system for event screenshot, notification system for email and short message notification, so that a complete event flow from event occurrence to event processing feedback is continuously carried out until the termination of the event flow; for the analysis and judgment of abnormal events, the device management personnel form a positive feedback, reduce the probability of event occurrence, and complete the logical closed loop of the event management system.

[0033] An electronic device, comprising: a memory and at least one processor;

[0034] The memory stores computer execution instructions.

[0035] The at least one processor executes the computer execution instructions stored in the memory, so that the at least one processor executes the event stream management method as described above.

[0036] A computer readable storage medium, the computer readable storage medium stores computer execution instructions, when the processor executes the computer execution instructions, realizes the event stream management method as described above.

[0037] The event stream management method, system, device and medium of the application have the following advantages:

[0038] (I) This invention is based on eKuiper as the central hub of streaming processing, and includes an event rule configuration module, an event aggregation module, an event analysis module, and an event processing module. The event aggregation module uses events published to the RabbitMQ component as the source of the event stream; the event analysis module combines the original event information of the event source with business rules to aggregate complete event stream data and send it back to RabbitMQ; the event processing module subscribes to the final event data, parses and processes the format, sends it to the relevant business system for event resolution, analyzes and judges the event data and feeds it back to the management personnel, and finally completes the closed loop of the event management system.

[0039] (ii) The present invention adopts a modular design. Event acquisition and event processing configuration are often not in the same domain. When an event occurs, it is unknown what method or action will be executed to resolve the event itself. A medium is needed to connect the two. Therefore, the event management system decouples the event module and the business module to improve the system's flexibility and scalability.

[0040] (III) Through the design of the four modules of the fluidized system, the present invention enables the complete life cycle of an event from its generation to its expiration to exist in a streaming manner. This overcomes the problems of soaring system resource consumption and large event response delays in traditional event management systems when dealing with massive event sources, improves the real-time performance and efficiency of event processing, and reduces maintenance difficulty and cost. Attached Figure Description

[0041] The invention will be further described below with reference to the accompanying drawings.

[0042] Appendix Figure 1 This is a flowchart illustrating the event flow management method. Detailed Implementation

[0043] The event streaming management method, system, device, and medium of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example 1:

[0045] As attached Figure 1 As shown in the figure, this embodiment provides an event streaming management method, which is as follows:

[0046] S1. Event Rule Configuration: Define event management rules; where event management rules include the event source to be monitored, event type, event listening time period, event push role, event processing rules, and event linkage actions;

[0047] S2, Event Aggregation: Collects the generated events and temporarily stores them in a message queue;

[0048] S3, event analysis: after receiving the event subscription, the event rule is stored and analyzed according to the event rule;

[0049] S4, event processing: processing the event.

[0050] After the event rule configuration in step S1 of the embodiment is completed, the event rule extension message is sent to the processing hub based on the eKuiper middleware, and the event rule is used as the stream rule of the corresponding event stream; the collection, transmission, processing and linkage rules of the event stream are specified through the event rule configuration, thereby providing parameter dependence for subsequent functions.

[0051] In the event aggregation process in step S2 of the embodiment, the data of the RabbitMQ open source component is accessed into the data stream as a data source according to the eKuiper source plug-in; and the same rule event supports multiplexing an event stream.

[0052] Among them, the RabbitMQ open source component is used as a message queue, and according to the event stream rule configured by the event rule, the message queue stores system topic events, business topic events and intelligent analysis topic events in the corresponding configuration of multiple topic type exchanges.

[0053] After the event aggregation, it needs to be shunted according to the event rule configuration, and the shunting depends on the eKuiper open source component, and the suitable data source plug-in needs to be developed according to the business requirements;

[0054] Or,

[0055] Develop event stream channel multiplexing function to improve data processing efficiency.

[0056] In the event analysis process in step S3 of the embodiment, the event stream is subscribed to the related topic according to the business rule or the event terminal, the event raw data is parsed, the data is cleaned, the business event processing extension information is bound with the original information to form the event complete information, and then the eKuiper sink plug-in is used to import the event complete information into the event stream, and finally the event complete information is sent back to the final event topic of the RabbitMQ open source component.

[0057] In the embodiment, after the event source is configured, the stream rule configured for the corresponding event source is queried and applied in the event rule configuration process.

[0058] The stream concept of event analysis is bound with the specific business process ID or the specified event monitoring device ID, and supports accessing multiple protocol data sources to obtain higher scalability and flexibility; wherein the data sources include MySQL, RabbitMQ, MQTT, Redis and HTTP.

[0059] When subscribing to access data in the RabbitMQ open source component, the technology and business process of event analysis are explained. After accessing the event stream data, the data is cleaned by writing the sink plug-in of the eKuiper component, the data is analyzed according to the self-defined rule information in the business system, the concerned data is left and is cut and assembled, and the cutting and assembling are specifically: the trigger action and linkage information corresponding to the event are bound, and finally the event data containing business expansion information is formed;

[0060] For the event data containing event original information and corresponding processing rule expansion information, the event analysis sends the event data back to the exchange of different topics of the RabbitMQ open source component for event processing subscription.

[0061] In the event processing process in step S4 of the embodiment, the event data after the event analysis specification is subscribed, the specific event source and corresponding processing rule are obtained after analysis, and the processing request is further sent to the specific business processing system according to the event rule, such as the event recording of the video recording system, the event screenshot of the screenshot system, and the email and short message notification of the notification system. A complete event flow from event occurrence to event processing feedback is continuously generated until the termination of the event flow;

[0062] For the analysis and judgment of abnormal events, the device management personnel are positively fed back, the probability of event occurrence is reduced, and the logical closed loop of the event management system is completed.

[0063] Embodiment 2:

[0064] The embodiment provides an event stream management system, which comprises:

[0065] An event rule configuration module is configured to define an event management rule; wherein the event management rule comprises an event source to be monitored, an event type, an event listening time period, an event pushing role, an event processing rule, and an event linkage action;

[0066] An event aggregation module is configured to access data of a RabbitMQ open source component into a data stream as a data source according to a self-developed eKuiper source plug-in, and support multiplexing one event stream for events of the same rule;

[0067] An event analysis module is configured to combine event source original event information with a business rule to aggregate complete event stream data and send the complete event stream data back to the RabbitMQ open source component;

[0068] An event processing module is configured to subscribe to final event data, send related business systems for event solving processing after format processing, and analyze and judge event data to feed back to management personnel, and complete a closed loop of the event management system.

[0069] After the event rule configuration module in this embodiment completes the event rule configuration, an event rule extension message is sent to the processing hub based on the eKuiper middleware, and the event rule is taken as a stream rule of a corresponding event stream; and the collection, transmission, processing and linkage rules of the event stream are specified through the event rule configuration, thereby providing parameter dependencies for subsequent functions.

[0070] The event aggregation module in this embodiment uses the RabbitMQ open source component as a message queue, and according to the event stream rule configured by the event rule configuration, a corresponding exchange storage system of multiple topic types in the message queue stores system topic events, business topic events and intelligent analysis topic events; after the event aggregation comes, it needs to be shunted according to the event rule configuration, and the shunting depends on the eKuiper open source component, and a suitable data source plug-in needs to be developed according to the business requirements.

[0071] The event analysis module in this embodiment binds the stream concept and the specific business process ID or the specified event monitoring device ID, and supports accessing multiple protocol data sources, thereby obtaining higher scalability and flexibility; wherein the data sources include MySQL, RabbitMQ, MQTT, Redis and HTTP;

[0072] When subscribing to access data in the RabbitMQ open source component, the event analysis technology and the business process are explained; after the event stream data is accessed, the sink plug-in mode of the eKuiper component is used to clean the data, the data is analyzed according to the self-defined rule information in the business system, the concerned data is left and is cut and assembled, and the cutting and assembling specifically means that the trigger action and the linkage information of the corresponding event are bound, and finally the event data containing the business expansion information is formed; for the event data containing the event original information and the corresponding processing rule extension information, the event analysis sends the event data back to the exchange of different topics in the RabbitMQ open source component for event processing subscription.

[0073] The event processing module in this embodiment subscribes to the event data after the event analysis specification, analyzes to obtain the specific event source and the corresponding processing rule, and further sends the processing request to the specific business processing system according to the event rule, such as the video recording system for event recording, the screenshot system for event screenshot, the notification system for email and short message notification, so that a complete event stream from event occurrence to event processing feedback is continuously generated until the termination of the event stream; for the analysis and judgment of abnormal events, the device management personnel form a positive feedback, reduce the probability of event occurrence, and complete the logical closed loop of the event management system.

[0074] Embodiment 3:

[0075] The embodiment also provides an electronic device, comprising a memory and at least one processor;

[0076] The memory stores computer-executable instructions.

[0077] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the event stream management method according to any one of the embodiments.

[0078] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), programmable logic devices (PLD) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor can be a microprocessor or any conventional processor.

[0079] The memory can be used to store computer programs and / or modules, and the processor can realize various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store operating systems, application programs required by at least one function, etc.; and the data storage area can store data created during use of the terminal, etc. In addition, the memory can also include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a secure digital (SD) card, a flash memory card, at least one disk storage period, a flash memory device, or other volatile solid-state memory devices.

[0080] Embodiment 4:

[0081] The embodiment also provides a computer-readable storage medium, wherein a plurality of instructions are stored, the instructions are loaded by a processor, and the processor executes the event stream management method in any one of the embodiments. Specifically, a system or device provided with a storage medium can be provided, the storage medium stores software program codes for realizing the functions of any one of the above embodiments, and the computer (or CPU or MPU) of the system or device reads and executes the program codes stored in the storage medium.

[0082] In this case, the program codes read from the storage medium can realize the functions of any one of the above embodiments, and thus the program codes and the storage medium storing the program codes constitute a part of the present embodiment.

[0083] The storage medium for providing the program code includes floppy disks, hard disks, magneto-optical disks, optical disks (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), magnetic tapes, nonvolatile memory cards and ROM. Alternatively, the program code can be downloaded from a server computer via a communication network.

[0084] In addition, it should be clear that not only the program code read by the computer can be executed, but also the operating system or the like operating on the computer can be caused to perform part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0085] In addition, it should be understood that the program code read by the storage medium is written into the memory provided in the expansion board inserted into the computer or the memory provided in the expansion unit connected to the computer, and then the CPU or the like installed on the expansion board or the expansion unit is caused to perform part or all of the actual operations based on the instructions of the program code, thereby realizing the functions of any one of the above embodiments.

[0086] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not limited thereto; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can still be modified, or part or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. An event streaming management method, characterized in that, The method is as follows: Event rule configuration: Define event management rules; where event management rules include the event source to be monitored, event type, event listening time period, event push role, event processing rules, and event linkage actions; Event aggregation: Based on the self-developed eKuiper source plugin, data from the RabbitMQ open-source component is integrated into the data stream as the data source, and a single event stream can be reused for events with the same rules; Event analysis: After subscribing to and receiving events, the event data is stored and analyzed according to the event rules; Event handling: Handling events; Among them, after configuring the event source, the flow rules configured for the corresponding event source during the event rule configuration process are queried and applied; The event analysis flow concept is bound to a specific business process ID or a specified event monitoring device ID, and supports access to data sources with multiple protocols; among which, data sources include MySQL, RabbitMQ, MQTT, Redis, and HTTP; When subscribing to data from the RabbitMQ open-source component, the technical and business processes of event analysis are explained. After the event stream data is accessed, the data is cleaned by writing a sink plugin for the eKuiper component. The data is analyzed according to the custom rules in the business system. The data of interest is retained and then trimmed and assembled. The trimming and assembly process involves binding the triggering actions and linkage information of the corresponding events to form event data containing extended business information. For event data that includes original event information and corresponding extended processing rule information, event analysis sends the event data back to the switches of different topics in the RabbitMQ open-source component for use by event processing subscriptions.

2. The event streaming management method according to claim 1, characterized in that, After the event rules are configured, the event rule extension message is sent to the processing hub based on the eKuiper middleware, and the event rules are used as the flow rules of the corresponding event stream. Then, the event rule configuration clarifies the collection, transmission, processing and linkage rules of the event stream, providing parameter dependencies for subsequent functions.

3. The event streaming management method according to claim 1, characterized in that, During the event aggregation process, data from the RabbitMQ open-source component is incorporated into the data stream as the data source based on the eKuiper source plugin, and a single event stream can be reused for events with the same rules. The RabbitMQ open-source component is used as a message queue. According to the event flow rules configured by the event rules, the message queue is configured with various topic types of exchange storage system topic events, business topic events and intelligent analysis topic events. Once events are aggregated, they need to be distributed according to the event rules or an event stream channel reuse function needs to be developed. Distribution depends on the eKuiper open-source component, and suitable data source plugins need to be developed according to business needs.

4. The event streaming management method according to claim 1, characterized in that, During event analysis, the event stream subscribes to relevant topics according to business rules or event terminals, parses the original event data, cleans the data, and binds the extended information of business event processing with the original information to form the complete event information. Then, it is imported into the event stream through the eKuiper sink plugin and finally sent back to the final event topic of the RabbitMQ open source component.

5. The event streaming management method according to any one of claims 1 to 4, characterized in that, During the event handling process, the event data after subscribing to the event analysis specifications is parsed to obtain the specific event source and corresponding processing rules, and the processing request is further sent to the specific business processing system according to the event rules; The analysis and evaluation of abnormal events provide positive feedback to equipment managers, reduce the probability of such events occurring, and complete the logical closed loop of the event management system.

6. An event streaming management system, characterized in that, The system includes: The event rule configuration module is used to define event management rules; these rules include the event sources to be monitored, event types, event listening time periods, event push roles, event processing rules, and event linkage actions. The event aggregation module is used to access data from the RabbitMQ open-source component into the data stream as the data source based on the self-developed eKuiper source plugin. It supports reusing an event stream for events with the same rules. The event analysis module is used to combine the raw event information from the event source with business rules to aggregate complete event stream data and send it back to the RabbitMQ open-source component. The event handling module is used to subscribe to the final event data, parse and process the format, send it to the relevant business systems for event resolution, and analyze and evaluate the event data to provide feedback to the management personnel, thus completing the closed loop of the event management system. The event analysis module binds the concept of flow to a specific business process ID or a specified event monitoring device ID, and supports access to data sources with multiple protocols, including MySQL, RabbitMQ, MQTT, Redis, and HTTP. When subscribing to data from the RabbitMQ open-source component, the technical and business processes of event analysis are explained. After the event stream data is received, the data is cleaned by writing a sink plugin for the eKuiper component. The data is analyzed according to custom rules in the business system, and the relevant data is retained and then trimmed and assembled. Specifically, trimming and assembly involves binding the triggering actions and linkage information for the corresponding events, ultimately forming event data containing extended business information. For event data containing the original event information and corresponding processing rule extension information, the event analysis sends the event data back to the exchanges of different topics in the RabbitMQ open-source component for use by event processing subscribers.

7. The event streaming management system according to claim 6, characterized in that, After completing the event rule configuration, the event rule configuration module sends the event rule extension message to the processing center based on the eKuiper middleware and uses the event rule as the flow rule for the corresponding event stream. Then, the event rule configuration clarifies the collection, transmission, processing and linkage rules of the event stream, providing parameter dependencies for subsequent functions. The event aggregation module uses the RabbitMQ open-source component as a message queue. According to the event flow rules configured by the event rules, the message queue is configured with various topic types of exchange storage for system topic events, business topic events, and intelligent analysis topic events. After the events are aggregated, they need to be distributed according to the event rules. The distribution depends on the eKuiper open-source component, and suitable data source plugins need to be developed according to business needs. The event processing module subscribes to the event data after event analysis specifications, parses it to obtain the specific event source and corresponding processing rules, and further sends the processing request to the specific business processing system according to the event rules; the analysis and evaluation of abnormal events provides positive feedback to equipment managers, reduces the probability of events occurring, and completes the logical closed loop of the event management system.

8. An electronic device, characterized in that, include: Memory and at least one processor; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the event streaming management method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the event streaming management method as described in any one of claims 1 to 5.

Images