Distributed processing method of alarm event and related device
By introducing distributed processing devices into the monitoring and alarm system, dynamically configure alarm processing rules and perform event convergence, suppression and aggregation, the problem of low processing efficiency of massive alarm events is solved, and high-performance real-time processing and system stability are achieved.
Patent Information
- Application Number
- CN202411035093.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-30
- Publication Date
- 2024-11-22
AI Technical Summary
The existing monitoring and alarm system has poor convergence effect when handling massive alarm events, resulting in the impact of system stability and efficiency.
By introducing a distributed processing device into the alarm generation service, and dynamically configure alarm processing rules using the configuration management database or interface, the convergence, suppression and aggregation of alarm events can be achieved, and a relatively small number of final events are generated.
It improves the efficiency of handling massive alarm events, meets the high-performance real-time processing requirements for massive alarm events on servers, reduces redundant alarms, and improves the reliability and maintainability of the system.
Smart Images

Figure CN119003274A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of data processing technology, and in particular to a distributed processing method for alarm events and related devices. Background Art
[0002] At present, the monitoring alarm system has an alarm flooding situation, which seriously affects the stability and efficiency of enterprise application systems. To this end, specific processing rules are often used to converge the massive alarm events to reduce the number of alarm events.
[0003] However, current processing rules can only be set in a fixed manner, so the convergence effect on a large number of alarm events of a certain application system is relatively poor. Summary of the invention
[0004] In view of the above problems, the present application provides a distributed processing device, method and electronic device for alarm events, in order to improve the convergence effect of alarm events. The specific scheme is as follows:
[0005] A first aspect of the present application provides a distributed processing device for alarm events, comprising at least: a first service module, and a second service module;
[0006] The first service module is used to obtain an alarm event generated by the alarm generation service and store the alarm event in a storage database; and obtain an alarm processing rule and store the alarm processing rule in the storage database; wherein the alarm processing rule is obtained from a configuration management database, or is generated using a configuration management interface in response to an alarm configuration instruction;
[0007] The second service module is used to obtain the alarm processing rules and multiple alarm events from the storage database; process the alarm events according to the alarm processing rules to obtain at least one final event; the final event indicates that the monitored server is in a corresponding alarm state.
[0008] In a possible implementation, the second service module includes:
[0009] An event monitoring submodule, used for monitoring the event change state of the storage database, and obtaining a corresponding alarm event from the storage database when the event change state indicates an event change;
[0010] The event aggregation submodule is used to process the alarm events obtained by the event monitoring submodule according to the alarm processing rules to obtain at least one final event.
[0011] In a possible implementation, the second service module further includes:
[0012] The input-output queue is used to store the alarm events obtained by the event monitoring submodule from the storage database; the alarm events in the input-output queue are provided to the event aggregation submodule according to the first-in-first-out rule.
[0013] In a possible implementation, the second service module further includes:
[0014] an identification storage submodule, configured to obtain an event identification from a cache database; after obtaining the alarm event generated by the alarm generation service, the first service module stores the event identification of the alarm event in the cache database;
[0015] The event monitoring submodule monitors the event change status of the storage database according to the event identifier in the identifier storage submodule.
[0016] In a possible implementation, the method further includes:
[0017] The third service module is used to collect system monitoring data corresponding to the monitored server; obtain server metadata from the configuration management database according to tag information in the system monitoring data; integrate the system monitoring data and the server metadata to obtain system integration data; and transmit the system integration data to the alarm generation service;
[0018] The first service module is further used for: generating monitoring rules in response to monitoring configuration instructions, and providing the monitoring rules to the alarm generation service; the alarm generation service processes the system integration data according to the monitoring rules to generate the alarm event.
[0019] In a possible implementation, the second service module publishes the final event to an event bus, so as to output the final event through the event bus.
[0020] A second aspect of the present application provides a distributed processing method for an alarm event, comprising:
[0021] Using the first service module, obtaining an alarm event generated by the alarm generation service, and storing the alarm event in a storage database;
[0022] Using the first service module, obtaining an alarm processing rule and storing the alarm processing rule in the storage database; wherein the alarm processing rule is obtained from a configuration management database, or is generated using a configuration management interface in response to an alarm configuration instruction;
[0023] The second service module is used to obtain the alarm processing rules and the multiple alarm events from the storage database; the alarm events are processed according to the alarm processing rules to obtain at least one final event; the final event indicates that the monitored server is in a corresponding alarm state.
[0024] In a possible implementation, the alarm processing rule includes at least one of the following:
[0025] An alarm convergence rule, wherein the alarm convergence rule is used to converge multiple alarm events into one alarm event according to a convergence label;
[0026] An alarm suppression rule, wherein the alarm suppression rule is used to retain alarm events with a high alarm level among alarm events with the same label;
[0027] Alarm aggregation rules, where the alarm aggregation rules are used to aggregate multiple alarm events into one alarm event according to an aggregation tag or aggregation content.
[0028] A third aspect of the present application provides a computer program product, comprising computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the distributed processing method for alarm events of the above-mentioned first aspect or any implementation method of the first aspect.
[0029] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0030] The memory is used to store computer programs;
[0031] The processor is used to execute the computer program so that the electronic device can implement the distributed processing method for alarm events of the above-mentioned first aspect or any implementation manner of the first aspect.
[0032] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement a distributed processing method for alarm events according to the first aspect or any implementation method of the first aspect.
[0033] By means of the above technical scheme, in a distributed processing method and related device for alarm events provided by the present application, the alarm processing rules are obtained from the configuration management database by the deployed first service module or the alarm processing rules are generated in response to the user's alarm configuration instructions by using the configuration management interface. In this way, after the alarm events obtained from the alarm generation service are stored in the storage database by the first service module, the alarm processing rules and alarm events can be obtained from the storage data by the deployed second service module. In this way, the alarm events can be processed according to the alarm processing rules, thereby processing a large number of alarm events into a relatively small number of final events to characterize that the monitored server is in a corresponding alarm state. It can be seen that the alarm processing rules obtained by the configuration management database or the configuration management interface in the present application can be dynamically configured in real time. Based on this, the large number of alarm events generated by the alarm generation service can be processed in real time with high performance, which can meet the convergence effect of the large number of alarm events on the server. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0035] Figure 1 A schematic diagram of the structure of a distributed processing device for alarm events provided in an embodiment of the present application;
[0036] Figure 2 A partial structural diagram of a distributed processing device for alarm events provided in an embodiment of the present application;
[0037] Figure 3 A schematic diagram of another part of the structure of a distributed processing device for alarm events provided in an embodiment of the present application;
[0038] Figure 4 A schematic diagram of another part of the structure of a distributed processing device for alarm events provided in an embodiment of the present application;
[0039] Figure 5 Another structural schematic diagram of a distributed processing device for alarm events provided in an embodiment of the present application;
[0040] Figure 6 A flowchart of a distributed processing method for alarm events provided in an embodiment of the present application;
[0041] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0042] Figure 8 This is a system architecture diagram of the high-performance interactive alarm processing based on a distributed architecture proposed in this application;
[0043] Fig. 9 The overall flow chart of the system for high-performance interactive alarm processing based on distributed architecture proposed in this application;
[0044] Fig.10 This is a schematic diagram of the interaction between the relevant components in the process of processing alarm events and obtaining alarm convergence configuration in the learning software API of the monitoring center proposed in this application. DETAILED DESCRIPTION
[0045] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.
[0046] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0047] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.
[0048] refer to Figure 1 , is a schematic diagram of the structure of a distributed processing device for alarm events provided in an embodiment of the present application, which can be deployed in an electronic device capable of data processing, such as a terminal or a server. The technical solution in this embodiment is mainly used to improve the convergence effect of alarm events.
[0049] Specifically, the device in this embodiment may include: a first service module 101 and a second service module 102; wherein:
[0050] The first service module 101 is used to obtain the alarm event generated by the alarm generation service and store the alarm event in a storage database; and obtain the alarm processing rule and store the alarm processing rule in the storage database.
[0051] Among them, the alarm processing rules are obtained from the configuration management database, or generated using the configuration management interface in response to the alarm configuration instruction. Based on this, in the case of different alarm event convergence scenarios applied in this embodiment, the alarm processing rules can be adjusted in the configuration management database, or the alarm processing rules can be configured using the configuration management interface. The alarm generation service can be a service that generates corresponding alarm events according to the monitoring rules, and can also be called an external data source, that is, a data source that can provide a large number of alarm events.
[0052] For example, the first service module 101 can be a web service, and the configuration management database can be referred to as CMDB (Configuration Management Database), which stores pre-configured or real-time adjusted alarm processing rules. The web service can establish a data connection with the CMDB, so that the web service can send a corresponding request to the CMDB, and the CMDB returns the alarm processing rules to the web service; or, the web service provides a configuration management interface to the user, and the user can configure or dynamically adjust the alarm processing rules according to the actual business monitoring requirements. In addition, the user can also configure or dynamically adjust the monitoring rules according to the actual business monitoring requirements, and the monitoring rules are used for the alarm generation service to generate alarm events.
[0053] It should be noted that the alarm event is an alarm event of the monitored server. That is, the alarm generation service monitors the monitored server according to the monitoring rules, for example, determines whether the CPU utilization exceeds a preset threshold, etc., and generates a corresponding alarm event.
[0054] The monitored server may be an independent server or a server in a distributed server group.
[0055] The second service module 102 is used to obtain an alarm processing rule and a plurality of alarm events from a storage database; and process the alarm event according to the alarm processing rule to obtain at least one final event.
[0056] Among them, the final event represents that the monitored server is in a corresponding alarm state. For example, the second service module 102 can be an application programming interface (API) service, which can process a large number of alarm events obtained from the storage database according to alarm processing rules, such as suppression, convergence and aggregation, so that the number of final events obtained can be significantly less than the number of alarm events before being processed.
[0057] In one implementation, the alarm processing rule may include at least one of the following:
[0058] Alarm convergence rules: Alarm convergence rules are used to converge multiple alarm events into one alarm event according to the convergence label;
[0059] Alarm suppression rules: Alarm suppression rules are used to retain alarm events with higher alarm levels among alarm events with the same label;
[0060] Alarm aggregation rules are used to aggregate multiple alarm events into one alarm event according to aggregation tags or aggregation contents.
[0061] In the specific implementation, the alarm suppression rule is as follows: for multiple alarm events with the same convergence label, the high-level alarm event suppresses the low-level alarm event. For example, the CPU utilization of a server a-server in the a-cluster computer room is 60% and 80% for a period of time. According to the alarm suppression rule, only the alarm event with CPU utilization exceeding 80% is retained; and the alarm convergence rule can be based on the alarm event obtained by the alarm suppression rule, aggregated with the a-cluster computer room as the dimension, and the final event of a-cluster in the time period is aggregated into:
[0062] The CPU utilization rate of a-server in the cluster room exceeds 80%;
[0063] The CPU utilization rate of b-server in a-cluster computer room exceeds 90%, etc.
[0064] Assume that the aggregation tag is "{cluster="a-cluster"}" and the aggregation content is "a-cluster computer room". Based on the alarm aggregation rule, the alarm content is aggregated with "a-cluster computer room" as the dimension to form a final event.
[0065] According to the above technical solution, in a distributed processing device for alarm events provided by an embodiment of the present application, the alarm processing rules are obtained from the configuration management database by the deployed first service module, or the alarm processing rules are generated in response to the user's alarm configuration instructions by using the configuration management interface. In this way, after the first service module obtains the alarm events from the alarm generation service and stores them in the storage database, the alarm processing rules and alarm events can be obtained from the storage data by the deployed second service module. In this way, the alarm events can be processed according to the alarm processing rules, thereby processing a large number of alarm events into a relatively small number of final events to characterize that the monitored server is in a corresponding alarm state. It can be seen that in this embodiment, the alarm processing rules obtained by the configuration management database or the configuration management interface can be dynamically configured in real time. Based on this, the large number of alarm events generated by the alarm generation service can be processed in real time with high performance, which can meet the convergence effect of the large number of alarm events on the server.
[0066] In one implementation, the second service module 102 may include the following structure: Figure 2 As shown in:
[0067] The event monitoring submodule 201 is used to monitor the event change status of the storage database, and obtain the corresponding alarm event from the storage database when the event change status indicates that there is an event change.
[0068] Among them, the event monitoring submodule 201 can be a Reflector component in the learning API service. The Reflector component can monitor the event change status of the alarm event in the storage database. The event change status indicates whether there is an alarm event with a state change such as addition, deletion, or modification. If the event change status indicates that there is an event change, the Reflector component requests the storage database to obtain the corresponding alarm event, such as a newly added alarm event.
[0069] The event aggregation submodule 202 is used to process the alarm events obtained by the event monitoring submodule 201 according to the alarm processing rules to obtain at least one final event.
[0070] Among them, the event aggregation submodule 202 can be a Handler component in the learning software API service. The Handler component processes the alarm events obtained by the Reflector component from the storage database according to alarm processing rules such as alarm convergence rules, alarm suppression rules, and alarm aggregation rules to obtain the final event. The final event can be used for output. In this way, through the convergence, suppression and aggregation of alarm events, a large number of alarm events can be converged into a small number of final events, so that the number of final events output is relatively small.
[0071] Based on the above implementation scheme, the second service module 102 may also include the following structure, such as Figure 3 As shown in:
[0072] The input and output queue 203 is used to store the alarm events obtained by the event monitoring submodule 201 from the storage database.
[0073] The alarm events in the input / output queue 203 may be provided to the event aggregation submodule 202 according to a first-in-first-out rule.
[0074] The input and output queue 203 may also be referred to as DeltaFIFO. For example, the Reflector component in the learning API service can monitor the event change status of the alarm event in the storage database. If the event change status indicates an event change, the Reflector component requests the storage database to obtain the corresponding alarm event and stores the alarm event in DeltaFIFO. After that, the Handler component processes the alarm event in DeltaFIFO according to the alarm processing rules to obtain the final event.
[0075] Based on the above implementation scheme, the second service module 102 may also include the following structure, such as Figure 4 As shown in:
[0076] The identifier storage submodule 204 is used to obtain the event identifier from the cache database.
[0077] After obtaining the alarm event generated by the alarm generation service, the first service module 101 stores the event identifier of the alarm event in the cache database, and the event monitoring submodule 201 monitors the event change status in the storage database according to the event identifier in the identifier storage submodule 204 .
[0078] For example, the event identifier may be a key or ID of an alarm event, etc., which is used to uniquely identify the corresponding alarm event.
[0079] In one implementation, the device in this embodiment may also include the following structure: Figure 5 As shown in:
[0080] The third service module 204 is used to collect system monitoring data corresponding to the monitored server; obtain server metadata from the configuration management database according to the tag information in the system monitoring data; then integrate the system monitoring data and the server metadata to obtain system integration data; transmit the system integration data to the alarm generation service, so that the alarm generation service can process the system integration data according to the monitoring rules to determine whether the system integration data meets the monitoring rules. If so, a corresponding alarm event is generated.
[0081] The system monitoring data can be obtained through an open source tool deployed on the monitored server for collecting system-level indicators, i.e., system monitoring data, such as the node-exporter program, and the third service module 204 obtains the system monitoring data collected by the open source tool. For example, the system monitoring data may include the following data content: basic system information, process and service status, custom application indicators, operating status of monitoring services, and security-related indicators.
[0082] Specifically, basic system information may include: CPU usage, memory usage, disk usage, network traffic; process and service status may include: number and status of running processes, startup status of each service, average system load, number of containers, running status, resource consumption, etc. Custom application indicators may include: application-specific monitoring indicators, such as request response time, request volume, error rate, etc.; monitoring service running status may include: network connectivity, HTTP service response status code, TCP port status, etc.; security-related indicators may include: security events, intrusion detection and other security-related monitoring indicators.
[0083] It should be noted that the server metadata may include: node (server) label, service label, owner, network tuple and other information.
[0084] In a specific implementation, the first service module 101 is further used to generate monitoring rules in response to monitoring configuration instructions, and provide the monitoring rules to the alarm generation service. In other words, the monitoring rules used by the alarm generation service are user-configured monitoring rules obtained by the first service module 101 using the configuration management interface. In this way, the alarm generation service can process the system integration data according to the monitoring rules to generate corresponding alarm events.
[0085] For example, the web service provides the user with a configuration interface of monitoring rules through the configuration management interface, and the user performs monitoring configuration operations on the configuration interface, so that the web service can obtain the monitoring configuration instructions through the configuration management interface and generate corresponding monitoring rules in response to the monitoring configuration instructions. In this way, dynamic adjustment of monitoring rules is achieved to meet corresponding business monitoring requirements.
[0086] In one implementation, the second service module 102 may publish the final event to an event bus, such as eventbus, to output the final event through the event bus.
[0087] For example, external systems such as email, WeChat, telephone and other alarm channels can subscribe to the corresponding alarm topic through the event bus to obtain the corresponding alarm events in real time.
[0088] refer to Figure 6 , is a flowchart of a distributed processing method for alarm events provided in an embodiment of the present application, which can be applied to electronic devices capable of data processing, such as terminals or servers. Figure 1 The distributed processing device for alarm events shown in the figure. The technical solution in this embodiment is mainly used to improve the convergence effect of alarm events.
[0089] Specifically, the method in this embodiment may include the following process:
[0090] Step 601: using the first service module, obtain the alarm event generated by the alarm generation service, and store the alarm event in a storage database.
[0091] Step 602: Utilize the first service module to obtain alarm processing rules and store the alarm processing rules in the storage database.
[0092] The alarm processing rules are obtained from a configuration management database, or are generated using a configuration management interface in response to alarm configuration instructions.
[0093] Step 603: using the second service module, obtain an alarm processing rule and a plurality of alarm events from a storage database; and process the alarm events according to the alarm processing rule to obtain at least one final event.
[0094] The final event indicates that the monitored server is in a corresponding alarm state.
[0095] According to the above technical solution, in a distributed processing method for alarm events provided by an embodiment of the present application, the alarm processing rules are obtained from the configuration management database by the deployed first service module, or the alarm processing rules are generated in response to the user's alarm configuration instructions by using the configuration management interface. In this way, after the first service module obtains the alarm events from the alarm generation service and stores them in the storage database, the alarm processing rules and alarm events can be obtained from the storage data by the deployed second service module. In this way, the alarm events can be processed according to the alarm processing rules, thereby processing a large number of alarm events into a relatively small number of final events to characterize that the monitored server is in a corresponding alarm state. It can be seen that in this embodiment, the alarm processing rules obtained by the configuration management database or the configuration management interface can be dynamically configured in real time. Based on this, the large number of alarm events generated by the alarm generation service can be processed in real time with high performance, which can meet the convergence effect of the large number of alarm events on the server.
[0096] In one implementation, the alarm processing rule includes at least one of the following:
[0097] An alarm convergence rule, wherein the alarm convergence rule is used to converge multiple alarm events into one alarm event according to a convergence label;
[0098] An alarm suppression rule, wherein the alarm suppression rule is used to retain alarm events with a high alarm level among alarm events with the same label;
[0099] Alarm aggregation rules, where the alarm aggregation rules are used to aggregate multiple alarm events into one alarm event according to an aggregation tag or aggregation content.
[0100] In one implementation, step 603 may be implemented in the following manner:
[0101] First, the event monitoring submodule in the second service module is used to monitor the event change status of the storage database. When the event change status indicates an event change, the corresponding alarm event is obtained from the storage database; then, the event aggregation submodule in the second service module is used to process the alarm events obtained by the event monitoring submodule according to the alarm processing rules to obtain at least one final event.
[0102] Based on the above implementation, step 603 may also include the following processing:
[0103] After using the event monitoring submodule in the second service module to obtain the corresponding alarm event from the storage database, the alarm event obtained by the event monitoring submodule from the storage database is stored in the input-output queue, and then the event aggregation submodule is used to obtain the alarm event from the input-output queue according to the first-in-first-out rule, and then the alarm event is processed according to the alarm processing rule to obtain at least one final event.
[0104] Based on the above implementation, step 603 may also include the following processing:
[0105] The event identifier is obtained from the cache database using the identifier storage submodule. Based on this, after the alarm event generated by the alarm generation service is obtained using the first service module, the event identifier of the alarm event is stored in the cache database. Based on this, in this embodiment, the event monitoring submodule is used to monitor the event change state of the storage database according to the event identifier in the identifier storage submodule.
[0106] In one implementation, the method in this embodiment may further include the following processing:
[0107] Using the third service module, collecting system monitoring data corresponding to the monitored server; obtaining server metadata from the configuration management database according to tag information in the system monitoring data; integrating the system monitoring data and the server metadata to obtain system integration data; transmitting the system integration data to the alarm generation service;
[0108] Based on this, in this embodiment, the first service module is used to generate monitoring rules in response to monitoring configuration instructions, and the monitoring rules are provided to the alarm generation service; the alarm generation service processes the system integration data according to the monitoring rules to generate alarm events.
[0109] In one implementation, the method in this embodiment may further include the following processing:
[0110] The final event is published to the event bus by using the second service module, so as to output the final event through the event bus.
[0111] An embodiment of the present application further provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:
[0112] The memory is used to store computer programs;
[0113] The processor is used to execute the computer program so that the electronic device can implement the distributed processing method of the alarm event as described above.
[0114] Specifically, refer to Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include but is not limited to fixed terminals such as mobile phones, notebook computers, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.
[0115] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage device 708 to a random access memory (RAM) 703. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 703. The processing device 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0116] Typically, the following devices may be connected to the I / O interface 705: an input device 706 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 707 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 708 including, for example, a memory card, a hard disk, etc.; and a communication device 709. The communication device 709 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0117] An embodiment of the present application also provides a computer program product including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements a distributed processing method for any alarm event provided in the embodiment of the present application.
[0118] A computer storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement a distributed processing method for any alarm event provided in an embodiment of the present application.
[0119] Taking the monitoring scenario of an Internet company as an example, the technical solution of this application is illustrated below:
[0120] At present, most Internet companies implement monitoring and alarming methods mainly in the following ways:
[0121] Prometheus (an open source monitoring system and time series database) stores the indicator data of the collection system, such as CPU utilization, memory usage, network traffic, etc., and stores this data in the form of time series in the built-in time series database. In addition, the Alertmanager module of Prometheus is responsible for receiving and processing alarms from Prometheus, and notifying and processing them according to pre-configured policies. However, the alarm processing rules of Prometheus are based on the PromQL language, which has certain limitations in flexibility and may be difficult to meet the needs of some complex scenarios.
[0122] Based on the above defects, it is necessary to introduce an alarm technology solution with real-time alarm event processing and interactive policy management to meet the challenges of the growing complex alarm scenarios. By connecting to the alarm events generated by Promethus and supporting dynamic configuration of alarm convergence rules, alarms are issued after complex aggregation, association and other calculation logics. Among them, high-performance processing of massive alarm events according to convergence rules is the key point.
[0123] In view of this, the present application provides a distributed computing scenario for monitoring events. By connecting to the alarm events generated by Promethus and supporting dynamic configuration of alarm convergence rules, alarms are issued after complex aggregation, association and other computing logics, so as to achieve high-performance real-time processing of massive alarm events according to convergence rules, so as to effectively respond to the monitoring and alarm challenges in modern IT operation and maintenance, and improve the reliability and maintainability of the system.
[0124] To achieve the above objectives, the present application provides the following distributed computing method for monitoring events. The technical solution in the present application is as follows:
[0125] S1: Data collection: Deploy the node-exporter program on each server node to report the collected metrics (system monitoring data) to the collector service (i.e. the third service module mentioned above) through the external access layer.
[0126] S2: Data aggregation and storage: The collector service uses the host name on the metric label (such as host="server1") to request CMDB to obtain machine metadata (i.e., server metadata in the previous article). The metadata includes node labels, service labels, owners, network tuples, and other information. The metadata is integrated with the metric and the processed metric is sent to Prometheus (i.e., Fig. 9 The data is stored on an external data source in .
[0127] S3: Monitoring rule configuration and management: Through the configuration management interface provided by the Web service of the monitoring center (i.e., the first service module mentioned above), users can configure and dynamically adjust monitoring rules and alarm convergence rules (there can also be alarm suppression rules, alarm aggregation rules, etc.) according to different business monitoring needs. At the same time, the Web service requests CMDB to obtain the business line metadata to which the rule belongs based on the monitoring rule tag, so as to facilitate subsequent alarm message notifications to the business members corresponding to the business line.
[0128] S4: Monitoring rules take effect: The Web service writes the user's monitoring rules into the Promethus configuration file through the monitoring configuration interface provided by Prometheus, and stores the user's alarm convergence rules persistently in the database.
[0129] S5: Alarm event transfer: When the metric data collected by Promethus meets the user's monitoring rules, a large number of alarm events are generated. The Web service monitors the generation of alarm events through the Webhook interface provided by Promethus, writes the alarm event data to the database, and writes the alarm event ID to the cache.
[0130] S6: Alarm event processing and notification: The learning API service of the monitoring center (i.e., the second service module mentioned above) performs intelligent processing on alarm events, aggregates operations based on the alarm convergence rules in the database, and sends alarm notifications. The key process of this patent is that the learning API service of the monitoring center uses message queues and event buses for distributed processing of alarm events and dynamic acquisition of alarm convergence rules.
[0131] In the specific implementation, Figure 8 In the high-performance interactive alarm processing process based on a distributed architecture proposed in this application, the system architecture diagram mainly includes the following structure:
[0132] (1) External access layer: The external access layer includes Ingress and other external access devices, which are responsible for receiving external monitoring data and alarm events. These data and events are usually generated by various monitoring tools and systems, such as Prometheus. The main task of this layer is to ensure that data can enter the monitoring platform smoothly.
[0133] (2) Monitoring center: It consists of four parts: learning API service, database, Web service, and collector service. The contents are as follows:
[0134] a. Learningware API service: the core part of the monitoring platform, responsible for receiving and processing various alarm events (events), does not contain specific business data but only some algorithms or logic. External system access only cares about input and output and does not care about the underlying computing logic. It includes the following subcomponents:
[0135] a.1Controller is responsible for obtaining monitoring data and alarm events from the database;
[0136] a.2 Manage incremental changes in data through mechanisms such as Reflector and DeltaFIFO;
[0137] a.3Handler processes alarm events, including primary alarm rules and advanced intelligent alarm systems, and publishes the processed events to the Event Bus;
[0138] a.4Event Bus is responsible for publishing and subscribing events, ensuring efficient communication and collaboration between components;
[0139] a.5Counter deploys suppression aggregation rules (i.e., suppression rules, aggregation rules, convergence rules, etc. mentioned above) and intelligent detection system to count and compile various indicators of alarm events and provide statistical data for alarm processing.
[0140] b. Database (DB layer and cache layer): stores and manages configuration metadata of WEB services (collectively referred to as alarm rules configured by users) and hot data (i.e. data that is frequently queried).
[0141] c. Web service (including alarm management module and configuration management module): responsible for providing configuration management and alarm management interfaces, as well as integration with external systems (for web services, other services are called external services). It includes the following subcomponents: the configuration management interface provides a friendly user interface and connects to CMDB to obtain convergence strategies, i.e., alarm processing rules, such as alarm convergence rules, supports dynamic configuration and management of monitoring and alarm strategies, and allows operation and maintenance engineers to adjust monitoring and alarm rules according to business needs.
[0142] d. Collector service (agent management and configuration management): responsible for collecting system performance indicators from various monitoring nodes and sending these data to Promethus storage.
[0143] e. Configuration management (CMDB).
[0144] (3) External data source: Compared with the monitoring center, it serves as the external input source of monitoring data and the output source of collector services. It mainly provides storage and data source for system performance and alarm events.
[0145] (4) Collector client (node-ecporter): Receives system performance indicator data from monitoring nodes, formats the data and sends it to the monitoring center for storage. It is the initial collection point for monitoring data and ensures the accuracy and timeliness of the data.
[0146] like Fig. 9 As shown in , the overall flow chart of the system proposed in this application mainly includes the following steps in the high-performance interactive alarm processing process based on the distributed architecture:
[0147] 1. Data collection: Each server node deploys the node-exporter program, which is used to collect system-level performance indicator data, such as CPU utilization, memory usage, disk I / O, and network traffic. The data collected by Node-exporter is uploaded to the collector service through the external access layer.
[0148] 2. Data aggregation and storage: After the collector service receives the metric (system monitoring data), it requests the CMDB to obtain the machine's metadata based on the host name on the metric label, including node labels, service labels, owners, network tuples, etc. The processed metric data is sent to Prometheus for storage, and Prometheus stores the data in its built-in time series database for subsequent analysis and processing.
[0149] 3. Configuration and management of monitoring rules: Through the configuration management interface provided by the Web service of the monitoring platform, users can configure and dynamically adjust monitoring rules and alarm convergence rules according to different business monitoring requirements. The Web service will request CMDB to obtain the business line metadata to which the rule belongs based on the monitoring rule tag, so as to facilitate the subsequent alarm message notification to the corresponding business members.
[0150] 4. Effectiveness of monitoring rules: The Web service writes the user-configured monitoring rules into the Prometheus configuration file through the monitoring configuration interface provided by Prometheus, and stores the user's alarm convergence rules in the database for dynamic calling during the alarm event processing.
[0151] 5. Alarm event transfer: When the metric data collected by Prometheus meets the user's monitoring rules, a large number of alarm events will be generated. The Web service monitors the alarm events through the Webhook interface provided by Prometheus, writes these event data to the database, and writes the alarm events to the cache to improve the speed and efficiency of subsequent processing.
[0152] 6. Alarm event processing and notification: The learning API service of the monitoring center intelligently processes the received alarm events, and aggregates the alarm events according to the alarm convergence rules in the database, reducing repeated and redundant alarms and improving alarm processing efficiency.
[0153] It should be noted that the Learningware API service uses self-developed message queues and event buses to perform distributed processing of alarm events and dynamically obtain alarm convergence rules. Figure 8 As shown in , the component interaction process proposed in this application in the process of processing alarm events and obtaining alarm convergence configuration in the learning software API of the monitoring middle station mainly includes the following steps:
[0154] 1) When Reflector detects data changes in the event table, it triggers a response change event (CRUD, i.e. Create, Read, Update, Delete) and stores the resource object in the DeltaFIFO in memory.
[0155] 2) DeltaFIFO is a first-in-first-out message queue used to temporarily store the changed data monitored by Reflector to ensure that the data is processed in order.
[0156] 3) Reflector stores the resource objects consumed from DeltaFIFO in Indexer (the identifier storage module mentioned above). Indexer is completely consistent with Redis's event Indexer data (obj), avoiding reading data from remote Redis every time and reducing the pressure on Redis and the learning software API service cluster.
[0157] 4) After the Handler monitors the message out of the queue, it generates it to the event bus. The Handler aggregates the alarm events according to the alarm convergence rules in the database and sends the processed alarm events to the notification system.
[0158] Based on the above scheme, the following optimization scheme is proposed:
[0159] like Fig.10 As shown in , the component interaction process proposed in this application in the process of processing alarm events and obtaining alarm convergence configuration in the learning software API of the monitoring middle station mainly includes the following steps:
[0160] ① and ② monitor changes: Reflector monitors the event table in the database according to the event ID in the cache. When the alarm event data in the table changes (such as add, delete, modify, and query operations), a change event will be triggered. For example, in step 5, "alarm event transfer", the WEB service obtains a Promethus alarm source event and writes it to the event table in the database. Reflector asynchronously monitors the addition of an alarm event source in the event table, and triggers a new event in the ListAndWatch function.
[0161] ③Store in indexer: Reflector takes these event IDs from DeltaFIFO and stores them in a storage called Indexer. The event ID of Indexer is the same as the event ID in Redis, so there is no need to read data from Redis every time, which reduces the pressure on Redis and API services.
[0162] ④ Temporary storage: Reflector stores these change events in a first-in-first-out queue called DeltaFIFO, which is used to temporarily store these changed data to ensure that the data is processed in the order in which they occur.
[0163] ⑤, ⑥ and ⑦ event processing and notification: When the Handler listens to messages coming out of the queue, it will put these messages on the event bus. The Handler processes these events according to the suppression rules in the database and the CMDB convergence rules, and then sends the processed alarms to the notification system.
[0164] In summary, the core points of the technical solution of this application are as follows:
[0165] (1) Alarm events generated by connecting to Promethus:
[0166] Implementation method: It includes two core parts: Metric (monitoring data) linkage CMDB data assembly, and webhook monitoring and docking with Promethus alarm event source. The execution subject is: the web service of the monitoring center.
[0167] (2) Support dynamic configuration of alarm convergence rules:
[0168] Implementation method: According to Fig. 9 There are two ways to get the overall timing diagram of the system. Fig. 9 In step 3, the suppression rules configured by the user through the web interface are persistently stored in the database and can be obtained by reading the database when the alarm event is calculated. The second approach is to use CMDB as the IT infrastructure configuration management center to provide dynamic tag-based convergence strategies. The execution subjects are: web services and CMDB of the monitoring center.
[0169] (3) Complex aggregation, association and other calculation logic:
[0170] Implementation method: Figure 8 The workflow of the learningware API service in the middle system architecture. The execution subjects are: monitoring the web service of the middle platform and the components of the learningware API service.
[0171] It can be seen that the alarm convergence mechanism introduced in this application can aggregate and associate these redundant alarms, greatly reduce the number of alarms, directly improve the alarm processing efficiency of operation and maintenance engineers, reduce false alarms and missed alarms, and greatly improve the accuracy of alarms. Furthermore, this application can timely discover and handle potential hidden dangers and anomalies, ensure the continuous availability and stability of services, thereby reducing system downtime, thereby improving user experience and satisfaction. This will significantly enhance the market competitiveness and revenue of enterprises.
[0172] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.
[0173] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0174] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0175] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
Claims
1. A distributed processing device for alarm events, It is characterized in that At least comprising: a first service module, and a second service module; The first service module is used to obtain an alarm event generated by the alarm generation service and store the alarm event in a storage database; and obtain an alarm processing rule and store the alarm processing rule in the storage database; wherein the alarm processing rule is obtained from a configuration management database, or is generated using a configuration management interface in response to an alarm configuration instruction; The second service module is used to obtain the alarm processing rules and multiple alarm events from the storage database; process the alarm events according to the alarm processing rules to obtain at least one final event; the final event indicates that the monitored server is in a corresponding alarm state.
2. The device according to claim 1, It is characterized in that The second service module includes: An event monitoring submodule, used for monitoring the event change state of the storage database, and obtaining a corresponding alarm event from the storage database when the event change state indicates an event change; The event aggregation submodule is used to process the alarm events obtained by the event monitoring submodule according to the alarm processing rules to obtain at least one final event.
3. The device according to claim 2, It is characterized in that The second service module also includes: The input-output queue is used to store the alarm events obtained by the event monitoring submodule from the storage database; the alarm events in the input-output queue are provided to the event aggregation submodule according to the first-in-first-out rule.
4. The device according to claim 2, It is characterized in that The second service module also includes: an identification storage submodule, configured to obtain an event identification from a cache database; after obtaining the alarm event generated by the alarm generation service, the first service module stores the event identification of the alarm event in the cache database; The event monitoring submodule monitors the event change status of the storage database according to the event identifier in the identifier storage submodule.
5. The device according to claim 1 or 2, It is characterized in that Also includes: The third service module is used to collect system monitoring data corresponding to the monitored server; Obtaining server metadata from the configuration management database according to tag information in the system monitoring data; Integrating the system monitoring data and the server metadata to obtain system integration data; transmitting the system integration data to the alarm generation service; The first service module is further used for: generating monitoring rules in response to monitoring configuration instructions, and providing the monitoring rules to the alarm generation service; the alarm generation service processes the system integration data according to the monitoring rules to generate the alarm event.
6. The device according to claim 1 or 2, It is characterized in that The second service module publishes the final event to an event bus to output the final event through the event bus.
7. A distributed processing method for alarm events, It is characterized in that include: Using the first service module, obtaining an alarm event generated by the alarm generation service, and storing the alarm event in a storage database; Using the first service module, obtaining an alarm processing rule and storing the alarm processing rule in the storage database; wherein the alarm processing rule is obtained from a configuration management database, or is generated using a configuration management interface in response to an alarm configuration instruction; The second service module is used to obtain the alarm processing rules and the multiple alarm events from the storage database; the alarm events are processed according to the alarm processing rules to obtain at least one final event; the final event indicates that the monitored server is in a corresponding alarm state.
8. The method according to claim 7, It is characterized in that The alarm processing rule includes at least one of the following: An alarm convergence rule, wherein the alarm convergence rule is used to converge multiple alarm events into one alarm event according to a convergence label; An alarm suppression rule, wherein the alarm suppression rule is used to retain alarm events with a high alarm level among alarm events with the same label; Alarm aggregation rules, where the alarm aggregation rules are used to aggregate multiple alarm events into one alarm event according to an aggregation tag or aggregation content.
9. An electronic device, It is characterized in that The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the distributed processing method for alarm events as described in any one of claims 7 to 8.
10. A computer storage medium, It is characterized in that The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the distributed processing method for alarm events as described in any one of claims 7 to 8.
