Emergency scheme indexing method and system based on server early warning

Abstract

The invention relates to the technical field of operation and maintenance management, in particular to an emergency scheme indexing method and system based on server early warning. The method comprises the steps of collecting server data indexes; alarming the server data index based on an early warning rule to obtain alarm information; the obtained alarm information is stored in an MYSQL through a webhook; carrying out emergency plan search by utilizing a B + tree index based on the alarm information; and a corresponding emergency plan is obtained. According to the invention, the emergency plan content is associated with the alarm rule, and intelligent emergency response is realized. And corresponding emergency plans are automatically matched according to different alarm information, so that the efficiency and accuracy of fault handling are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of operation and maintenance management, and in particular to an emergency plan indexing method and system based on server early warning. Background Art

[0002] Prometheus is an open source monitoring and early warning tool that can collect various server indicators and provide flexible early warning functions. Through this monitoring tool, comprehensive monitoring of the server can be achieved.

[0003] In the existing technology, Prometheus stores the collected monitoring data locally by default, and performs query and calculation locally. This may cause some problems for large-scale long-term storage and horizontal expansion. Prometheus mainly focuses on real-time monitoring and short-term storage, and has relatively limited support for long-term storage and historical data query. According to the alarm information prompts of Prometheus for server monitoring, it may be impossible to quickly find the optimal solution to handle the alarm. Summary of the invention

[0004] In order to solve the above-mentioned problems, the present invention provides an emergency plan indexing method and system based on server early warning.

[0005] In a first aspect, the present invention provides an emergency plan indexing method based on server early warning, which adopts the following technical solution: A method for indexing emergency plans based on server early warning, comprising: Collect server data indicators; Alarm the server data indicators based on the early warning rules and obtain the alarm information; Store the acquired alarm information in MYSQL through webhook; Use B+ tree index to search for emergency plans based on alarm information; Draw up corresponding emergency plans.

[0006] Furthermore, the collecting of server data indicators includes periodically pulling indicator data from PushGateway based on Prometheus Server, and using a remote storage adapter of Prometheus to push the indicator data to an external influxdb database for data storage.

[0007] Furthermore, the server data indicators are alarmed based on the alarm rules to obtain alarm information, including alarming by analyzing the data indicators and setting thresholds, wherein the data name is used as a unique identifier.

[0008] Furthermore, the acquired alarm information is stored in MYSQL via webhook, wherein the Dispatcher in the Alertmanager module periodically acquires new alarm information from the webhook and routes the alarm information to corresponding groups according to the routing tree in the predefined YAML configuration.

[0009] Furthermore, the alarm information is routed to the corresponding group according to the routing tree in the predefined YAML configuration, including using the group to perform a flush operation within a set time interval, and processing and delivering the alarm information to the MySQL database through the notification pipeline in the group.

[0010] Furthermore, the emergency plan search based on the alarm information using the B+ tree index includes constructing a primary key, a unique index and a common index of a table for the alarm information in the MySQL database based on the B+ tree index.

[0011] Furthermore, the emergency plan search based on the alarm information using the B+ tree index includes establishing an index based on the alarm information, wherein different index establishment methods are adopted according to the attribute values ​​of the alarm information; extracting keywords from the alarm information, submitting the keywords to the index, obtaining the index position, and obtaining the index content based on the index position.

[0012] In a second aspect, an emergency plan indexing system based on server early warning includes: The data acquisition module is configured to collect server data indicators; The alarm rule module is configured to alarm the server data indicators based on the early warning rules to obtain alarm information; The storage module is configured to store the acquired alarm information system server calling the API interface in MYSQL; The plan search module is configured to search for emergency plans based on the alarm information using a B+ tree index and generate corresponding emergency plans.

[0013] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor of a terminal device and executing the emergency plan indexing method based on server warning.

[0014] In a fourth aspect, the present invention provides a terminal device including a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, and the instructions are suitable for being loaded by the processor and executing the emergency plan indexing method based on server warning.

[0015] In summary, the present invention has the following beneficial technical effects: The present invention can specify the URL address of Remote Write in the Prometheus configuration file for long-term storage of large-scale data in Prometheus. Once the configuration item is set, Prometheus sends the collected sample data to the adapter in the form of HTTP. The system uses Influxdb for remote storage.

[0016] Improve availability and use Prometheus's federation function and multi-instance deployment to achieve high availability. Through the federation function, multiple Prometheus instances can be aggregated into a whole for global query and monitoring.

[0017] By setting up an emergency plan library and matching emergency plans according to alarm information, the search speed of emergency plans is improved.

[0018] Intelligent emergency plan: Associate the emergency plan content with the alarm rules to achieve intelligent emergency response. Automatically match the corresponding emergency plan according to different alarm information to improve the efficiency and accuracy of fault response.

[0019] Fault self-healing mechanism: Implement fault self-healing mechanism based on alarm rules and historical data, and automatically identify and handle faults through automated scripts or tools, reduce manual intervention, and improve system stability and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of an emergency plan indexing method based on server early warning according to embodiment 1 of the present invention. DETAILED DESCRIPTION

[0021] The present invention is further described in detail below in conjunction with the accompanying drawings.

[0022] Example 1 Reference Figure 1 , a contingency plan indexing method based on server early warning in this embodiment includes: Collect server data indicators; Alarm the server data indicators based on the early warning rules and obtain the alarm information; The obtained alarm information is notified through webhook (web callback or push API interface of http service), and the system server calls the API interface and stores it in MYSQL; Use B+ tree index to search for emergency plans based on alarm information; Draw up corresponding emergency plans.

[0023] The server data indicators are collected, including periodically pulling indicator data from PushGateway (an intermediate service that allows indicators to be pushed from jobs that cannot be captured) based on Prometheus Server, and using Prometheus's remote storage adapter to push the indicator data to an external influxdb database for data storage.

[0024] The server data indicators are alarmed based on the early warning rules to obtain alarm information, including issuing early warnings by analyzing the data indicators and setting thresholds, wherein the data name is used as a unique identifier.

[0025] The obtained alarm information is stored in MYSQL, wherein the Dispatcher (the component responsible for receiving, routing and sending alarms) in the Alertmanager (an independent alarm module for processing alarms generated by the Prometheus monitoring system) module periodically obtains new alarm information from the webhook (a web callback or a push API interface for http services) and routes the alarm information to the corresponding group according to the routing tree in the pre-defined YAML configuration.

[0026] The alarm information is routed to the corresponding group according to the routing tree in the predefined YAML configuration, including using the group to perform a flush operation within a set time interval, and processing and delivering the alarm information to the MySQL database through the notification pipeline in the group.

[0027] The emergency plan search based on the alarm information using the B+ tree index includes constructing the primary key, unique index and common index of the table for the alarm information in the MySQL database based on the B+ tree index.

[0028] Specifically, the following steps are included: S1. Collect server data indicators; Among them, the monitoring data collection methods are mainly divided into push and pull. The push mode means that the monitored point actively pushes status data to the PushGateway (an intermediary service that allows indicators to be pushed from jobs that cannot be crawled) component, and the Prometheus Server regularly pulls indicator data from the PushGateway (an intermediary service that allows indicators to be pushed from jobs that cannot be crawled).

[0029] Considering that data generally changes in time series and the collection frequency is high (may reach 1 second for high monitoring requirements), Prometheus uses local disk to store indicator data, which may face some problems for large-scale long-term storage and horizontal expansion. To solve these problems, use Prometheus's Remote Storage Adapter to push indicator data to an external influxdb database for data storage.

[0030] In the Prometheus ecosystem, alerts are generated in the Prometheus Server by calculating alert rules. AlertManager (an independent alert module used to process alerts generated by the Prometheus monitoring system) receives alert message requests generated by Prometheus, and then performs certain processing, such as grouping, silencing, etc., and then sends them to the corresponding receiver for processing through the alert routing.

[0031] S2. Alarm the server data indicators based on the warning rules to obtain the alarm information; Among them, Prometheus warning rules mainly rely on the collection of indicators (metric), and the purpose of warning is achieved by analyzing the indicators and setting thresholds. Each warning rule consists of 5 parts. Name (alert), a unique identifier for similar rules, used to distinguish different rules; trigger condition (expr), this is a PromQL expression, for example, the CPU usage exceeds 70%, and the status of the warning is Inactive before the trigger condition is met; duration (for), for example, the CPU usage exceeds 70% for 30 seconds. Within 30 seconds, the warning status is pending, and it enters the firing state after 30 seconds; labels (labels), labels mark the warnings for classification and positioning. You can filter and locate according to the labels; annotations (annotations), provide a detailed description and additional instructions for the warning, which can help operation and maintenance personnel better understand and handle the warning situation. This system can directly complete the operation of adding rules in the user interface without manually creating rule configuration files, writing rules, configuring Prometheus, and restarting Prometheus to make the new rules effective. In the alarm rule template configuration page, you can see the server general template provided by the system, which can be copied, modified, etc. S3. Notify the obtained alarm information through webhook (web callback or http service push API interface) and store it in MYSQL; Among them, the alarm sent by Prometheus will first be notified through webhook (web callback or push API interface of http service), and the system server will call the API interface to store the alarm information in MYSQL.

[0032] In Alertmanager (an independent alert module for processing alerts generated by the Prometheus monitoring system), there is a separate goroutine (a lightweight thread for concurrent execution of Go programs) called Dispatcher (a component responsible for receiving, routing, and sending alerts). Dispatcher (a component responsible for receiving, routing, and sending alerts) periodically gets new alerts from webhooks (web callbacks or push API interfaces for http services) and routes the alerts to the corresponding groups according to the routing tree in the pre-defined YAML configuration.

[0033] The group will be flushed after a set time interval (group_interval) (the interval is the group_interval in the configuration parameter). The flush operation means that the group alarm will enter a chain processing flow of the Notification Pipeline.

[0034] The notification pipeline determines the sending target for this group of alerts based on the pre-defined configuration, and performs operations such as suppression logic, silencing logic, deduplication logic, sending and retry logic, and finally realizes the delivery of the alert.

[0035] Through the internal architecture diagram of Alertmanager (an independent alarm module used to handle alarms generated by the Prometheus monitoring system) and the above description, the alarm process can be more clearly understood. Prometheus sends alarms to Alertmanager (an independent alarm module used to handle alarms generated by the Prometheus monitoring system), and Alertmanager (an independent alarm module used to handle alarms generated by the Prometheus monitoring system) notifies through webhook (web callback or push API interface of http service). Dispatcher (the component responsible for receiving, routing and sending alarms) regularly pulls new alarms from webhook and routes them to the corresponding groups according to the routing tree. The groups are refreshed regularly, and then processed and delivered through the notification pipeline. This process ultimately ensures the timely processing and sending of alarms.

[0036] S4. Search for emergency plans based on alarm information using B+ tree index; Among them, an index is essentially a data structure used to improve the efficiency of retrieving target data. The system database uses MySQL. In the MySQL database, B+ tree indexes are commonly used to construct the primary key, unique index, and ordinary index of a table. By using B+ tree indexes, the target data can be quickly located, the number of disk I / O operations can be reduced, and the efficiency of database queries can be improved. Characteristics of B+ tree: Each node in the tree can contain at most m children (m >= 2); Except for the root node and leaf nodes, each other node has at least [ceil(m / 2)] children (where ceil(x) is a ceiling function); If the root node is not a leaf node, it has at least 2 children (special case: the root node without children, that is, the root node is a leaf node, and the whole tree has only one root node); All leaf nodes appear on the same layer (the bottom layer). Leaf nodes are external nodes that store content, that is, key and value; Other nodes are internal nodes that store indexes, that is, key and next; The keywords of internal nodes key: K[1], K[2], …, K[M - 1]; And K[i] < K[i + 1]; The pointers of content nodes next: P[1], P[2], …, P[M]; Among them, P[1] points to the subtree with keywords less than K[1], P[M] points to the subtree with keywords greater than K[M - 1], and other P[i] point to the subtree with keywords belonging to (K[i - 1], K[i]).

[0037] Application of B+ tree. When the system detects a server warning, detailed warning information, including the warning time, warning reason, etc., will be displayed on the page. This information can help the operation and maintenance personnel quickly understand the specific situation of the server problem. In addition, the system also provides an emergency plan management page, and the administrator can quickly retrieve the corresponding emergency plan according to the warning keywords. In this way, when a warning occurs, the administrator can quickly find the corresponding emergency plan and take appropriate measures to deal with the problem to minimize potential risks and impacts.

[0038] Specifically, the alarm data type becomes complex, usually character type and floating point type, with obvious temporal and structural characteristics. When searching for emergency plans based on alarm information matching, there is a great possibility that multiple types of data need to be jointly queried. At this time, searching by indexing can significantly improve the efficiency of the query. Step 1: Carefully analyze the data that needs to be indexed and determine which attribute values ​​need to be indexed; Step 2: According to the data type of the attribute value, adopt different indexing methods: If the attribute value is character type, an inverted index (InvertedIndex) should be created for it. Inverted index can efficiently support operations such as full-text search; If the attribute value is numeric, a B+ tree index should be created. B+ tree index can efficiently support operations such as range query and sorting. Repeat the first step. If there is already an index for the data attribute, there is no need to create a new node at the top level of the index. You only need to add the data to the existing index structure. Repeat the above steps until all data have corresponding indexes, indicating that the entire operation is completed. Query algorithm: First, carefully study the query conditions, extract the keywords, and submit these keywords to the index dictionary. If the index flag is False, it means that the query result is empty, that is, there is no required data in the index file. If it is True, it is necessary to determine the data type of the returned result, locate the corresponding index position according to its type, and read the vocabulary number. Then, read the B+ tree index or inverted index content through the vocabulary number, effectively integrate the search content, and reasonably sort the query results to obtain the final query result.

[0039] S5. Draw up a corresponding emergency plan.

[0040] As a further embodiment, (1) Deploy and install Prometheus to query and calculate the data pushed by other hosts.

[0041] (2) Deploy Prometheus and node_exporter.

[0042] (3) Write the existing Prometheus data into the influxdb library of the operation and maintenance management system to achieve data docking.

[0043] (4) Modify the Prometheus configuration file, add the global application code, find the global label in the configuration file, and add the external_labels item to distinguish them.

[0044] (5) Upload the self-signed certificate provided by the system to the specified path of the server where Prometheus is located. If Prometheus is deployed in a container, mount the certificate to the specified path in the container.

[0045] (6) Install node_exporter. After Prometheus is installed, confirm whether each exporter can obtain monitoring data normally. If it can obtain monitoring data normally, create a monitoring method on the system.

[0046] (7) Configure alarm rules in the system and set alarm thresholds.

[0047] (8) Input emergency plans for different warning information.

[0048] As a further embodiment, Due to the storage limitation of a single node in Prometheus, large batches of data cannot be stored for a long time. To solve this problem, Prometheus provides a remote read and write interface and selects a suitable time series database to achieve long-term storage of large-scale data. After adding a remote write address in the Prometheus configuration file and configuring the target address of remote_write, it will read the data, write the sampled data to the memory queue of each shard, and finally initiate a request to the remote target address.

[0049] The federation mode allows Prometheus servers to grab selected time series data from other Prometheus servers. The main Prometheus instance integrates the monitoring data, and the sub-Prometheus instance remotely writes it to InfluxDB. The integrated monitoring data is read remotely by the main Prometheus instance to achieve centralized management and unified query of monitoring data. This architecture can effectively distribute the monitoring system, making the collection, integration and query of monitoring data more flexible and efficient. The sub-prometheus in each area is responsible for collecting the monitoring indicators in the area and pushing them to InfluxDB. The main Prometheus can obtain the global monitoring view for unified analysis and alarm. Through the federation mechanism, Prometheus can achieve cross-cluster monitoring data integration and improve the scalability and maintainability of the monitoring system.

[0050] Example 2 This embodiment provides an emergency plan indexing system based on server early warning, including: The data acquisition module is configured to collect server data indicators; The alarm rule module is configured to alarm the server data indicators based on the early warning rules to obtain alarm information; The storage module is configured to store the acquired alert information in the AlertProvider; The plan search module is configured to search for emergency plans based on the alarm information using a B+ tree index and generate corresponding emergency plans.

[0051] A computer-readable storage medium stores a plurality of instructions, wherein the instructions are suitable for being loaded and executed by a processor of a terminal device, for example, a method for indexing emergency plans based on server early warning.

[0052] A terminal device includes a processor and a computer-readable storage medium, wherein the processor is used to implement various instructions; the computer-readable storage medium is used to store multiple instructions, wherein the instructions are suitable for being loaded by the processor and executing the emergency plan indexing method based on server warning.

[0053] The above are all preferred embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for indexing emergency plans based on server early warning, characterized in that: include: Collect server data indicators; Alarm the server data indicators based on the early warning rules and obtain the alarm information; Store the acquired alarm information in MYSQL through webhook; Use B+ tree index to search for emergency plans based on alarm information; Draw up corresponding emergency plans.

2. The method for indexing emergency plans based on server early warning according to claim 1, characterized in that: The server data indicators are collected, including periodically pulling indicator data from PushGateway based on Prometheus Server, and using a remote storage adapter of Prometheus to push the indicator data to an external influxdb database for data storage.

3. The method for indexing emergency plans based on server early warning according to claim 2, characterized in that: The server data indicators are alarmed based on the early warning rules to obtain alarm information, including issuing early warnings by analyzing the data indicators and setting thresholds, wherein the data name is used as a unique identifier.

4. The method for indexing emergency plans based on server early warning according to claim 3, characterized in that: The obtained alarm information is stored in MYSQL through webhook, wherein the Dispatcher in the Alertmanager module periodically obtains new alarm information from the webhook and routes the alarm information to the corresponding group according to the routing tree in the pre-defined YAML configuration.

5. The method for indexing emergency plans based on server early warning according to claim 4, characterized in that: The alarm information is routed to the corresponding group according to the routing tree in the predefined YAML configuration, including using the group to perform a flush operation within a set time interval, and processing and delivering the alarm information to the MySQL database through the notification pipeline in the group.

6. The method for indexing emergency plans based on server early warning according to claim 5, characterized in that: The emergency plan search based on the alarm information using the B+ tree index includes constructing the primary key, unique index and common index of the table for the alarm information in the MySQL database based on the B+ tree index.

7. The method for indexing emergency plans based on server early warning according to claim 6, characterized in that: The emergency plan search based on alarm information using B+ tree index includes establishing an index based on the alarm information, wherein different index establishment methods are adopted according to the attribute values ​​of the alarm information; extracting keywords from the alarm information, submitting the keywords to the index, obtaining the index position, and obtaining the index content based on the index position.

8. An emergency plan indexing system based on server early warning, characterized in that: include: The data acquisition module is configured to collect server data indicators; The alarm rule module is configured to alarm the server data indicators based on the early warning rules to obtain alarm information; The storage module is configured to store the acquired alarm information system server calling the API interface in MYSQL; The plan search module is configured to search for emergency plans based on the alarm information using a B+ tree index and generate corresponding emergency plans.

9. A computer-readable storage medium storing a plurality of instructions, characterized in that: The instructions are suitable for being loaded by a processor of a terminal device and executed by a method for indexing emergency plans based on server warning as claimed in claim 1.

10. A terminal device, comprising a processor and a computer-readable storage medium, wherein the processor is used to implement each instruction; and the computer-readable storage medium is used to store multiple instructions, characterized in that: The instructions are suitable for being loaded by a processor and executing the emergency plan indexing method based on server early warning as claimed in claim 1.

Images