Historical monitoring alarm playback system for time series indicator data

Abstract

The present invention relates to a historical monitoring alarm playback system for time series indicator data, which belongs to the field of information technology and includes: an indicator historical monitoring management terminal starts a corresponding acquisition module, a strategy algorithm module, etc. according to the historical time period and indicator task configuration information specified by the user, and ensures that the strategy data of each acquisition cycle is calculated before the historical time series data by communicating messages with the strategy algorithm module and the state machine algorithm module. The strategy algorithm module calculates the dynamic threshold according to the strategy configuration and the strategy data, and pushes it to the state machine algorithm module. The state machine algorithm module calculates the historical time series data stream according to the dynamic threshold. For data that meets the trigger threshold, an alarm message is generated and pushed to the alarm data merging module or the alarm module for alarm processing. The present invention solves the problem of historical data monitoring alarm playback of indicator time series data from different data sources, and improves the accuracy of historical monitoring alarm playback.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a historical monitoring alarm playback system for time series indicator data. Background Art

[0002] In large-scale edge networks, the number of nodes can reach thousands or even tens of thousands. Each node is composed of multiple physical or virtual devices, so the total number of devices can reach tens of thousands or even hundreds of thousands. These nodes not only carry various services but also generate massive amounts of multi-dimensional data, including device performance metrics, application metrics, detection quality metrics, and basic service and network metrics.

[0003] Device performance indicators are key criteria for evaluating device performance and resource utilization, such as response time, throughput, CPU utilization, memory usage, and network traffic. Business application indicator data is business dimension indicator data generated by various applications, including container-level indicators and application traffic data from different channels. Probe quality indicator data is used to perceive the status of the edge network, including network probe results, such as TCP connection success rate, RTT, ICMP packet loss rate and latency, MTR routing information, and DNS resolution. Basic service and network indicator data includes the operating status and performance indicators of basic services, such as database cluster services, business scheduling services, message queue cluster services, as well as network data such as switch bandwidth and network type.

[0004] In traditional metric monitoring, the process of monitoring and alerting time-series metric data primarily relies on manually set thresholds to detect alarms. However, replaying alarms for historical monitoring of time-series data is a challenge. Due to the dynamic changes in the network environment in large-scale edge networks, the thresholds for the same metric at different time points may vary. This requires the ability to dynamically adjust the thresholds during replay to ensure alarm accuracy. However, existing monitoring systems often struggle to achieve this, resulting in inaccurate playback of historical monitoring alarms. Summary of the Invention

[0005] The technical problem to be solved by the present invention is: how to provide a historical monitoring alarm playback system for time series indicator data, aiming to solve at least one of the above technical problems.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] This application provides a historical monitoring alarm playback system for time series indicator data, which adopts the following technical solutions:

[0008] A historical monitoring alarm playback system for time series indicator data, comprising:

[0009] The indicator history monitoring management terminal is used to obtain the specified historical time period and indicator task configuration information of the indicator task, wherein the indicator task configuration information is either non-automatic calculation threshold monitoring configuration information or automatic calculation threshold monitoring configuration information;

[0010] When the indicator task configuration information of the indicator task is automatic calculation threshold monitoring configuration information, starting the indicator data collection end and the indicator historical data calculation end based on the specified historical time period and the automatic calculation threshold monitoring configuration information;

[0011] The indicator data collection end is configured to collect first indicator data based on the specified historical time period and the automatically calculated threshold monitoring configuration information, and send the first indicator data to the indicator data stream calculation end, wherein the first indicator data includes first historical time series indicator data and policy data, wherein the policy data is data collected according to a preset collection granularity and a preset collection interval;

[0012] The indicator data stream calculation end is used to perform first alarm playback processing on the indicator task according to the first indicator data and the automatic calculation threshold monitoring configuration information to generate first alarm information.

[0013] The beneficial effects of the present invention are: the system supports indicator tasks of non-automatic calculation of threshold monitoring configuration information and automatic calculation of threshold monitoring configuration information, making the monitoring method more flexible and the monitoring strategy can be adjusted according to actual needs. By collecting strategy data, it is possible to support calculations based on dynamic thresholds. Such dynamic thresholds can better reflect the actual operating conditions of the system than fixed thresholds, thereby improving the accuracy and effectiveness of alarms. In the process of historical monitoring alarm playback of time series indicator data, different modules (such as data acquisition end, calculation end, etc.) work together, each performing their own duties, and can process data in parallel, improving data processing efficiency, making alarm playback faster and more efficient. This modular design makes the system structure clear, decouples the modules, and facilitates expansion and maintenance.

[0014] Users can uniformly monitor and customize alarms and analysis for large-scale, historical metrics data from diverse sources. This allows them to identify potential issues, validate alarm rules, evaluate alarm responses, perform data analysis and troubleshooting, conduct forecasting and capacity planning, and optimize monitoring systems. The system is also pluggable and extensible, making it easy to add additional functionality, further reducing development, management, and operational costs.

[0015] Furthermore, it also includes:

[0016] The indicator history monitoring management terminal includes a scheduling control module:

[0017] The scheduling control module is configured to start the indicator data collection end and the indicator historical data calculation end based on the specified historical time period and the non-automatic calculation threshold monitoring configuration information when the indicator task configuration information of the indicator task is non-automatic calculation threshold monitoring configuration information;

[0018] The indicator data collection end is further configured to collect second indicator data based on the specified historical time period and the non-automatic calculation threshold monitoring configuration information, and send the second indicator data to the indicator data stream calculation end, where the second indicator data includes second historical time series indicator data;

[0019] The indicator data stream calculation end is further used to perform a second alarm playback process on the indicator task according to the second indicator data and the non-automatic calculation threshold monitoring configuration information to generate second alarm information.

[0020] The beneficial effects of adopting the above further scheme are:

[0021] For indicator tasks whose indicator task configuration information is non-automatic calculation of threshold monitoring configuration information, the system does not need to perform complex strategy data calculations and dynamic threshold settings, thereby simplifying the data processing process. By simplifying the data processing process, improving alarm processing efficiency, ensuring the accuracy of alarm information, and providing flexibility and scalability, the system can handle such tasks more efficiently and accurately to meet the different needs of users.

[0022] Furthermore, the automatic calculation threshold monitoring configuration information includes first basic configuration information and first collection configuration information. The first basic configuration information includes a task name, an identifier for enabling the automatic threshold calculation setting, and a dimension label name. The first collection configuration information includes a data source for the first indicator data, screening conditions for collecting the first indicator data, a processing method for the first indicator data, and a collection period for the first indicator data.

[0023] The indicator data collection terminal includes an indicator data collection module, an indicator data post-processing module and a collection and push module;

[0024] The indicator data collection module is configured to collect first historical time series indicator data based on the specified historical time period and the first collection configuration information;

[0025] Collecting policy data based on the specified historical time period, the first collection configuration information, and the first basic configuration;

[0026] The indicator data post-processing module is configured to perform a first encapsulation process on the first historical time series indicator data and the policy data based on the first acquisition configuration;

[0027] The collection and pushing module is used to store the first historical time series indicator data and the policy data after the first encapsulation process in a historical message queue, and send the historical message queue to the indicator data flow calculation end.

[0028] The beneficial effect of adopting this further solution is that the automatic calculation threshold monitoring configuration information includes the first basic configuration information and the first collection configuration information, allowing users to flexibly configure tasks according to actual needs. The first basic configuration information includes options such as the task name, automatic threshold calculation settings, and dimension label names, providing users with more fine-grained control. The first collection configuration information includes settings such as the data source, filtering conditions, post-processing method, and collection cycle, ensuring the accuracy and effectiveness of data collection. The indicator data collection terminal supports data collection from multiple data sources.

[0029] The collection module is responsible for collecting the first indicator data and the strategy data that automatically calculates the threshold value according to the data source of the first collection configuration of each indicator and the specified historical time period, avoiding unnecessary data redundancy and improving the efficiency and accuracy of data collection.

[0030] The indicator data post-processing module is responsible for packaging and processing the collected data to ensure the uniformity and standardization of the data format. This helps reduce errors and ambiguities in subsequent data processing and improves the efficiency and accuracy of data processing.

[0031] The historical message queue sends the first indicator data to the indicator data stream calculation end, ensuring the reliability and stability of data transmission. The historical message queue has a caching and fault-tolerance mechanism, which can ensure the integrity and order of data even in the case of network fluctuations or system busyness.

[0032] Furthermore, the indicator task configuration information also includes strategy calculation configuration information and state machine calculation configuration information, and the indicator data stream calculation end includes an indicator data distribution module, a strategy algorithm calculation module, and a state machine calculation module;

[0033] The indicator data distribution module is used to obtain the first historical indicator time series data and the strategy data in the historical message queue;

[0034] Pushing the policy data to a policy algorithm calculation module based on the automatically calculated threshold monitoring configuration information;

[0035] and obtaining a push data message sent by the indicator history monitoring management terminal, and sending the first historical indicator time series data to the state machine calculation module based on the push data message;

[0036] The policy threshold calculation module is used to calculate the dynamic threshold of the indicator task based on the policy data and the policy calculation configuration information of the automatic calculation threshold monitoring configuration information, and send the dynamic threshold to the state machine calculation module;

[0037] After the dynamic threshold is successfully sent to the state machine calculation module, a message indicating successful policy processing is returned to the indicator history monitoring management terminal;

[0038] The state machine calculation module is configured to update the threshold of the state machine calculation configuration information based on the dynamic threshold, and generate first historical alarm information based on the updated state machine calculation configuration and the first historical indicator time series data, and

[0039] The first historical alarm message is sent to an alarm message queue.

[0040] The beneficial effect of adopting this further solution is that by introducing policy calculation configuration and state machine calculation configuration, the system can dynamically calculate thresholds based on policy data and generate alarm information based on the updated state machine configuration. This dynamic threshold calculation method better reflects the actual operating status of the system than fixed thresholds, thereby improving the accuracy and effectiveness of alarms.

[0041] The introduction of policy calculation configuration and state machine calculation configuration enables the system to be flexibly configured according to different indicator task requirements. Users can adjust the policy algorithm and state machine rules according to actual conditions to adapt to different monitoring scenarios and requirements.

[0042] The indicator historical data distribution module, policy threshold calculation module and state machine calculation module work together to achieve an efficient process of data distribution, processing and alarm generation.

[0043] Furthermore, the indicator data stream calculation end also includes a historical alarm data merging module and a historical alarm data label expansion module:

[0044] The historical alarm data merging module is used to merge the historical alarm information on the first historical alarm information based on the merged alarm configuration information in the indicator task configuration information when there is merged alarm configuration information in the automatic calculation threshold monitoring configuration information, and send the first historical alarm message after the merged historical alarm information processing to the alarm message queue or the historical alarm data tag expansion module;

[0045] The historical alarm data tag extension module is used to perform extended tag processing on the first indicator historical alarm information based on the configuration of the extended tag in the indicator task configuration information when there is an extended tag configuration in the automatic calculation threshold monitoring configuration information, and send the first historical alarm message after extended tag processing to the alarm message queue.

[0046] The beneficial effect of adopting this further solution is that when the automatically calculated threshold monitoring configuration information includes a merged alarm configuration, the historical alarm data merging module can merge the first historical alarm information. The system can consolidate multiple related alarm information into one, avoiding redundancy and duplication of alarm information, allowing users to more clearly understand the overall alarm situation, and improving the readability and effectiveness of alarm information.

[0047] By expanding the alarm data's tags, we can provide richer alarm information, helping users more accurately judge and handle alarm situations. Users can quickly understand the source and importance of the alarm based on the information in the expanded tags, and take appropriate measures to address the alarm situation.

[0048] Furthermore, the indicator data stream calculation also includes an indicator history task message subscription module;

[0049] The indicator history task message subscription module is used to obtain the indicator task configuration information sent by the indicator history monitoring management terminal.

[0050] The beneficial effect of adopting this further solution is that, through the indicator history task message subscription module, it subscribes to the created or updated indicator task configuration information and the policy data pushed by the program for automatic calculation of specific indicator thresholds. Whenever a new indicator task is created or an existing indicator task is updated, the subscription module can immediately capture these changes and respond accordingly. This real-time and dynamic nature ensures the accuracy and timeliness of indicator data stream calculations.

[0051] Furthermore, it also includes:

[0052] The indicator history monitoring and management terminal also includes a message processing module;

[0053] The message processing module is configured to monitor a message sent by the indicator data collection module indicating that the indicator data has been successfully collected based on the collection configuration information in the automatic calculation threshold monitoring configuration information and the specified historical time period; or

[0054] Monitoring the policy processing success message sent by the policy threshold calculation module;

[0055] Based on the message of successful policy processing, a push data message is sent to the indicator historical data distribution module, wherein the push data message is used to notify the indicator historical data distribution module to push the first historical time series indicator to the state machine module; or

[0056] Monitor the message reported by the state machine calculation module that the first historical alarm information is successfully generated.

[0057] The beneficial effect of adopting this further solution is that the historical indicator monitoring management terminal can monitor the status of each link in data collection, processing, threshold calculation, and alarm generation in real time by monitoring the message feedback from each module. This ensures the real-time and accuracy of the entire monitoring process, allowing the system to promptly identify and address problems, thereby improving monitoring quality.

[0058] The indicator history monitoring management terminal monitors the message feedback from each module and automatically performs corresponding operations, such as sending push data messages and ensuring that the policy data (used for custom threshold calculations) within each collection cycle is calculated earlier than the historical time series data (the indicator data required for monitoring alarm replay). This ensures that the state machine calculation module uses dynamic thresholds with higher accuracy, thereby improving the accuracy of subsequent alarm replay processing. This reduces the complexity of manual operations, simplifies operation and maintenance, and improves work efficiency.

[0059] Furthermore, the indicator data collection terminal supports indicator data collection from multiple data sources, including message queues, databases, or obtaining data for a specified historical time period through a data query API.

[0060] The beneficial effect of adopting this further solution is that, by supporting multiple data sources, the system can flexibly adapt to different data collection scenarios. Whether it is real-time data capture from the message queue, querying historical data from the database, or obtaining data for a specific time period through the data query API, the system can meet user needs and improve the flexibility and versatility of data collection.

[0061] Based on the characteristics of the data source, the system can select the most appropriate data collection method to improve data collection efficiency. It can also provide unified monitoring, customized alarms, and analysis for large-scale historical indicator data from different data sources.

[0062] Furthermore, the non-automatic calculation threshold monitoring configuration information includes second basic configuration information and second collection configuration information, the second basic configuration information includes a task name, an identifier that the automatic threshold calculation setting is not enabled, and a dimension label name, and the second collection configuration information includes a data source for the second indicator data, screening conditions for collecting the second indicator data, a processing method for the second indicator data, and a collection period for the second indicator data;

[0063] The indicator data collection module is further configured to collect second historical time series indicator data based on the specified historical time period and the second set of collection configuration information;

[0064] The indicator data post-processing module is further configured to perform a second packaging process on the second historical time series indicator data based on the second acquisition configuration;

[0065] The collection and pushing module is further configured to store the first historical time series indicator data after the second encapsulation process in a historical message queue, and send the historical message queue to the indicator data stream calculation end.

[0066] The beneficial effect of adopting the above further solution is that the collection module is responsible for collecting the second indicator data according to the data source of the second collection configuration of each indicator and the specified historical time period, avoiding unnecessary data redundancy and improving the efficiency and accuracy of data collection.

[0067] Furthermore, the indicator historical data distribution module is further configured to obtain the second historical indicator time series data of the historical message queue;

[0068] Pushing the second historical indicator time series data to the state machine algorithm module based on the non-automatic calculation threshold monitoring configuration information;

[0069] The state machine calculation module is further configured to generate second historical alarm information based on the state machine calculation configuration of the non-automatic calculation threshold monitoring configuration information and the second historical indicator time series data.

[0070] The beneficial effect of adopting this further solution is that it enables the system to flexibly handle indicator tasks under different configurations. For indicator tasks that do not have automatically calculated threshold monitoring configurations, the system can directly generate alarm information based on the state machine calculation configuration and historical indicator time series data, without the need for additional threshold calculations. This flexibility allows the system to adapt to different monitoring needs, improving its versatility and adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A structural block diagram of a time series indicator data history monitoring and alarm playback system provided by one embodiment of the present invention;

[0072] Figure 2 A schematic block diagram of the data interaction process of the time series indicator data history monitoring alarm playback system provided by an embodiment of the present invention

[0073] Figure 3 A schematic diagram of the workflow of the indicator history monitoring module management terminal provided by one embodiment of the present invention;

[0074] Figure 4 A schematic diagram of the workflow of an indicator data collection terminal provided by one embodiment of the present invention;

[0075] Figure 5 A schematic diagram of the workflow of an indicator data stream calculation terminal provided by one embodiment of the present invention; DETAILED DESCRIPTION

[0076] The present application is further described in detail below with reference to the accompanying drawings.

[0077] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0078] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0079] Figure 1 The figure shows a structural block diagram of a time series indicator data history monitoring alarm playback system provided by an embodiment of the present invention.

[0080] like Figure 1 As shown in the figure, the architecture of the historical monitoring alarm playback system consists of three parts: the indicator history monitoring module management end, the indicator history data collection end, and the indicator history data calculation end. This architectural design has high flexibility and scalability in deployment.

[0081] When the data volume is relatively small or the system scale is small, the three modules (Indicator History Monitoring Module Management, Indicator History Data Collection, and Indicator History Data Calculation) can be deployed on the same machine (server). This deployment method simplifies system configuration and management, reduces network transmission overhead, and is suitable for lightweight application scenarios.

[0082] When the amount of data is large, each module can be deployed separately on a different machine, and the modules communicate with each other through message queues. As an asynchronous communication mechanism, message queues have the effects of decoupling, buffering, and reliability.

[0083] The indicator data collection terminal supports indicator data collection from multiple data sources, including message queues, databases, or obtaining data for a specified historical time period through a data query API.

[0084] Specifically, such as Figure 2 As shown, the indicator history monitoring management terminal is used to obtain the specified historical time period and indicator task configuration information of the indicator task, and the indicator task configuration information is any one of the non-automatic calculation threshold monitoring configuration information and the automatic calculation threshold monitoring configuration information;

[0085] When the indicator task configuration information of the indicator task is automatic calculation threshold monitoring configuration information, starting the indicator data collection end and the indicator historical data calculation end based on the specified historical time period and the automatic calculation threshold monitoring configuration information;

[0086] When the indicator task configuration information of the indicator task is non-automatic calculation threshold monitoring configuration information, starting the indicator data collection end and the indicator historical data calculation end based on the specified historical time period and the non-automatic calculation threshold monitoring configuration information;

[0087] The indicator data collection end is configured to collect first indicator data based on the specified historical time period and the automatically calculated threshold monitoring configuration information, and send the first indicator data to the indicator data stream calculation end, wherein the first indicator data includes first historical time series indicator data and policy data, wherein the policy data is data collected according to a preset collection granularity and a preset collection interval;

[0088] further configured to collect second indicator data based on the specified historical time period and the non-automatic calculation threshold monitoring configuration information, and send the second indicator data to the indicator data stream calculation end, where the second indicator data includes second historical time series indicator data;

[0089] The indicator data stream calculation end is used to perform a first alarm playback process on the indicator task according to the first indicator data and the automatic calculation threshold monitoring configuration information, and generate first alarm information;

[0090] It is also used to perform a second alarm playback process on the indicator task according to the second indicator data and the non-automatic calculation threshold monitoring configuration information to generate second alarm information.

[0091] In the above implementation, policy data is used as an example. Policy data is a set of time-series indicator data. For example, the collection granularity is one cycle (some indicator tasks are 1 minute, some are 5 minutes, etc.). For historical monitoring replay, each time point is collected is considered a cycle. For example, if the collection granularity is one minute and the current time is 11:03, then the policy data is a set of data at 11:03 every day for a week or a period of time. This data is used to calculate the dynamic threshold according to the corresponding policy calculation configuration. This dynamic threshold serves as the threshold used by the state machine to monitor the time-series data at 11:03.

[0092] Time-Series Data is a series of data points arranged in chronological order, used to record the changes of a certain indicator or measurement over time. Time-series indicator data usually includes timestamps, indicator values, labels or metadata.

[0093] In the above embodiment, if Figure 4 As shown, the indicator history monitoring management end obtains the indicator task of a specified historical time period, verifies the legitimacy of the specified historical time period, and after the legitimacy verification of the specified historical time period is passed, determines whether the indicator task is an indicator task configured for automatic calculation of threshold monitoring. If the indicator task is an indicator task configured for automatic calculation of threshold monitoring, the indicator history monitoring management end starts the indicator data collection end and the indicator history data calculation end for alarm replay processing based on the indicator task configured for automatic calculation of threshold monitoring.

[0094] By supporting indicator tasks with both automatic and non-automatic threshold monitoring configurations, monitoring becomes more flexible, allowing monitoring strategies to be adjusted based on actual needs. By collecting policy data, dynamic threshold calculations can be supported. These dynamic thresholds better reflect the system's actual operating conditions than fixed thresholds, thereby improving alarm accuracy and effectiveness. During the playback of historical monitoring alarms for time-series indicator data, different modules (such as data acquisition and calculation) work together, each performing their own functions. This allows for parallel data processing, improving data processing efficiency and making alarm playback faster and more efficient. This modular design provides a clear system structure and decouples modules, facilitating expansion and maintenance.

[0095] The management end of the indicator history monitoring module consists of a scheduling control module and a message processing module;

[0096] like Figure 1 and Figure 2 As shown, the scheduling control module is used to receive the historical time period specified by the user, and according to the configuration of the indicator task, schedule the modules related to the indicator historical data collection end and the indicator historical data calculation end (collection module, strategy threshold calculation module and state machine algorithm module, etc.).

[0097] The message processing module is used to send and receive messages of successful reception by the collection strategy module, the collection historical time series data module and the calculation module, and controls the strategy data in each collection cycle (data used to automatically calculate the dynamic threshold) to be calculated earlier than the historical time series data (indicator data that needs to be monitored for alarm replay) so that the state machine algorithm module can use accurate dynamic thresholds.

[0098] It should be noted that the indicator task configuration information includes basic configuration, collection configuration, policy calculation configuration and state machine calculation configuration. The basic configuration includes the task name, whether to enable automatic threshold calculation settings and dimension label name. The collection configuration includes the data source, collection filtering conditions, post-processing method and collection cycle.

[0099] The strategy calculation configuration is used to define the calculation logic and rules of the indicator task, including determining the formulas, algorithms and related parameter settings required for calculating the indicator.

[0100] State machine calculation configurations define the state transitions and calculation flow for metric tasks. A state machine is a model that describes system state changes and the transition relationships between states. Within metric tasks, state machine calculation configurations can trigger different calculation logic and state transitions based on task execution and data changes.

[0101] When the indicator task configuration information is non-automatic calculation threshold monitoring configuration information, the automatic calculation threshold monitoring configuration information includes first basic configuration information and first collection configuration information, the first basic configuration information includes the task name, an identifier for enabling automatic threshold calculation settings, and a dimension label name, and the first collection configuration information includes a data source for the first indicator data, screening conditions for collecting the first indicator data, a processing method for the first indicator data, and a collection period for the first indicator data;

[0102] When the indicator task configuration information is non-automatic calculation threshold monitoring configuration information, the non-automatic calculation threshold monitoring configuration information includes second basic configuration information and second collection configuration information, the second basic configuration information includes the task name, an identifier that the automatic threshold calculation setting is not enabled, and the dimension label name, and the second collection configuration information includes the data source of the second indicator data, the screening conditions for collecting the second indicator data, the processing method of the second indicator data, and the collection period of the second indicator data.

[0103] The indicator historical data collection terminal consists of a collection module, an indicator data post-processing module, and a collection and push module;

[0104] The collection module is used to collect indicator data and strategy data that automatically calculates thresholds based on the data source configured for the collection of each indicator and the specified historical time period.

[0105] The indicator data post-processing module is used to encapsulate and process the historical data of each indicator according to the collection configuration.

[0106] The collection and push module is used to push the historical data preliminarily processed by each indicator collection module and the historical data processed by the post-processing module to the historical message queue.

[0107] The indicator historical data calculation end consists of the indicator historical data distribution module, the indicator historical task message subscription module, the strategy threshold calculation module, the state machine algorithm module, the historical alarm data merging module and the historical alarm data label expansion module;

[0108] The indicator historical data distribution module is used to consume the historical indicator time series data pushed to the historical message queue by the collection module and the policy data for calculating dynamic thresholds by the policy algorithm.

[0109] The indicator history task message subscription module is used to subscribe to the created or updated indicator monitoring task configuration information and the strategy data pushed by the program for automatically calculating the threshold value of a specific indicator.

[0110] The policy threshold calculation module is used to calculate the threshold information (dynamic threshold) required for the corresponding historical indicator time series data, and push the threshold configuration information to the data queue required by the state machine.

[0111] The state machine algorithm module is used to calculate the threshold of the status (continuity, jitter, linear smoothing) of the historical time series indicator data pushed to the queue by the consumption collection module, and generate historical alarm information.

[0112] The historical alarm data merging module is used to merge historical alarm messages based on whether the indicator monitoring task has merging-related configurations.

[0113] The historical alarm data tag extension module is used to extend the tag operation of the indicator historical alarm message based on whether the indicator monitoring has the configuration of the extended tag.

[0114] The following is a detailed description of each of the above modules.

[0115] Figure 3 A schematic diagram of the workflow of the indicator history monitoring module management terminal provided by an embodiment of the present invention is shown.

[0116] like Figure 3 As shown, the scheduling control module is used to start the corresponding acquisition module, strategy algorithm module, state machine algorithm module and alarm module according to the specified historical time period specified by the user and the automatically calculated threshold monitoring configuration information created;

[0117] It is also used to start the corresponding acquisition module, state machine algorithm module and alarm module according to the specified historical time period specified by the user and the created non-automatic calculation threshold monitoring configuration information.

[0118] The message processing module is configured to monitor a message sent by the indicator data collection module indicating that the indicator data has been successfully collected based on the collection configuration information in the automatic calculation threshold monitoring configuration information and the specified historical time period; or

[0119] monitoring a message sent by the indicator data collection module indicating that indicator data has been successfully collected based on the collection configuration information in the non-automatic calculation threshold monitoring configuration information and the specified historical time period;

[0120] or,

[0121] Monitoring the policy processing success message sent by the policy threshold calculation module;

[0122] Based on the message of successful policy processing, a push data message is sent to the indicator historical data distribution module, wherein the push data message is used to notify the indicator historical data distribution module to push the first historical time series indicator to the state machine module; or

[0123] Monitor the message of successful generation of the first historical alarm information or the message of successful generation of the second historical alarm information reported by the state machine calculation module.

[0124] In the above implementation, the message module indicator of the indicator history monitoring module management terminal monitors the message feedback of each module to obtain real-time status of each link of data collection, processing, threshold calculation, and alarm generation. This ensures the real-time and accuracy of the entire monitoring process, enabling the system to promptly identify and address problems, thereby improving monitoring quality.

[0125] The indicator history monitoring management terminal monitors the message feedback from each module and automatically performs corresponding operations, such as sending push data messages and ensuring that the policy data (used for custom threshold calculations) within each collection cycle is calculated earlier than the historical time series data (the indicator data required for monitoring alarm replay). This ensures that the state machine calculation module uses dynamic thresholds with higher accuracy, thereby improving the accuracy of subsequent alarm replay processing. This reduces the complexity of manual operations, simplifies operation and maintenance, and improves work efficiency.

[0126] Figure 4 A schematic diagram of the workflow of an indicator data collection terminal provided by one embodiment of the present invention;

[0127] like Figure 4 As shown, the indicator data collection module is used to collect first historical time series indicator data based on the specified historical time period and the first collection configuration information;

[0128] Collecting policy data based on the specified historical time period, the first collection configuration information, and the first basic configuration;

[0129] The indicator data post-processing module is configured to perform a first encapsulation process on the first historical time series indicator data and the policy data based on the first acquisition configuration;

[0130] The collection and pushing module is used to store the first historical time series indicator data and the policy data after the first encapsulation process in a historical message queue, and send the historical message queue to the indicator data flow calculation end.

[0131] The indicator data collection module is further configured to collect second historical time series indicator data based on the specified historical time period and the second set of collection configuration information;

[0132] The indicator data post-processing module is further configured to perform a second packaging process on the second historical time series indicator data based on the second acquisition configuration;

[0133] The collection and pushing module is further configured to store the first historical time series indicator data after the second encapsulation process in a historical message queue, and send the historical message queue to the indicator data stream calculation end.

[0134] In the above implementation, the basic configuration and the collection configuration are described. Table 1 is an introduction to the basic configuration of the indicator task.

[0135] Table 1:

[0136]

[0137] Table 2 introduces the collection configuration of indicator tasks.

[0138] Table 2:

[0139]

[0140]

[0141] Since the acquisition module needs to have the ability to adapt to a variety of data sources, the acquisition module includes but is not limited to obtaining data for a specified historical time period from message queues, databases (such as MySQL, InfluxDB, ClickHouse), or through data query APIs. The acquisition module establishes connections with various data sources and uses corresponding protocols and interfaces to obtain indicator data. For message queues, the acquisition module can subscribe to specific message topics or queues and obtain data in real time. For databases, the acquisition module can execute query statements or use specific APIs to extract data. For data obtained through data query APIs, the acquisition module can communicate with the API through HTTP requests or other protocols and obtain the returned data.

[0142] When the task configuration for an indicator task uses automatic threshold calculation, the acquisition module collects the policy data and the first historical time-series indicator data and performs preliminary processing. This includes converting the data into a specific format, unifying the timestamp format, and standardizing indicator naming. This formatting ensures that indicator data from different data sources is presented consistently during subsequent processing and analysis.

[0143] If the acquisition configuration requires post-processing configuration, the first indicator data post-processing module will perform aggregation operations on the first historical time series indicator data and strategy data. The purpose of aggregation is to merge fine-grained indicator data into higher-level summary data, thereby reducing the number of fine-grained states in the subsequent state machine. Through aggregation, similar indicator data are merged and their statistical values, such as average, maximum, minimum, etc., are calculated. The aggregated indicator data will be integrated into a unified data structure for encapsulation and pushed to the historical message queue through the first acquisition push module for subsequent processing by the indicator data stream calculation end. The indicator data stream calculation end can more efficiently process a smaller number of aggregated historical indicator time series data without having to pay attention to too many fine-grained states.

[0144] When the task configuration information of the indicator task is non-automatic calculation threshold monitoring configuration information, the collection module collects the second historical time series indicator data based on the collection configuration of the non-automatic calculation threshold monitoring configuration and the specified historical time period, and the method of performing preliminary processing or post-processing on the second historical time series indicator data is the same as the above method and will not be repeated here.

[0145] Figure 5 A schematic diagram of the workflow of an indicator data stream calculation terminal provided by one embodiment of the present invention;

[0146] like Figure 5 As shown, the indicator history task message subscription module is used to obtain the indicator task configuration information sent by the indicator history monitoring management terminal, and send the indicator task configuration information to the policy threshold calculation module, the state machine calculation module, etc.

[0147] The indicator data distribution module is used to obtain the first historical indicator time series data and the strategy data in the historical message queue;

[0148] Pushing the policy data to a policy algorithm calculation module based on the automatically calculated threshold monitoring configuration information;

[0149] and obtaining a push data message sent by the indicator history monitoring management terminal, and sending the first historical indicator time series data to the state machine calculation module based on the push data message;

[0150] The policy threshold calculation module is used to calculate the dynamic threshold of the indicator task based on the policy data and the policy calculation configuration information of the automatic calculation threshold monitoring configuration information, and send the dynamic threshold to the state machine calculation module;

[0151] After the dynamic threshold is successfully sent to the state machine calculation module, a message indicating successful policy processing is returned to the indicator history monitoring management terminal;

[0152] The state machine calculation module is configured to update the threshold of the state machine calculation configuration information based on the dynamic threshold, and generate first historical alarm information based on the updated state machine calculation configuration and the first historical indicator time series data, and

[0153] The first historical alarm message is sent to an alarm message queue or a historical alarm data merging module or a historical alarm data tag expansion module.

[0154] The indicator historical data distribution module is further used to obtain the second historical indicator time series data of the historical message queue;

[0155] Pushing the second historical indicator time series data to the state machine algorithm module based on the non-automatic calculation threshold monitoring configuration information;

[0156] The state machine calculation module is further configured to generate second historical alarm information based on the state machine calculation configuration of the non-automatic calculation threshold monitoring configuration information and the second historical indicator time series data.

[0157] The historical alarm data merging module is used to merge the historical alarm information on the first historical alarm information based on the merged alarm configuration information in the indicator task configuration information when there is merged alarm configuration information in the automatic calculation threshold monitoring configuration information, and send the first historical alarm message after the merged historical alarm information processing to the alarm message queue or the historical alarm data tag expansion module;

[0158] It is also used to, when there is merged alarm configuration information in the non-automatic calculation threshold monitoring configuration information, merge the historical alarm information on the second historical alarm information based on the merged alarm configuration information in the non-automatic calculation threshold monitoring configuration information, and send the second historical alarm message after the merged historical alarm information processing to the alarm message queue or the historical alarm data tag expansion module;

[0159] A historical alarm data tag extension module is configured to, when an extended tag configuration is provided in the automatic calculation threshold monitoring configuration information, perform extended tag processing on the first indicator historical alarm information based on the configuration of the extended tag in the indicator task configuration information, and send the first historical alarm message after the extended tag processing to the alarm message queue;

[0160] It is also used to perform extended tag processing on the second indicator historical alarm information based on the configuration of the extended tag in the non-automatic calculation threshold monitoring configuration information when there is an extended tag configuration in the non-automatic calculation threshold monitoring configuration information, and send the second historical alarm message after the extended tag processing to the alarm message queue.

[0161] In the above implementation, if the indicator task configuration information for historical data alarm replay is the automatic calculation threshold monitoring configuration information, then for each collection cycle, the indicator historical data distribution module needs to first push the policy data to the policy calculation module. The policy calculation module calculates and generates the threshold data and pushes it to the state machine calculation module. The state machine calculation module needs to notify the alarm playback management module. After the alarm playback module receives the message that the policy collection data has successfully generated the threshold, it will then collect and push the first historical time series data. After the state calculation module receives the historical time series data of this collection cycle, it will report the message. In order to push the collection data of the next cycle until the end time specified by the user is reached, the entire process requires the scheduling module to participate in the control.

[0162] The policy algorithm configuration defines how to calculate the thresholds and strategies for these indicator data. By pushing this data to the policy algorithm configuration, historical data over a period of time can be calculated and analyzed to determine whether certain predefined policies are triggered and take corresponding actions.

[0163] At the same time, the strategy algorithm module also subscribes to the basic configuration of the indicator task to obtain relevant configuration information. When the indicator data pushed by the distribution module is not associated with the existing monitoring indicator task, the strategy algorithm module will discard the indicator data and only process the data related to the task.

[0164] The state machine algorithm module receives both the indicator time series data from the distribution module and the calculated threshold data for the corresponding time period pushed by the strategy algorithm module. After receiving both types of data, the state calculation module reports successful data reception to the message processing module on the management side of the indicator history monitoring module. This allows the scheduling module on the management side of the indicator history monitoring module to collect and push data for the next cycle for the indicator task with automatic threshold calculation enabled. This information is used to update the threshold configuration for each indicator monitored in the state machine algorithm module.

[0165] The state machine algorithm is configured with monitoring of states such as con (continuous threshold monitoring), jog (jitter threshold monitoring), and smooth (linear smooth threshold monitoring). Each monitoring method monitors different indicator data and triggers alarms based on specific threshold conditions. For example, for continuous threshold monitoring, you can set an indicator (such as the device CPU idle percentage) to trigger an alarm when it is less than 10% for 6 consecutive times in the past 10 minutes. For jitter threshold monitoring, you can set the device offline state to trigger an alarm when the number of offline times exceeds 4 times in the past hour, and for indicator tasks that enable dynamic thresholds, these thresholds are threshold configurations pushed after the policy algorithm module calculates the latest thresholds. When the threshold conditions are met, the state machine algorithm module generates the corresponding indicator alarm data.

[0166] If the indicator task configuration information for historical data alarm replay is non-automatic threshold calculation monitoring configuration information, the indicator historical data distribution module pushes the second historical indicator time series data to the state machine calculation module, and the state machine calculation module performs monitoring alarm replay processing on the second historical indicator time series data according to the state machine algorithm configuration, which will not be repeated here.

[0167] In the above implementation, the alarm data merging module aggregates multiple alarm indicator data items with the same tag within the same time range according to configured merging rules. This reduces redundant alarm messages, allowing users to more clearly understand the alarm situation without being distracted by a large amount of duplicate alarm information. The merged data can be pushed to the alarm data tag expansion module or directly to the alarm module for subsequent alarm processing. By merging alarm data, the accuracy and readability of alarm information are improved.

[0168] The Alarm Data Extended Tag module extends certain attribute tags of alarm data, making it easier for users to view and identify them when alarm messages are pushed. Based on configured rules, this module extends tags to indicator alarm data triggered by the state machine or merged indicator alarm data. Extending tags provides additional information to alarm data, enabling users to better understand and address alarm situations.

[0169] For example, for an availability alarm for a specific source station, the alarm data extended tag module can expand the alarm data's tags to include attributes such as the source station name, source station client, and IP address. This allows users to more quickly identify important information such as the source station, corresponding IP address, and alarm level through these extended tags when analyzing or generating alarm messages for users. By expanding the tags of alarm data, we can provide richer alarm information, helping users more accurately assess and address alarm situations. Based on the information in the extended tags, users can quickly understand the source and importance of the alarm and take appropriate measures to address it.

[0170] This system adopts a modular design approach, dividing the monitoring system into multiple modules, including the indicator history monitoring module management module, the indicator history data collection module, and the indicator history data calculation module. This modular design makes the system structure clear and decouples the modules, making it easier to expand and maintain.

[0171] This system supports indicator tasks for both non-automatic calculation of threshold monitoring configuration information and automatic calculation of threshold monitoring configuration information, making the monitoring method more flexible. Monitoring strategies can be adjusted according to actual needs, and monitoring tasks can be flexibly adjusted and managed to adapt to different monitoring needs.

[0172] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0173] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of application involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the aforementioned application concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions applied for in this application.

Claims

1. A historical monitoring alarm playback system for time series indicator data, characterized in that: include: The indicator history monitoring management terminal is used to obtain the specified historical time period and indicator task configuration information of the indicator task, wherein the indicator task configuration information is either non-automatic calculation threshold monitoring configuration information or automatic calculation threshold monitoring configuration information; When the indicator task configuration information of the indicator task is automatic calculation threshold monitoring configuration information, starting the indicator data collection end and the indicator historical data calculation end based on the specified historical time period and the automatic calculation threshold monitoring configuration information; The indicator data collection end is configured to collect first indicator data based on the specified historical time period and the automatically calculated threshold monitoring configuration information, and send the first indicator data to the indicator data stream calculation end, wherein the first indicator data includes first historical time series indicator data and policy data, wherein the policy data is data collected according to a preset collection granularity and a preset collection interval; The indicator data stream calculation end is used to perform a first alarm playback process on the indicator task according to the first indicator data and the automatic calculation threshold monitoring configuration information, and generate first alarm information; The indicator task configuration information also includes strategy calculation configuration information and state machine calculation configuration information, and the indicator data stream calculation end includes an indicator data distribution module, a strategy algorithm calculation module, and a state machine calculation module; The indicator data distribution module is used to obtain the first historical indicator time series data and the strategy data in the historical message queue; Pushing the policy data to a policy algorithm calculation module based on the automatically calculated threshold monitoring configuration information; and obtaining a push data message sent by the indicator history monitoring management terminal, and sending the first historical indicator time series data to the state machine calculation module based on the push data message; a policy threshold calculation module, configured to calculate the dynamic threshold of the indicator task based on the policy data and the policy calculation configuration information of the automatic calculation threshold monitoring configuration information, and send the dynamic threshold to the state machine calculation module; After the dynamic threshold is successfully sent to the state machine calculation module, a message indicating successful policy processing is returned to the indicator history monitoring management terminal; The state machine calculation module is configured to update the threshold of the state machine calculation configuration information based on the dynamic threshold, and generate first historical alarm information based on the updated state machine calculation configuration and the first historical indicator time series data, and Sending the first historical alarm message to an alarm message queue; The indicator history monitoring and management terminal also includes a message processing module; The message processing module is configured to monitor a message sent by the indicator data collection module indicating that the indicator data has been successfully collected based on the collection configuration information in the automatic calculation threshold monitoring configuration information and the specified historical time period; or Monitoring the policy processing success message sent by the policy threshold calculation module; Based on the message of successful policy processing, a push data message is sent to the indicator historical data distribution module, wherein the push data message is used to notify the indicator historical data distribution module to push the first historical time series indicator to the state machine module; or Monitor the message reported by the state machine calculation module that the first historical alarm information is successfully generated.

2. A historical monitoring alarm playback system for time series indicator data according to claim 1, characterized in that: The indicator history monitoring management terminal includes a scheduling control module: The scheduling control module is configured to start the indicator data collection end and the indicator historical data calculation end based on the specified historical time period and the non-automatic calculation threshold monitoring configuration information when the indicator task configuration information of the indicator task is non-automatic calculation threshold monitoring configuration information; The indicator data collection end is further configured to collect second indicator data based on the specified historical time period and the non-automatic calculation threshold monitoring configuration information, and send the second indicator data to the indicator data stream calculation end, where the second indicator data includes second historical time series indicator data; The indicator data stream calculation end is further used to perform a second alarm playback process on the indicator task according to the second indicator data and the non-automatic calculation threshold monitoring configuration information to generate second alarm information.

3. A historical monitoring alarm playback system for time series indicator data according to claim 1 or 2, characterized in that: The automatic calculation threshold monitoring configuration information includes first basic configuration information and first collection configuration information. The first basic configuration information includes a task name, an identifier for enabling automatic threshold calculation settings, and a dimension tag name. The first collection configuration information includes a data source for first indicator data, screening conditions for collecting first indicator data, a processing method for first indicator data, and a collection period for first indicator data. The index data collection terminal includes an index data collection module, an index data post-processing module and a collection and push module; The indicator data collection module is configured to collect first historical time series indicator data based on the specified historical time period and the first collection configuration information; Collecting policy data based on the specified historical time period, the first collection configuration information, and the first basic configuration; The indicator data post-processing module is configured to perform a first encapsulation process on the first historical time series indicator data and the policy data based on the first acquisition configuration; The collection and pushing module is used to store the first historical time series indicator data and the policy data after the first encapsulation process in a historical message queue, and send the historical message queue to the indicator data flow calculation end.

4. The historical monitoring alarm playback system for time series indicator data according to claim 1 is characterized in that: The indicator data stream calculation end also includes a historical alarm data merging module and a historical alarm data label expansion module: The historical alarm data merging module is used to merge the historical alarm information on the first historical alarm information based on the merged alarm configuration information in the indicator task configuration information when there is merged alarm configuration information in the automatic calculation threshold monitoring configuration information, and send the first historical alarm message after the merged historical alarm information processing to the alarm message queue or the historical alarm data tag expansion module; The historical alarm data tag extension module is used to perform extended tag processing on the first indicator historical alarm information based on the configuration of the extended tag in the indicator task configuration information when there is an extended tag configuration in the automatic calculation threshold monitoring configuration information, and send the first historical alarm message after extended tag processing to the alarm message queue.

5. The historical monitoring alarm playback system for time series indicator data according to claim 1 is characterized in that: The indicator data stream calculation also includes an indicator history task message subscription module; The indicator history task message subscription module is used to obtain the indicator task configuration information sent by the indicator history monitoring management terminal.

6. The historical monitoring alarm playback system for time series indicator data according to claim 2 is characterized in that: The indicator data collection terminal supports indicator data collection from multiple data sources, including message queues, databases, or obtaining data for a specified historical time period through a data query API.

7. The historical monitoring alarm playback system for time series indicator data according to claim 3 is characterized in that: The non-automatic calculation threshold monitoring configuration information includes second basic configuration information and second collection configuration information, the second basic configuration information includes the task name, an identifier that the automatic threshold calculation setting is not enabled, and the dimension tag name, and the second collection configuration information includes the data source of the second indicator data, the screening conditions for collecting the second indicator data, the processing method of the second indicator data, and the collection period of the second indicator data; The indicator data collection module is further configured to collect second historical time series indicator data based on the specified historical time period and the second collection configuration information; The indicator data post-processing module is further configured to perform a second packaging process on the second historical time series indicator data based on the second acquisition configuration; The collection and pushing module is further configured to store the first historical time series indicator data after the second encapsulation process in a historical message queue, and send the historical message queue to the indicator data stream calculation end.

8. The historical monitoring alarm playback system for time series indicator data according to claim 1 is characterized in that: The indicator historical data distribution module is further used to obtain the second historical indicator time series data of the historical message queue; Pushing the second historical indicator time series data to the state machine algorithm module based on the non-automatic calculation threshold monitoring configuration information; The state machine calculation module is further configured to generate second historical alarm information based on the state machine calculation configuration of the non-automatic calculation threshold monitoring configuration information and the second historical indicator time series data.

Images