Monitoring data management system and method, electronic equipment and storage medium

Abstract

The invention provides a monitoring data management system and method, electronic equipment and a storage medium. The system comprises a monitoring alarm device, a data storage device and a control management device, when the hybrid deployment environment is in a single cluster mode, a first monitoring alarm device obtains and stores first monitoring data of a single cluster, and when the storage duration of the first monitoring data is longer than a preset duration, the first monitoring data is migrated to a data storage device for persistent storage; when the hybrid deployment environment is in a multi-cluster mode, the second monitoring alarm device obtains second monitoring data and migrates the second monitoring data to the data storage device for persistent storage; the control management device is connected with the first monitoring alarm device and the data storage device and used for responding to the monitoring data query request and querying target monitoring data matched with the monitoring data query request in the hybrid deployment environment. According to the method, long-term storage and efficient query of the monitoring data can be realized, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a monitoring data management system and method, an electronic device and a storage medium. BACKGROUND

[0002] Currently, containerized applications are increasingly popular in hybrid deployment (single cluster and multi-cluster coexist) environments. However, the traditional monitoring and alarm scheme exposes limitations in long-term storage and efficient query of data, complexity of multi-cluster monitoring and management, and other pain points when facing this complex trend. These challenges seriously hinder enterprises from achieving efficient operation and maintenance and system stability in complex containerized environments.

[0003] Therefore, there is an urgent need for a monitoring data management system that can dynamically adapt to hybrid deployment and achieve unified management of data to solve the above problems. SUMMARY

[0004] Therefore, there is an urgent need for a monitoring data management system that can dynamically adapt to hybrid deployment and achieve unified management of data to solve the above problems.

[0005] The first aspect of the embodiment of the present application provides a monitoring data management system, which is applied to a hybrid deployment environment, the hybrid deployment environment including a single cluster mode and a multi-cluster mode, the single cluster mode including a single cluster, and the multi-cluster mode including a central cluster and a plurality of sub-clusters; the monitoring management system includes a monitoring and alarm device, a data storage device, and a control management device, wherein the single cluster and each of the sub-clusters are deployed with the monitoring and alarm device, and the central cluster is deployed with the data storage device; when the hybrid deployment environment is in the single cluster mode, a first monitoring and alarm device corresponding to the single cluster acquires and stores first monitoring data of the single cluster, and when the storage time length of the first monitoring data is greater than a preset time length, the first monitoring data is migrated to the data storage device for persistent storage; when the hybrid deployment environment is in the multi-cluster mode, a second monitoring and alarm device corresponding to each of the sub-clusters acquires second monitoring data of each of the sub-clusters, and migrates the second monitoring data to the data storage device for persistent storage; the control management device is connected with the first monitoring and alarm device and the data storage device respectively, and is used to respond to a monitoring data query request and query target monitoring data matching the monitoring data query request in the hybrid deployment environment.

[0006] In a possible implementation, the control management apparatus is configured to query target monitoring data matching a monitoring data query request in the hybrid deployment environment in response to the monitoring data query request, including: when the hybrid deployment environment is in the single-cluster mode, the control management apparatus is configured to query the target monitoring data in the first monitoring alarm apparatus in response to the monitoring data query request, in a case where a storage duration of the target monitoring data is detected to be less than the preset duration; and in a case where the storage duration is detected to be greater than or equal to the preset duration, the control management apparatus is configured to query the target monitoring data in the data storage apparatus; when the hybrid deployment environment is in the multi-cluster mode, the control management apparatus is configured to query the target monitoring data in the data storage apparatus in response to the monitoring data query request.

[0007] In a possible implementation, the first monitoring alarm apparatus includes a first alarm rule module and a first information processing module; the first alarm rule module is configured to set, in the first monitoring alarm apparatus, a first alarm condition of the first monitoring data matching a first rule setting instruction in response to the first rule setting instruction, and generate first alarm information after the first alarm condition is triggered; and the first information processing module is configured to receive the first alarm information, and group and deduplicate the first alarm information; the data storage apparatus includes a second alarm rule module, and the second monitoring alarm apparatus includes a second information processing module; the second alarm rule module is configured to set, in the data storage apparatus, a second alarm condition of the second monitoring data matching a second rule setting instruction in response to the second rule setting instruction, and generate second alarm information after the second alarm condition is triggered; and the second information processing module is configured to receive the second alarm information, and group and deduplicate the second alarm information.

[0008] In a possible implementation, the control management apparatus includes a secondary processing module; the first information processing module is further configured to send the grouped and deduplicated first alarm information to the secondary processing module, and the secondary processing module is configured to perform secondary processing on the grouped and deduplicated first alarm information according to a preset processing strategy, to obtain first final alarm information; the second information processing module is further configured to send the grouped and deduplicated second alarm information to the secondary processing module, and the secondary processing module is configured to perform secondary processing on the grouped and deduplicated second alarm information according to the preset processing strategy, to obtain second final alarm information; and the preset processing strategy at least includes alarm sending interval adjustment, alarm duration calculation, and alarm object matching.

[0009] In a possible implementation, the control management apparatus comprises a rule conversion module; the rule conversion module is configured to, after the mixed deployment environment switches from the single-cluster mode to the multi-cluster mode, configure a first format of the first alarm condition to be compatible with the data storage apparatus and synchronize the first format to the second alarm rule module; and the rule conversion module is further configured to, after the mixed deployment environment switches from the multi-cluster mode to the single-cluster mode, configure a second format of the second alarm condition to be compatible with the first monitoring alarm apparatus and synchronize the second format to the first alarm rule module.

[0010] In a second aspect, the embodiments of the present application further provide a monitoring data management method, applied to the monitoring data management system of the first aspect; the method comprises: in response to a monitoring data query request, determining target monitoring data matched with the monitoring data query request and a current environment of the mixed deployment environment; when the current environment is the single-cluster mode, detecting whether a storage duration of the target monitoring data is less than the preset duration; in a case where it is detected that the storage duration is less than the preset duration, querying the target monitoring data in the first monitoring alarm apparatus; in a case where it is detected that the storage duration is greater than or equal to the preset duration, querying the target monitoring data in the data storage apparatus; and when the current environment is the multi-cluster mode, querying the target monitoring data in the data storage apparatus.

[0011] In a possible implementation, the method further comprises: when the current environment is the single-cluster mode, detecting in real time whether a first alarm condition matched with the first monitoring alarm apparatus is triggered; in a case where it is detected that the first alarm condition is triggered, generating first alarm information and grouping and deduplicating the first alarm information; when the current environment is the multi-cluster mode, detecting in real time whether a second alarm condition matched with the data storage apparatus is triggered; and in a case where it is detected that the second alarm condition is triggered, generating second alarm information and grouping and deduplicating the second alarm information.

[0012] In a possible implementation, after the grouping and deduplication of the first alarm information, the method further comprises: performing secondary processing on the grouped and deduplicated first alarm information according to a preset processing strategy to obtain first final alarm information; and after the grouping and deduplication of the second alarm information, the method further comprises: performing secondary processing on the grouped and deduplicated second alarm information according to the preset processing strategy to obtain second final alarm information; wherein the preset processing strategy at least comprises alarm sending interval adjustment, alarm duration calculation, and alarm object matching.

[0013] In a third aspect, an electronic device is provided. The electronic device includes a processor and a memory. The memory is configured to store instructions, and the processor is configured to invoke the instructions in the memory to cause the electronic device to perform the method for monitoring data management according to the second aspect.

[0014] In a fourth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the method for monitoring data management according to the second aspect.

[0015] Compared with the related art, the embodiments of the present application have at least the following advantages: by deploying the first monitoring alarm device in a single cluster and deploying the data storage device in a central cluster, when the hybrid deployment environment is in a single cluster mode, the first monitoring alarm device can acquire and store the first monitoring data of the single cluster, and when the storage duration of the first monitoring data is greater than a preset duration, the first monitoring data is migrated to the data storage device for persistent storage. On the one hand, the first monitoring alarm device only needs to temporarily save the first monitoring data, thereby greatly releasing the storage pressure of the first monitoring alarm device, realizing the lightweight deployment of the first monitoring alarm device, and on the other hand, realizing the long-term storage of the first monitoring data in the single cluster mode. By deploying the second monitoring alarm device for each sub-cluster, when the hybrid deployment environment is in a multi-cluster mode, the second monitoring data acquired by the second monitoring alarm device can be directly migrated to the data storage device for persistent storage, realizing the long-term storage of the second monitoring data in the multi-cluster mode. In addition, the control management device is configured to respond to a monitoring data query request and query target monitoring data matching the monitoring data query request in the hybrid deployment environment. That is, the user does not need to know whether the first monitoring alarm device or the data storage device provides the target monitoring data, and can perform monitoring data query without awareness, thereby realizing efficient monitoring data query and improving the user experience.

[0016] The technical effects obtained by the above-mentioned second aspect, third aspect and fourth aspect are similar to the technical effects obtained by the corresponding technical means in the first aspect, which will not be described here. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 A functional module schematic diagram of a monitoring data management system provided by an embodiment of the present application; Figure 2 A step flowchart of a monitoring data management method provided by an embodiment of the present application; Figure 3 A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to more clearly understand the above-mentioned objects, features and advantages of the present application, the present application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0019] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application. The described embodiments are merely a part of the embodiments of the present application, and are not all the embodiments.

[0020] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application.

[0021] It should be further noted that, herein, the terms “comprising”, “including” or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or apparatus including a list of elements not only includes those elements, but also includes other elements not expressly listed or inherent to such process, method, article or apparatus. Without more limitations, the element defined by the phrase “comprising a” does not exclude the presence of additional identical elements in the process, method, article or apparatus including the element.

[0022] In the present application, “at least one” means one or more, and “multiple” means two or more than two. “And / or” describes the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The terms “first”, “second”, “third”, “fourth” and the like (if any) in the specification and claims of the present application and the drawings are used to distinguish similar objects, and are not used to describe a specific order or sequence.

[0023] In the embodiments of the present application, the words “exemplary” or “for example” are used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as “exemplary” or “for example” should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of “exemplary” or “for example” is intended to present concepts in a concrete manner.

[0024] For the sake of understanding, some explanations of concepts related to the embodiments of the present application are given by way of example for reference.

[0025] Prometheus: An open-source cloud-native monitoring and alerting system designed for dynamic, distributed environments, supporting multi-dimensional data collection, storage, querying, and visualization.

[0026] Remote Write: An asynchronous data writing mechanism implemented by Prometheus, primarily used to push monitoring data to external storage systems.

[0027] Prometheus Operator: A tool developed by CoreOS for automating the management and deployment of Prometheus monitoring systems in Kubernetes clusters. It extends the Kubernetes API to simplify Prometheus configuration management in a declarative configuration manner, supports the automatic generation of custom resources (such as ServiceMonitor, PodMonitor), and is compatible with cloud service discovery mechanisms such as AWS and Azure.

[0028] PrometheusRule: A CRD (CustomResource Definition) in Prometheus Operator for managing alert rules, enabling automated alert triggering through defined alert rules or recording rules.

[0029] Alertmanager: An open-source alert management tool in the Prometheus ecosystem, primarily used to handle and route alerts from multiple monitoring systems, supporting functions such as grouping, suppression, deduplication, and silencing, and sending notifications through various channels such as email, Slack, and WeChat.

[0030] VictoriaMetrics: A high-performance, cost-effective, and scalable open-source time series database (TSDB) and monitoring solution designed for large-scale index data processing.

[0031] VMAlert: A lightweight alert component provided by VictoriaMetrics, used to execute alert rules and send notifications. It is compatible with Prometheus alert rules (such as PromQL query expressions) and can periodically read data from VictoriaMetrics instances and trigger alerts.

[0032] CDB-Monitor: A self-developed monitoring interface and alert processing platform, responsible for receiving requests from external monitoring systems and performing unified processing and alerting. It is mainly used to monitor database performance and health status. Different platforms provide different CDB-Monitor functions.[1] Please refer to Figure 1 , Figure 1 is a functional module diagram of an embodiment of the monitoring data management system. For ease of illustration, only the parts related to the embodiment of the present application are shown in the functional module diagram of the embodiment of the monitoring data management system, and those skilled in the art can understand that the diagram structure does not constitute a limitation on the system, and the system can include more or fewer components than the diagram, or combine certain components, or have a different component arrangement.

[0033] It should be noted that Figure 1 The monitoring data management system 10 shown is applied to a hybrid deployment environment, which includes a single cluster mode including a single cluster and a multi-cluster mode including a central cluster and multiple sub-clusters.

[0034] As Figure 1 shown, the monitoring data management system 10 includes a monitoring alarm device 101, a data storage device 102, and a control management device 103, wherein the monitoring alarm device 101 is deployed in the single cluster and each sub-cluster, and the data storage device 102 is deployed in the central cluster.

[0035] In this embodiment, the monitoring alarm device 101 includes a first monitoring alarm device 101A and a second monitoring alarm device 101B. When the hybrid deployment environment is in the single cluster mode, the first monitoring alarm device 101A corresponding to the single cluster acquires and stores first monitoring data of the single cluster, and when the storage duration of the first monitoring data is greater than a preset duration, the first monitoring data is migrated to the data storage device 102 for persistent storage. When the hybrid deployment environment is in the multi-cluster mode, the second monitoring alarm device 101B corresponding to each sub-cluster acquires second monitoring data of each sub-cluster, and migrates the second monitoring data to the data storage device 102 for persistent storage. The control management device 103 is connected with the first monitoring alarm device 101A and the data storage device 102 respectively, and is used to query target monitoring data matching a monitoring data query request in the hybrid deployment environment in response to the monitoring data query request.

[0036] In some embodiments, the control management apparatus 103 is configured to query target monitoring data matching a monitoring data query request in the hybrid deployment environment in response to the monitoring data query request, including: when the hybrid deployment environment is in the single-cluster mode, the control management apparatus 103 is configured to query the target monitoring data in the first monitoring alarm apparatus 101A in response to the monitoring data query request, in a case where a storage duration of the target monitoring data is detected to be less than a preset duration; and in a case where the storage duration is detected to be greater than or equal to the preset duration, the control management apparatus 103 is configured to query the target monitoring data in the data storage apparatus 102; when the hybrid deployment environment is in the multi-cluster mode, the control management apparatus 103 is configured to query the target monitoring data in the data storage apparatus 102 in response to the monitoring data query request.

[0037] It can be understood that the embodiment does not specifically limit the size of the preset duration, which can be set according to actual needs, for example, 1 day, 2 days, etc.

[0038] Compared with the related art, the embodiment has at least the following advantages: by deploying the first monitoring alarm apparatus 101A in a single cluster and the data storage apparatus 102 in a central cluster, when the hybrid deployment environment is in the single-cluster mode, the first monitoring alarm apparatus 101A can acquire and store first monitoring data of the single cluster, and when the storage duration of the first monitoring data is greater than a preset duration, the first monitoring data is migrated to the data storage apparatus 102 for persistent storage, on the one hand, the first monitoring alarm apparatus 101A only needs to temporarily save the first monitoring data, thereby greatly releasing the storage pressure of the first monitoring alarm apparatus 101A, realizing lightweight deployment of the first monitoring alarm apparatus 101A, and on the other hand, realizing long-term storage of the first monitoring data in the single-cluster mode; by deploying the second monitoring alarm apparatus 101B for each sub-cluster, when the hybrid deployment environment is in the multi-cluster mode, the second monitoring data acquired by the second monitoring alarm apparatus 101B can be directly migrated to the data storage apparatus 102 for persistent storage, realizing long-term storage of the second monitoring data in the multi-cluster mode. In addition, the control management apparatus 103 is configured to query target monitoring data matching a monitoring data query request in the hybrid deployment environment in response to the monitoring data query request, that is, the user does not need to know whether the first monitoring alarm apparatus 101A or the data storage apparatus 102 provides the target monitoring data, and can perform monitoring data query without awareness, thereby realizing efficient query of monitoring data and improving user experience.

[0039] For further reference Figure 1The first monitoring and alarming device 101A includes a first alarm rule module 101A1 and a first information processing module 101A2; the first alarm rule module 101A1 is configured to set a first alarm condition of first monitoring data matched with a first rule setting instruction in the first monitoring and alarming device 101A in response to the first rule setting instruction, and generate first alarm information after the first alarm condition is triggered; and the first information processing module 101A2 is configured to receive the first alarm information, and group and deduplicate the first alarm information; the data storage device 102 includes a second alarm rule module 1021, and the second monitoring and alarming device 101B includes a second information processing module 101B1; the second alarm rule module 1021 is configured to set a second alarm condition of second monitoring data matched with a second rule setting instruction in the data storage device 102 in response to the second rule setting instruction, and generate second alarm information after the second alarm condition is triggered; and the second information processing module 101B1 is configured to receive the second alarm information, and group and deduplicate the second alarm information.

[0040] It is worth noting that, Figure 1 The control management device 103 shown includes a secondary processing module 1031; the first information processing module 101A2 is further configured to send the first alarm information after grouping and deduplication to the secondary processing module 1031, and the secondary processing module 1031 is configured to perform secondary processing on the first alarm information after grouping and deduplication according to a preset processing strategy, to obtain first final alarm information; the second information processing module 1021 is further configured to send the second alarm information after grouping and deduplication to the secondary processing module 1031, and the secondary processing module 1031 is configured to perform secondary processing on the second alarm information after grouping and deduplication according to a preset processing strategy, to obtain second final alarm information; wherein the preset processing strategy at least includes alarm sending interval adjustment, alarm duration calculation, and alarm object matching. By performing secondary processing on the original alarm, dynamic alarm interval control, personalized alarm content template, intelligent alarm noise reduction, and multi-channel distribution are provided, the alarm management is optimized, more refined alarm logic is realized, and false positives and alarm storms are reduced.

[0041] For further reference Figure 1 The control management device 103 further includes a rule conversion module 1032; the rule conversion module 1032 is configured to, after switching from a single-cluster mode to a multi-cluster mode in a hybrid deployment environment, configure a first format of the first alarm condition to be compatible with the data storage device 102, and synchronize to the second alarm rule module 1021; and the rule conversion module 1032 is further configured to, after switching from a multi-cluster mode to a single-cluster mode in a hybrid deployment environment, configure a second format of the second alarm condition to be compatible with the first monitoring and alarming device 101A, and synchronize to the first alarm rule module 101A1.

[0042] In some embodiments, the monitoring and alarming device 101 is Prometheus, the data storage device 102 is VictoriaMetrics, and the control and management device 103 is CDB-Monitor.

[0043] It should be noted that, in the case where the monitoring and alarming device 101 is Prometheus, the first alarm rule module 101A1 and the second alarm rule module 1021 are both PrometheusRule, and the first information processing module 101A2 and the second information processing module 101B1 are Alertmanager; in the case where the data storage device 102 is VictoriaMetrics, the second alarm rule module 1021 is VMAlert.

[0044] For ease of understanding, the working process of the monitoring data management system 10 of the present embodiment will be specifically described below with the monitoring and alarming device 101 being Prometheus, the data storage device 102 being VictoriaMetrics, and the control and management device 103 being CDB-Monitor as examples: 1. Single-cluster monitoring data management scheme: (1) Monitoring data flow: In the single-cluster mode, Prometheus serves as the main short-term monitoring indicator collector, responsible for collecting various indicators within the cluster in real time and at high frequency. At the same time, in order to realize long-term storage and historical backtracking of monitoring data, all collected indicators are pushed to VictoriaMetrics in real time through the remote-write mechanism for persistent storage.

[0045] (2) Querying monitoring data: When querying monitoring data, intelligent data source selection is performed: for real-time data queries within a preset time length, the system will preferentially obtain data from Prometheus to ensure the highest response speed; while for historical data queries exceeding the preset time length, VictoriaMetrics is automatically retrieved to fully utilize its efficient long-term storage and aggregation query capabilities. This layered query strategy not only guarantees real-time performance but also optimizes storage costs.

[0046] (3) Monitoring Data Alarms: The alarm policy configuration is based on the PrometheusRule resource object of the Prometheus Operator, allowing users to define detailed alarm rule groups through declarative configuration. When an alarm condition is triggered, the alarm event will be received and initially processed by Alertmanager (such as grouping and deduplication). Subsequently, the alarm information will be sent to the CDB-Monitor component for secondary processing, including but not limited to alarm enrichment, alarm suppression, and advanced processing such as intelligent noise reduction, which significantly reduces false alarms and duplicate alarms and improves the effectiveness of alarms.

[0047] 2. Multi-cluster monitoring data management solution: (1) Monitoring Data Stream: In multi-cluster mode, each independent cluster's Prometheus instance is still responsible for collecting its local metrics. Victoria Metrics is only deployed in the central cluster (or management cluster) as a centralized storage repository for all cluster monitoring data. All sub-cluster Prometheus instances push their collected metric data to the central cluster's Victoria Metrics via a remote-write mechanism. This architecture allows the sub-cluster Prometheus instances to store only temporary metric information, greatly reducing their storage pressure and achieving lightweight deployment.

[0048] (2) Querying monitoring data: All monitoring data queries (whether real-time or historical data) are performed uniformly through Victoria Metrics in the central cluster. This provides users with a globally consistent monitoring view without needing to care about the specific cluster from which the data originates, simplifying the troubleshooting and analysis of cross-cluster issues.

[0049] (3) Monitoring Data Alarms: Alarm configuration and triggering across multiple clusters no longer rely on the PrometheusRule of each sub-cluster, but instead uniformly adopt the VMAlert component of VictoriaMetrics (as the alarm rule engine). VMAlert can directly evaluate and trigger alarm rules based on the global metric data stored in VictoriaMetrics. Alarm events are also initially processed by Alertmanager and then sent to CDB-Monitor for secondary intelligent processing, ensuring the consistency of alarm processing logic and advanced functionality.

[0050] 3. Seamless switching between single-cluster and multi-cluster modes: When the user chooses to switch from single-cluster mode to multi-cluster mode, CDB-Monitor automatically parses the original PrometheusRule configuration and converts it to VictoriaMetrics-compatible VMRules format, while also migrating the relevant recording rules[2] and alert rules to VictoriaMetrics. Conversely, when switching from multi-cluster to single-cluster, CDB-Monitor also converts VMRules to PrometheusRule. This intelligent rule migration mechanism greatly simplifies operations and ensures a smooth transition, avoiding monitoring blind spots or alert interruptions caused by mode switching.

[0051] 4. Monitoring management of CDB-Monitor: (1) Interface adaptation and non-perception query: CDB-Monitor encapsulates and adapts the relevant API interfaces of Prometheus and VictoriaMetrics, providing a unified query interface to the upper layer users. This means that operations personnel do not need to understand whether Prometheus or VictoriaMetrics is providing data, and can perform non-perception queries, chart display, and data analysis, greatly reducing learning costs and operational complexity.

[0052] (2) Unified management of alert and recording rules: CDB-Monitor provides a friendly graphical interface and API interface, allowing users to directly add, modify, and delete alert rules and recording rules without needing to delve into the underlying cluster and perform operations through command lines or by modifying configuration files. This not only improves configuration efficiency but also reduces the risk of misoperation, achieving standardized and centralized rule management.

[0053] (3) Unified index collection and standardization: CDB-Monitor has the ability to collect and standardize platform metrics for monitoring and alerting. It can automatically discover and register services or collect metrics of various dimensions such as containers, nodes, and applications through a pre-set discovery mechanism, eliminating the need for users to manually configure each metric collection method and ensuring the completeness and consistency of monitoring data.

[0054] (4) After receiving the original alert triggered by Alertmanager (i.e., the first alert information and second alert information after grouping and deduplication), CDB-Monitor can enrich it again. This includes but is not limited to: Matched alarm object: Based on the tags or content in the alarm information, intelligently identify and associate to specific alarm objects (e.g., which service, which Pod, which node), making the alarm information more contextual.

[0055] Personalized alarm templates: Custom alarm content allows users to design and configure alarm information according to specific business needs and monitoring targets, ensuring clear and accurate alerts when critical events occur.

[0056] Calculate alarm duration: Accurately record and calculate the duration of the alarm from triggering to recovery, providing data for subsequent fault analysis and SLA evaluation.

[0057] Dynamic alarm interval control: Alarm sending is no longer a simple fixed interval. CDB-Monitor can dynamically adjust the sending interval of the alarm according to the severity, importance, historical trends or specific strategies of the alarm, avoid "alarm storm", ensure important alarms can be delivered in time, and non-critical alarms can be appropriately delayed or merged.

[0058] Multi-channel alarm distribution: Alarm information can be intelligently distributed to multiple alarm channels such as DingTalk, WeChat for Enterprise, SMS, email, etc. according to alarm objects, severity levels, etc. to ensure that alarms can reach the correct responsible person.

[0059] Intelligent alarm noise reduction and convergence: CDB-Monitor not only simply forwards alarms, but also has more advanced alarm suppression and noise reduction logic. CDB-Monitor has an alarm convergence function. After users configure the aggregation period, alarm policy, and convergence object according to the rules, the platform will send the alarm information triggered within the aggregation period after merging. It can effectively reduce the number of alarms and reduce the alarm storm.

[0060] Please refer to Figure 2 , Figure 2 is a step flowchart of an embodiment of the monitoring data management method of the present application. The order of the steps in the flowchart can be changed according to different needs, and some steps can be omitted.

[0061] It should be noted that the monitoring data management method of the embodiments of the present application can be applied to the monitoring data management system 10 described above. The specific process of the present embodiment is shown in Figure 2 , including the following steps: S201, in response to a monitoring data query request, determining the target monitoring data matched with the monitoring data query request and the current environment of the hybrid deployment environment.

[0062] S202, when the current environment is the single-cluster mode, detecting whether the storage duration of the target monitoring data is less than a preset duration, and when it is detected that the storage duration is less than the preset duration, performing S203; otherwise, performing S204.

[0063] S203, querying the target monitoring data in the first monitoring alarm device.

[0064] S204, querying the target monitoring data in the data storage device.

[0065] S205, when the current environment is the multi-cluster mode, querying the target monitoring data in the data storage device.

[0066] In some embodiments, the method further comprises: when the current environment is the single-cluster mode, detecting in real time whether a first alarm condition matched with the first monitoring alarm device is triggered; when it is detected that the first alarm condition is triggered, generating first alarm information, and grouping and deduplicating the first alarm information; when the current environment is the multi-cluster mode, detecting in real time whether a second alarm condition matched with the data storage device is triggered; when it is detected that the second alarm condition is triggered, generating second alarm information, and grouping and deduplicating the second alarm information.

[0067] In some embodiments, after grouping and deduplicating the first alarm information, the method further comprises: performing secondary processing on the first alarm information after grouping and deduplicating according to a preset processing strategy, to obtain first final alarm information; and after grouping and deduplicating the second alarm information, the method further comprises: performing secondary processing on the second alarm information after grouping and deduplicating according to a preset processing strategy, to obtain second final alarm information; wherein the preset processing strategy at least includes alarm sending interval adjustment, alarm duration calculation, and alarm object matching.

[0068] Compared with the related art, the embodiments of the present application have at least the following advantages: by deploying the first monitoring alarm device in a single cluster and deploying the data storage device in a central cluster, in a single cluster mode of the hybrid deployment environment, the first monitoring alarm device can acquire and store the first monitoring data of the single cluster, and when the storage time length of the first monitoring data is greater than a preset time length, the first monitoring data is migrated to the data storage device for persistent storage. On the one hand, the first monitoring alarm device only needs to temporarily save the first monitoring data, thereby greatly releasing the storage pressure of the first monitoring alarm device, realizing the lightweight deployment of the first monitoring alarm device, and on the other hand, realizing the long-term storage of the first monitoring data in the single cluster mode. By deploying the second monitoring alarm device for each sub-cluster, in a multi-cluster mode of the hybrid deployment environment, the second monitoring data acquired by the second monitoring alarm device can be directly migrated to the data storage device for persistent storage, realizing the long-term storage of the second monitoring data in the multi-cluster mode. In addition, the control management device is configured to respond to a monitoring data query request and query target monitoring data matching the monitoring data query request in the hybrid deployment environment, that is, the user does not need to know whether the first monitoring alarm device or the data storage device provides the target monitoring data, and can perform monitoring data query without awareness, thereby realizing efficient query of the monitoring data and improving the user experience.

[0069] Please refer to Figure 3 , Figure 3 is a schematic diagram of an embodiment of an electronic device. In the embodiment of the present application, the electronic device 300 comprises a processor 301, a memory 302 and a display 303. Figure 3 Only part of the components of the electronic device 300 are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0070] The processor 301 can be a central processing unit (CPU), a microprocessor or other data processing chip in some embodiments, used to run the program code or process data stored in the memory 302, such as the monitoring data management method in the present application.

[0071] In some embodiments, the processor 301 can be a single server or a server group. The server group can be centralized or distributed. In some embodiments, the processor 301 can be local or remote. In some embodiments, the processor 301 can be implemented in a cloud platform. In an embodiment, the cloud platform can include a private cloud, a public cloud, a hybrid cloud, a community cloud, a distributed cloud, an inter-internal, a multiple cloud, etc., or any combination thereof.

[0072] The memory 302 can be an internal storage unit of the electronic device 300, such as a hard disk or a memory of the electronic device 300 in some embodiments. The memory 302 can also be an external storage device of the electronic device 300, such as a plug-in hard disk, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. equipped on the electronic device 300 in other embodiments.

[0073] Further, the memory 302 can include both an internal storage unit and an external storage device of the electronic device 300. The memory 302 is used to store application software and various data installed on the electronic device 300.

[0074] The display 303 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, etc. in some embodiments. The display 303 is used to display information of the electronic device 300 and to display visualized user application programs. The components 301-303 of the electronic device 300 communicate with each other through a system bus.

[0075] In an embodiment, the following steps can be implemented when the processor 301 executes the monitoring data management program in the memory 302: In response to a monitoring data query request, determining target monitoring data matched with the monitoring data query request and a current environment of the hybrid deployment environment; When the current environment is the single-cluster mode, detecting whether a storage duration of the target monitoring data is less than the preset duration; In a case where it is detected that the storage duration is less than the preset duration, querying the target monitoring data in the first monitoring alarm device; In a case where it is detected that the storage duration is greater than or equal to the preset duration, querying the target monitoring data in the data storage device; When the current environment is the multi-cluster mode, querying the target monitoring data in the data storage device.

[0076] It should be understood that, in addition to the above functions, the processor 301 can also implement other functions when executing the monitoring data management program in the memory 302. For details, refer to the description of the corresponding method embodiments.

[0077] Further, the type of the electronic device 300 is not limited in the embodiments of the present application, and the electronic device 300 can be a mobile phone, a tablet computer, a personal digital assistant (PDA), a wearable device, a laptop computer, or the like. Exemplary embodiments of the portable electronic device include, but are not limited to, a portable electronic device running an IOS, an android, a microsoft, or other operating system. The portable electronic device can also be other portable electronic devices, such as a laptop computer having a touch-sensitive surface (e.g., a touch panel), and the like. It should also be understood that in some other embodiments of the present application, the electronic device 300 can not be a portable electronic device, but a desktop computer having a touch-sensitive surface (e.g., a touch panel).

[0078] Correspondingly, the embodiments of the present application also provide a computer readable storage medium for storing computer readable programs or instructions, which are executed by a processor to implement the steps or functions in the monitoring data management method provided by the above-mentioned method embodiments.

[0079] Those skilled in the art can understand that all or part of the processes of the above-mentioned embodiments can be completed by a computer program instructing related hardware (such as a processor, a controller, etc.) to complete, and the computer program can be stored in a computer readable storage medium. The computer readable storage medium is a magnetic disk, an optical disk, a read-only memory, or a random access memory, etc.

[0080] The monitoring data management system, method, electronic device and storage medium provided by the present application are described in detail above, and the principle and implementation manner of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation manner and application range can be changed, and the above description of the embodiments should not be understood as a limitation of the present application.

Claims

1. A monitoring data management system, characterized in that, The monitoring data management system is applied to a hybrid deployment environment, which includes a single cluster mode and a multi-cluster mode. The single cluster mode includes a single cluster, and the multi-cluster mode includes a central cluster and multiple sub-clusters. The monitoring and management system includes: a monitoring and alarm device, a data storage device, and a control and management device, wherein the monitoring and alarm device is deployed in each individual cluster and each sub-cluster, and the data storage device is deployed in the central cluster; When the hybrid deployment environment is the single cluster mode, the first monitoring and alarm device corresponding to the single cluster acquires and stores the first monitoring data of the single cluster, and when the storage duration of the first monitoring data is longer than the preset duration, the first monitoring data is migrated to the data storage device for persistent storage. When the hybrid deployment environment is the multi-cluster mode, the second monitoring and alarm device corresponding to each sub-cluster obtains the second monitoring data of each sub-cluster and migrates the second monitoring data to the data storage device for persistent storage; The control and management device is connected to the first monitoring and alarm device and the data storage device respectively, and is used to respond to monitoring data query requests and query target monitoring data that matches the monitoring data query request in the hybrid deployment environment.

2. The monitoring data management system according to claim 1, characterized in that, The control and management device is used to respond to monitoring data query requests and query target monitoring data that matches the monitoring data query request in the hybrid deployment environment, including: When the hybrid deployment environment is in the single cluster mode, the control and management device responds to the monitoring data query request and, if it detects that the storage time of the target monitoring data is less than the preset time, queries the target monitoring data in the first monitoring alarm device. If the storage duration is detected to be greater than or equal to the preset duration, the target monitoring data is queried from the data storage device. When the hybrid deployment environment is in the multi-cluster mode, the control and management device responds to the monitoring data query request and queries the target monitoring data in the data storage device.

3. The monitoring data management system according to claim 1, characterized in that, The first monitoring and alarm device includes a first alarm rule module and a first information processing module; The first alarm rule module is used to respond to the first rule setting instruction, set the first alarm condition of the first monitoring data that matches the first rule setting instruction in the first monitoring alarm device, and generate the first alarm information after the first alarm condition is triggered. The first information processing module is used to receive the first alarm information and to group and deduplicate the first alarm information; The data storage device includes a second alarm rule module, and the second monitoring and alarm device includes a second information processing module; The second alarm rule module is used to respond to the second rule setting instruction, set the second alarm condition of the second monitoring data that matches the second rule setting instruction in the data storage device, and generate the second alarm information after the second alarm condition is triggered; The second information processing module is used to receive the second alarm information and to group and deduplicate the second alarm information.

4. The monitoring data management system according to claim 3, characterized in that, The control and management device includes a secondary processing module; The first information processing module is further configured to send the first alarm information after grouping and deduplication to the secondary processing module. The secondary processing module is configured to perform secondary processing on the first alarm information after grouping and deduplication according to a preset processing strategy to obtain the first final alarm information. The second information processing module is further configured to send the grouped and deduplicated second alarm information to the secondary processing module. The secondary processing module is configured to perform secondary processing on the grouped and deduplicated second alarm information according to the preset processing strategy to obtain the second final alarm information. The preset processing strategy includes at least alarm sending interval adjustment, alarm duration calculation, and alarm object matching.

5. The monitoring data management system according to claim 3, characterized in that, The control and management device includes a rule conversion module; The rule conversion module is used to configure the first format of the first alarm condition to be compatible with the data storage device and synchronize it to the second alarm rule module after the hybrid deployment environment switches from the single cluster mode to the multi-cluster mode. The rule conversion module is also used to configure the second format of the second alarm condition to be compatible with the first monitoring alarm device and synchronize it to the first alarm rule module after the hybrid deployment environment switches from the multi-cluster mode to the single-cluster mode.

6. A method for managing monitoring data, characterized in that, Applied to the monitoring data management system according to any one of claims 1 to 5; The method includes: In response to a monitoring data query request, determine the target monitoring data that matches the monitoring data query request and the current environment of the hybrid deployment environment; When the current environment is in the single cluster mode, it is detected whether the storage duration of the target monitoring data is less than the preset duration; If the storage duration is detected to be less than the preset duration, the target monitoring data is queried in the first monitoring alarm device; If the storage duration is detected to be greater than or equal to the preset duration, the target monitoring data is queried from the data storage device. When the current environment is in the multi-cluster mode, the target monitoring data is queried from the data storage device.

7. The monitoring data management method according to claim 6, characterized in that, The method further includes: When the current environment is the single cluster mode, it is detected in real time whether the first alarm condition matching the first monitoring and alarm device is triggered. If the first alarm condition is detected to be triggered, a first alarm message is generated, and the first alarm message is grouped and deduplicated. When the current environment is in multi-cluster mode, it is detected in real time whether the second alarm condition matching the data storage device is triggered. If the second alarm condition is detected to be triggered, a second alarm message is generated, and the second alarm message is grouped and deduplicated.

8. The monitoring data management method according to claim 7, characterized in that, After grouping and deduplicating the first alarm information, the method further includes: The first alarm information after grouping and deduplication is processed a second time according to the preset processing strategy to obtain the first final alarm information; After grouping and deduplicating the second alarm information, the method further includes: The second alarm information after grouping and deduplication is processed a second time according to the preset processing strategy to obtain the second final alarm information; The preset processing strategy includes at least alarm sending interval adjustment, alarm duration calculation, and alarm object matching.

9. An electronic device, the electronic device comprising a processor and a memory, characterized in that, The memory is used to store instructions, and the processor is used to call the instructions in the memory to cause the electronic device to execute the monitoring data management method as described in any one of claims 6 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed on an electronic device, cause the electronic device to perform the monitoring data management method as described in any one of claims 6 to 8.

Images