Data monitoring method, device, equipment and storage medium

Abstract

The present application discloses a data monitoring method, apparatus, device and storage medium, which are applied to an indicator monitoring device, wherein the indicator monitoring device includes an InfluxDB time series database, and the method includes: collecting indicator data of each host in the cluster; storing the indicator data of each host in the first storage space of the InfluxDB time series database; the InfluxDB time series database only allows each host in the cluster to access; based on a preset aggregation strategy, querying the first indicator data from the first storage space, performing periodic aggregation operations on the first indicator data, and obtaining the aggregated second indicator data; and displaying the second indicator data on the indicator monitoring device interface. In this way, only the indicator data of each host in the current cluster is stored through the InfluxDB time series database, and when the stored indicator data is subsequently queried, only each host in the current cluster can access the InfluxDB time series database. Therefore, the InfluxDB time series database is not prone to failure or start-up and shutdown, causing the indicator monitoring device to crash, thereby ensuring the operational stability of the indicator monitoring device.

Description

Technical Field

[0001] The present application relates to big data technology, and in particular to a data monitoring method, apparatus, device and storage medium. Background Art

[0002] Ambari is cluster management software from the Apache open source community, providing integration, management, and monitoring capabilities for third-party services. The Ambari Metrics System (AMS) within Ambari monitors services and hosts across the entire cluster. AMS's metrics display allows operations personnel to visually visualize the operational status of each service and host, enabling them to perform resource forecasting and fault location tasks in a timely manner.

[0003] The existing AMS implementation of indicator data monitoring primarily involves three steps: indicator collection, indicator storage, and indicator aggregation. AMS's underlying indicator storage utilizes its own HBase database, which relies on the Hadoop Distributed File System (HDFS) for underlying operation. Because the HDFS service deployed in the Ambari cluster is externally accessible, the frequent reading, writing, and aggregation of indicator data in the AMS HBase database can easily place significant load on the HDFS service. Consequently, any HDFS service failure or shutdown can cause AMS downtime, impacting indicator monitoring across all Ambari services and hosts. Summary of the Invention

[0004] In order to solve the above technical problems, the present application provides a data monitoring method, device, equipment and storage medium.

[0005] The technical solution of this application is achieved as follows:

[0006] In a first aspect, a data monitoring method is provided, which is applied to an indicator monitoring device, wherein the indicator monitoring device includes an InfluxDB time series database. The method includes:

[0007] Collecting metrics data for each host in the cluster; wherein the metrics data includes the host's own metrics data and the metrics data of components deployed on the host;

[0008] Storing the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database only allows each host in the cluster to access;

[0009] Based on a preset aggregation strategy, query first indicator data from the first storage space, perform a periodic aggregation operation on the first indicator data, and obtain aggregated second indicator data;

[0010] The second indicator data is displayed on the indicator monitoring device interface.

[0011] In the above technical solution, the InfluxDB time series database is also integrated with a Telegraf plug-in; wherein, the Telegraf plug-in is used to collect component indicator data of services outside the Hadoop ecosystem deployed on the host.

[0012] In the above technical solution, the aggregation strategy includes aggregation category, aggregation object identifier and aggregation time; based on the preset aggregation strategy, the first indicator data is queried from the first storage space, and the first indicator data is periodically aggregated to obtain the aggregated second indicator data, including: according to the aggregation category and the aggregation object identifier, the continuous query statement function of the InfluxDB time series database is used to periodically query the first storage space to obtain the first indicator data within the aggregation time; calculate at least one of the maximum value, minimum value and average value of the first indicator data within the aggregation time to obtain the second indicator data corresponding to the aggregation object identifier.

[0013] In the above technical solution, when the aggregation category indicates that the aggregation strategy is for host-level aggregation, the aggregation object identifier includes a host identifier and an indicator identifier; wherein, the indicator identifier is the identifier of the host's own indicator or a component identifier; when the aggregation category indicates that the aggregation strategy is for cluster-level aggregation, the aggregation object identifier includes an indicator identifier.

[0014] In the above technical solution, the method further includes: storing the second indicator data in a second storage space corresponding to the aggregation strategy; wherein different aggregation strategies correspond to different second storage spaces.

[0015] In the above technical solution, displaying the second indicator data on the indicator monitoring device interface includes: obtaining at least one second indicator data corresponding to at least one aggregation strategy from the second storage space, and displaying it on the indicator monitoring device interface.

[0016] In the above technical solution, storing the indicator data of each host in the first storage space of the InfluxDB time series database includes: adjusting the indicator data in JSON format to indicator data in a storage format suitable for the InfluxDB time series database.

[0017] In a second aspect, a data monitoring apparatus is provided, which is applied to an indicator monitoring device, wherein the indicator monitoring device includes an InfluxDB time series database, and the apparatus includes:

[0018] A collection unit, configured to collect index data of each host in the cluster; wherein the index data includes the host's own index data and the index data of components deployed on the host;

[0019] A storage unit, configured to store the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database only allows each host in the cluster to access;

[0020] an aggregation unit, configured to query first indicator data from the first storage space based on a preset aggregation policy, perform a periodic aggregation operation on the first indicator data, and obtain aggregated second indicator data;

[0021] A display unit is configured to display the second indicator data on the indicator monitoring device interface.

[0022] According to a third aspect, an indicator monitoring device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor, wherein the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0023] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0024] The present application discloses a data monitoring method, which is applied to an indicator monitoring device, wherein the indicator monitoring device includes an InfluxDB time series database, and the method includes: collecting indicator data of each host in the cluster; wherein the indicator data includes the host's own indicator data and the component indicator data of the service deployed on the host; storing the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database only allows each host in the cluster to access it; based on a preset aggregation strategy, querying the first indicator data from the first storage space, performing periodic aggregation operations on the first indicator data, and obtaining the aggregated second indicator data; and displaying the second indicator data on the indicator monitoring device interface. In this way, only the indicator data of each host in the current cluster is stored through the InfluxDB time series database, and when the stored indicator data is subsequently queried, only each host in the current cluster can access the InfluxDB time series database. Therefore, the InfluxDB time series database is not prone to failure or start-up and shutdown, causing the indicator monitoring device to crash, thereby ensuring the operational stability of the indicator monitoring device. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the first flow chart of the data monitoring method in an embodiment of the present application;

[0026] Figure 2 This is a structural diagram of the AMS architecture composition in an embodiment of the present application;

[0027] Figure 3 This is a second flow chart of the data monitoring method in an embodiment of the present application;

[0028] Figure 4 This is a structural diagram of the data monitoring device in an embodiment of the present application;

[0029] Figure 5 This is a structural diagram of the data monitoring device in an embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0031] Example 1

[0032] The present invention provides a data monitoring method. Figure 1 This is a first flow chart of the data monitoring method in an embodiment of the present application.

[0033] like Figure 1 As shown, the data monitoring method is applied to an indicator monitoring device, which includes an InfluxDB time series database. The data monitoring method may specifically include:

[0034] Step 101: Collecting indicator data of each host in the cluster; wherein the indicator data includes the host's own indicator data and the indicator data of components deployed on the host;

[0035] It should be noted that the host's own metrics include at least the network, disk, and central processing unit (CPU). Services deployed on the host include at least HDFS, Kafka, Flume, other databases (such as MySQL), message queues (such as RocketMQ), and middleware. The HDFS service components include the NameNode, SecondaryNameNode, and DataNode; the Kafka service components include at least the producer, broker, and topic; since the MySQL database service is not a distributed service, the MySQL database service itself is the only component; the Rocaetmq message queue service components include at least two Rocaetmq child message queues.

[0036] In practical applications, the indicator monitoring device can be AMS. This application provides an AMS architecture diagram. Figure 2 This is a schematic diagram of the structure of the AMS architecture in the embodiment of this application. Figure 2 As shown, AMS includes Ambari Server 20, MetricsMonitor 21, TimelineWebService 22, Hadoop Service Sinks 23, InfluxDB 24, and Telegraf 25. AMS metric collection is divided into host metrics and component metrics from services deployed on the host. Metrics Monitor 21 collects host metrics, submitting these metrics to AMS's TimelineWebService 22 via POST. Hadoop Service Sinks 23 collects metrics from services deployed on the host (the services here only include the HDFS, Kafka, and Flume services mentioned in the previous paragraph), submitting these metrics to AMS's TimelineWebService 22 via POST. The other databases (such as MySQL), message queues (such as RocketMQ), and middleware services mentioned in the previous paragraph are collected by the Telegraf 25 plug-in.

[0037] It should be noted that the InfluxDB time series database of the indicator monitoring device is integrated with the Telegraf plug-in. The Telegraf plug-in is mainly used to collect component indicator data of services outside the Hadoop ecosystem deployed on the host (i.e. other databases, message queues, and middleware). This solves the problem that existing indicator monitoring devices only support the collection of component indicator data of services within the Hadoop ecosystem, and further expands the space for data monitoring of subsequent indicator monitoring devices.

[0038] The accuracy of the host's own indicator data and the component indicator data deployed on the host mentioned above are both at the second level (for example, collected once every 10 seconds), which is very high.

[0039] Step 102: storing the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database is only accessible to each host in the cluster;

[0040] It's important to note that InfluxDB is an open-source time-series database developed by InfluxData. Written in Go, it focuses on high-performance querying and storage of time-series data. Since the InfluxDB time-series database doesn't provide services for external use, it's less likely to experience failures or system startups that could cause AMS downtime, ensuring that AMS monitors the metrics of every host in the cluster.

[0041] It should be noted that the first storage space is a storage space for storing the collected indicator data of each host. In other words, the indicator data stored in the first storage space is indicator data that has not been processed.

[0042] In practice, before implementing this step, you need to create a first storage space in the InfluxDB time series database. This first storage space sets a retention period for stored indicator data. The indicator data stored in the first storage space includes the collection time information. If the retention period stored in the first storage space from the collection time is greater than or equal to the preset retention period threshold, the corresponding indicator data is automatically deleted, saving a certain amount of storage space.

[0043] In some embodiments, this step specifically includes: adjusting the indicator data in JSON format to indicator data in a storage format suitable for the InfluxDB time series database.

[0044] It should be noted that the indicator data of each host in the cluster collected in step 101 is in JSON format, and the indicator data in JSON format cannot be directly stored in the InfluxDB time series database. Therefore, before storage, the indicator data in JSON format needs to be adjusted to indicator data in a storage format suitable for the InfluxDB time series database.

[0045] In actual applications, any indicator data in JSON format for each host collected in step 101 includes an indicator and indicator data corresponding to the indicator at different time points, and any indicator data in the adjusted storage format adapted to the InfluxDB time series database includes at least two indicators and indicator data corresponding to each indicator at the same time point.

[0046] For example, the component metrics data for the InfluxDB time series database storage service can be: namenode, host = server01, component = namenode, dfs.name.FSsystem = 4.0, capacity = 5, jvm.commit = 8 1590384733668000000. This shows that it includes two metrics, FSsystem and capacity, with corresponding values ​​of 4.0 and 5, respectively.

[0047] The InfluxDB time series database stores host metrics data, which can be: HOST, host = server01, component = HOST mem free = 40962386, cpu_usage = 75, disk_total = 102417671590384733668000000. This shows that the three metrics are HOST mem free, cpu_usage, and disk_total, with corresponding values ​​of 40962386, 75, and 10241767, respectively.

[0048] Here combined Figure 2 This step is to write (Write) the host's own indicator data submitted to TimelineWebService22 and the component indicator data of the service deployed on the host to the InfluxDB time series database.

[0049] Before explaining the following steps, it's important to note that while the accuracy of the indicator data stored in the first storage space in this step is at the second level, while high, it doesn't make it easy for monitoring personnel to visually analyze the overall performance of the indicator data. Therefore, to enable intuitive analysis of the indicator data in the first storage space, it's necessary to reduce the accuracy of the indicator data from the second level to the level of a certain level, such as a certain hour, or even a certain day.

[0050] Regarding reducing the precision of the indicator data, step 103 is used to reduce the precision of the indicator data of the first storage space.

[0051] Step 103: Based on a preset aggregation strategy, query first indicator data from the first storage space, perform periodic aggregation operations on the first indicator data, and obtain aggregated second indicator data;

[0052] It should be noted that the aggregation strategy is an aggregation strategy for the indicator data in the first storage space. Before performing the aggregation operation, the first indicator data must be queried from the first storage space according to the aggregation strategy. The data precision of the first indicator data is at the second level. The aggregation operation on the first indicator data is to reduce the first indicator data with second-level data precision to the second indicator data with hierarchical, hourly, or daily data precision, thereby achieving the purpose of reducing the indicator data precision.

[0053] In some embodiments, the aggregation strategy includes an aggregation category, an aggregation object identifier, and an aggregation time; this step specifically includes: based on the aggregation category and the aggregation object identifier, using the continuous query statement function of the InfluxDB time series database to periodically query the first storage space to obtain the first indicator data within the aggregation time; calculating at least one of the maximum value, minimum value, and average value of the first indicator data within the aggregation time to obtain the second indicator data corresponding to the aggregation object identifier.

[0054] Step 104: Display the second indicator data on the indicator monitoring device interface.

[0055] It should be noted that the second indicator data is the indicator data obtained by performing an aggregation operation on the first indicator data, and the second indicator data includes at least one of the maximum value, the minimum value and the average value of the first indicator data within the aggregation time.

[0056] Here, the indicator monitoring device interface intuitively displays at least one of the maximum value, minimum value and average value of the first indicator data within the aggregation time, helping monitoring personnel to quickly understand the general situation of the first indicator data and improve the monitoring ability of the indicator data.

[0057] In some embodiments, the method further includes: storing the second indicator data in a second storage space corresponding to the aggregation strategy; wherein different aggregation strategies correspond to different second storage spaces.

[0058] It should be noted that the second storage space refers to the second storage space of the InfluxDB time series database. The number of second storage spaces depends on the number of aggregation strategies.

[0059] In actual applications, different second storage spaces are created in the InfluxDB time series database for different aggregation strategies in advance. After the corresponding second indicator data is obtained based on a certain aggregation strategy, it is first stored in the second storage space corresponding to the aggregation strategy, so that it can be directly obtained from the second storage space later.

[0060] In some embodiments, this step specifically includes: obtaining at least one second indicator data corresponding to at least one aggregation strategy from the second storage space, and displaying it on the indicator monitoring device interface.

[0061] Here combined Figure 2 In this step, the TimelineWebService 22 component obtains at least one second indicator data corresponding to at least one aggregation strategy from the second storage space of the InfluxDB 24 time series database in a GET manner, and the Ambari Server 20 component then obtains the at least one second indicator data in a GET manner and displays it on the indicator monitoring device interface.

[0062] Here, the execution entity of steps 101 to 104 may be a processor of the data monitoring device.

[0063] With the above technical solution, only the indicator data of each host in the current cluster is stored in the InfluxDB time series database. When the stored indicator data is subsequently queried, only each host in the current cluster can access the InfluxDB time series database. Therefore, the InfluxDB time series database is not prone to failures or start-up and shutdown that may cause the indicator monitoring equipment to crash, ensuring the operational stability of the indicator monitoring equipment.

[0064] Example 2

[0065] Based on the above embodiment, the data monitoring method is further described. Figure 3 This is a second flow chart of the data monitoring method in an embodiment of the present application.

[0066] like Figure 3 As shown, the data monitoring method is applied to the indicator monitoring device, which includes the InfluxDB time series database. The specific steps are as follows:

[0067] Step 301: Collecting indicator data of each host in the cluster; wherein the indicator data includes the host's own indicator data and the indicator data of components deployed on the host;

[0068] In some embodiments, the InfluxDB time series database further integrates a Telegraf plug-in; wherein, the Telegraf plug-in is used to collect component indicator data for other services outside the Hadoop ecosystem deployed on the host.

[0069] It should be noted that the InfluxDB time series database of the indicator monitoring device is integrated with the Telegraf plug-in. The Telegraf plug-in is mainly used to collect component indicator data of services outside the Hadoop ecosystem deployed on the host (i.e. other databases, message queues, and middleware). This solves the problem that existing indicator monitoring devices only support the collection of component indicator data of services within the Hadoop ecosystem, further expanding the space for data monitoring of indicator monitoring devices.

[0070] Step 302: Store the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database only allows each host in the cluster to access;

[0071] It's important to note that InfluxDB is an open-source time-series database developed by InfluxData. Written in Go, it focuses on high-performance querying and storage of time-series data. Since the InfluxDB time-series database doesn't provide services for external use, it's less likely to experience failures or system startups that could cause AMS downtime, ensuring that AMS monitors the metrics of every host in the cluster.

[0072] In some embodiments, this step specifically includes: adjusting the indicator data in JSON format to indicator data in a storage format suitable for the InfluxDB time series database.

[0073] Before explaining the following steps, it's important to note that while the accuracy of the indicator data stored in the first storage space in this step is at the second level, while high, it doesn't make it easy for monitoring personnel to visually analyze the overall performance of the indicator data. Therefore, to enable intuitive analysis of the indicator data in the first storage space, it's necessary to reduce the accuracy of the indicator data from the second level to the level of a certain level, such as a certain hour, or even a certain day.

[0074] Regarding reducing the precision of the indicator data, the precision of the indicator data is reduced by steps 303 and 304 .

[0075] Step 303: Based on the aggregation category and the aggregation object identifier, use the continuous query statement function of the InfluxDB time series database to periodically query the first storage space to obtain first indicator data within the aggregation time;

[0076] It should be noted that when the aggregation type indicates aggregation at the host level, the aggregation object identifier includes the host identifier and the indicator identifier, wherein the indicator identifier is the identifier of the host's own indicator and / or component identifier.

[0077] Here, host-level aggregation refers to the aggregation of indicator data for different hosts, so the aggregation object identifier needs to consider the host identifier. Step 301 has made it clear that the host's indicator data includes two types of indicator data, namely the host's own indicator data and component indicator data. Different types of indicator data need to be aggregated separately. Therefore, when aggregating the host's own indicator data, the aggregation object identifier also needs to consider the host's own indicator identifier; when aggregating component indicator data, the aggregation object identifier also needs to consider the component identifier. Among them, there is no specific restriction on the order of aggregating different types of indicator data, and it can be parallel or serial.

[0078] It should be noted that when the aggregation type indicates cluster-level aggregation, the aggregation object identifier includes an indicator identifier, wherein the indicator identifier is the identifier of the host's own indicator and / or component identifier.

[0079] Cluster-level aggregation refers to the aggregation of metric data corresponding to the same metric across all hosts in the cluster. Therefore, the aggregation object identifier does not need to consider the host identifier. Similarly, host metric data includes two types of metric data: the host's own metric data and component metric data. Different types of metric data need to be aggregated separately. Therefore, when aggregating the host's own metric data, the aggregation object identifier only needs to consider the host's own metric identifier; when aggregating component metric data, the aggregation object identifier only needs to consider the component identifier. There are no specific restrictions on the order in which different types of metric data are aggregated; they can be performed in parallel or serially.

[0080] It should be noted that the continuous query statement function of the InfluxDB time series database is used to periodically query the first storage space, replacing the existing AMS backend frequently enabling threads to frequently query the indicator data in the HBase database through Phoenix. This makes the overall AMS backend storage architecture more lightweight, greatly relieves the monitoring pressure of AMS, and improves the efficiency of AMS in monitoring indicator data.

[0081] Specifically, for step 303, if it is determined that the aggregation category indicates aggregation at the host level, and the aggregation object identifier includes a host identifier and a component identifier, the continuous query statement function of the InfluxDB time series database is used to perform continuous periodic queries on the indicator data of each component on each host within the aggregation time from the first storage space. Assuming that the indicator data of each component on each host in the first storage space is a whole segment of indicator data, M segments of indicator data can be obtained after the query, that is, the M segments of indicator data corresponding to each component on each host constitute the first indicator data within the aggregation time. Wherein, M is a positive integer greater than or equal to 2. If it is determined that the aggregation category indicates cluster-level aggregation, and the aggregation object identifier includes a component identifier, the continuous query statement function of the InfluxDB time series database is used to perform continuous periodic queries on the indicator data of the same component for all hosts from the first storage space within the aggregation time. Assuming that the indicator data of each component of the same component in the first storage space is a whole segment of indicator data, after the query, the whole segment of indicator data for each component in the same component is divided into P segments of indicator data (which can be expressed as P1, P2...Pp). The P1 segment indicator data of each component in the same component is integrated as one segment of indicator data of the first indicator data within the aggregation time. Where P is a positive integer greater than or equal to 2; p is a positive integer greater than 0.

[0082] It should be noted that the accuracy of the first indicator data within the aggregation time is at the second level.

[0083] Step 304: Calculate at least one of the maximum value, minimum value, and average value of the first indicator data within the aggregation time to obtain second indicator data corresponding to the aggregation object identifier;

[0084] Here, the aggregation time is measured in minutes, hours, or days. By averaging the first indicator data with second-level precision during the aggregation time, second indicator data with graded, hourly, or daily precision can be obtained. Furthermore, the maximum and / or minimum values ​​within the first indicator data can be calculated and used as second indicator data for monitoring personnel's reference.

[0085] Step 305: Display the second indicator data on the indicator monitoring device interface.

[0086] In some embodiments, the method further includes: storing the second indicator data in a second storage space corresponding to the aggregation strategy; wherein different aggregation strategies correspond to different second storage spaces.

[0087] In some embodiments, this step specifically includes: obtaining at least one second indicator data corresponding to at least one aggregation strategy from the second storage space, and displaying it on the indicator monitoring device interface.

[0088] With the above technical solution, only the indicator data of each host in the current cluster is stored in the InfluxDB time series database. When the stored indicator data is subsequently queried, only each host in the current cluster can access the InfluxDB time series database. Therefore, the InfluxDB time series database is not prone to failures or start-up and shutdown that may cause the indicator monitoring equipment to crash, ensuring the operational stability of the indicator monitoring equipment.

[0089] Example 3

[0090] Based on the above embodiments, the present application provides a storage strategy table for storing indicator data in an InfluxDB time series database. Table 1 is a storage strategy table for storing indicator data in an InfluxDB time series database in an embodiment of the present application.

[0091] Table 1. Storage strategy for storing indicator data in the InfluxDB time series database

[0092]

[0093] It should be noted that storage spaces are created in the InfluxDB time series database in advance according to the storage policies and storage durations given in Table 1, namely, default, Host_minutes, Host_hours, Host_days, Cluster_minutes, Cluster_hours, and Cluster_days.

[0094] Specifically, the indicator data of each host in the cluster can be collected at 10-second intervals. The 10-second precision data is stored in the default (i.e., the first storage space) of the InfluxDB time series database. If the storage time in the default exceeds 1 day, the corresponding indicator data will be automatically deleted.

[0095] When aggregating the indicator data in default, the aggregation categories include host-level aggregation and cluster-level aggregation, and the aggregation time can include 5 minutes, 1 hour, and 365 days. Specifically, for host-level aggregation, the aggregation object identifier includes the host identifier and the indicator identifier (including the host's own indicators and component indicators). If the aggregation time is 5 minutes, the corresponding indicator data within 5 minutes is queried from default, and at least one of the maximum, minimum, and average values ​​of the corresponding indicator data within 5 minutes is calculated to obtain 5-minute precision data. The 5-minute precision data is stored in the Host_minutes (i.e., the second storage space) of the InfluxDB time series database. If the storage time in Host_minutes exceeds 1 week, the corresponding indicator data will be automatically deleted.

[0096] For host-level aggregation, the aggregation object identifier includes the host identifier and the indicator identifier (including the host's own indicators and component indicators). If the aggregation time is 1 hour, the corresponding indicator data within 1 hour is queried from the default, and at least one of the maximum, minimum, and average values ​​of the corresponding indicator data within 1 hour is calculated to obtain 1-hour precision data. The 1-hour precision data is stored in the Host_hours (i.e., the second storage space) of the InfluxDB time series database. If the storage time in Host_hours exceeds 30 days, the corresponding indicator data is automatically deleted.

[0097] For host-level aggregation, the aggregation object identifier includes the host identifier and the indicator identifier (including the host's own indicators and component indicators). If the aggregation period is 365 days, the corresponding indicator data within 365 days is queried from the default, and at least one of the maximum, minimum, and average values ​​of the corresponding indicator data within 365 days is calculated to obtain 365-day precision data. The 365-day precision data is stored in the Host_days (i.e., the second storage space) of the InfluxDB time series database. If the storage time in Host_days exceeds 30 days, the corresponding indicator data is automatically deleted.

[0098] For cluster-level aggregation, the aggregate object identifier includes the indicator identifier (including host indicators and component indicators). If the aggregation period is 5 minutes, the corresponding indicator data within 5 minutes is queried from the default, and at least one of the maximum, minimum, and average values ​​of the corresponding indicator data within 5 minutes is calculated to obtain 5-minute precision data. The 5-minute precision data is stored in the Cluster_minutes (i.e., the second storage space) of the InfluxDB time series database. If the storage time in Cluster_minutes exceeds 30 days, the corresponding indicator data is automatically deleted.

[0099] For cluster-level aggregation, the aggregate object identifier includes the indicator identifier (including host indicators and component indicators). If the aggregation time is 1 hour, the corresponding indicator data within 1 hour is queried from the default, and at least one of the maximum, minimum, and average values ​​of the corresponding indicator data within 1 hour is calculated to obtain 1-hour precision data. The 1-hour precision data is stored in the Cluster_hours (i.e., the second storage space) of the InfluxDB time series database. If the storage time in Cluster_hours exceeds 1 year, the corresponding indicator data is automatically deleted.

[0100] For cluster-level aggregation, the aggregation object identifier includes the host identifier and the indicator identifier (including the host's own indicators and component indicators). If the aggregation period is 365 days, the corresponding indicator data within 365 days is queried from the default, and at least one of the maximum, minimum, and average values ​​of the corresponding indicator data within 365 days is calculated to obtain the 365-day precision data. The 365-day precision data is stored in the Cluster_days (i.e., the second storage space) of the InfluxDB time series database. If the storage time in Cluster_days exceeds 2 years, the corresponding indicator data is automatically deleted.

[0101] With the above technical solution, only the indicator data of each host in the current cluster is stored in the InfluxDB time series database. When the stored indicator data is subsequently queried, only each host in the current cluster can access the InfluxDB time series database. Therefore, the InfluxDB time series database is not prone to failures or start-up and shutdown that may cause the indicator monitoring equipment to crash, ensuring the operational stability of the indicator monitoring equipment.

[0102] Example 4

[0103] In order to implement the method of the embodiment of the present application, based on the same inventive concept, the embodiment of the present application further provides a data monitoring device, Figure 4 This is a schematic diagram of the structure of the data monitoring device in the embodiment of the present application. Figure 4 As shown, the data monitoring device is applied to an indicator monitoring device, which includes an InfluxDB time series database, specifically including:

[0104] The collection unit 401 is used to collect the index data of each host in the cluster; wherein the index data includes the host's own index data and the component index data deployed on the host;

[0105] The storage unit 402 is configured to store the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database is only accessible to each host in the cluster;

[0106] an aggregation unit 403 configured to query first indicator data from the first storage space based on a preset aggregation policy, perform a periodic aggregation operation on the first indicator data, and obtain aggregated second indicator data;

[0107] The display unit 404 is configured to display the second indicator data on the indicator monitoring device interface.

[0108] In some embodiments, the InfluxDB time series database further integrates a Telegraf plug-in; wherein, the Telegraf plug-in is used to collect component indicator data of services outside the Hadoop ecosystem deployed on the host.

[0109] In some embodiments, the aggregation strategy includes an aggregation category, an aggregation object identifier, and an aggregation time; the device includes: an aggregation unit 403, which is specifically used to perform periodic queries on the first storage space based on the aggregation category and the aggregation object identifier using the continuous query statement function of the InfluxDB time series database to obtain the first indicator data within the aggregation time; calculate at least one of the maximum value, minimum value, and average value of the first indicator data within the aggregation time to obtain the second indicator data corresponding to the aggregation object identifier.

[0110] In some embodiments, when the aggregation category indicates that the aggregation strategy is for host-level aggregation, the aggregation object identifier includes a host identifier and an indicator identifier; wherein, the indicator identifier is the identifier of the host's own indicator and / or component identifier; when the aggregation category indicates that the aggregation strategy is for cluster-level aggregation, the aggregation object identifier includes an indicator identifier.

[0111] In some embodiments, the method further includes: storing the second indicator data in a second storage space corresponding to the aggregation strategy; wherein different aggregation strategies correspond to different second storage spaces.

[0112] In some embodiments, the apparatus includes: a display unit 404, specifically configured to obtain at least one second indicator data corresponding to at least one aggregation strategy from the second storage space, and display the second indicator data on the indicator monitoring device interface.

[0113] In some embodiments, the apparatus includes: a storage unit 402 , specifically configured to adjust the indicator data in JSON format to indicator data in a storage format adapted to the InfluxDB time series database.

[0114] With the above technical solution, only the indicator data of each host in the current cluster is stored in the InfluxDB time series database. When the stored indicator data is subsequently queried, only each host in the current cluster can access the InfluxDB time series database. Therefore, the InfluxDB time series database is not prone to failures or start-up and shutdown that may cause the indicator monitoring equipment to crash, ensuring the operational stability of the indicator monitoring equipment.

[0115] The present application also provides another data monitoring device. Figure 5 This is a schematic diagram of the structure of the data monitoring device in the embodiment of the present application. Figure 5As shown, the data monitoring device includes: a processor 501 and a memory 502 configured to store a computer program that can be run on the processor;

[0116] The processor 501 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0117] Of course, in actual application, Figure 5 As shown, the various components in the data monitoring device are coupled together via a bus system 503. It is understood that the bus system 503 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 503 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 5 Various buses are labeled as bus system 503.

[0118] In practical applications, the processor may be at least one of an application-specific integrated circuit (ASIC), a digital signal processing device (DSPD), a programmable logic device (PLD), a field-programmable gate array (FPGA), a controller, a microcontroller, and a microprocessor. It is understood that for different devices, the electronic device used to implement the functions of the processor may also be other, and the embodiments of the present application do not specifically limit this.

[0119] The above-mentioned memory can be a volatile memory (volatile memory), such as a random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0120] In an exemplary embodiment, the present application also provides a computer-readable storage medium for storing a computer program.

[0121] Optionally, the computer-readable storage medium can be applied to any one of the methods in the embodiments of the present application, and the computer program enables the computer to execute the corresponding processes implemented by the processor in each method in the embodiments of the present application. For the sake of brevity, they are not repeated here.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0123] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0124] In addition, the functional units in the embodiments of the present invention can all be integrated into one processing module, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. It can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiments; and the above-mentioned storage medium includes various media that can store program codes, such as mobile storage devices, read-only memories (ROMs), random access memories (RAMs), magnetic disks or optical disks.

[0125] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0126] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0127] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments or device embodiments.

[0128] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A data monitoring method, applied to an indicator monitoring device, characterized in that: The indicator monitoring device includes an InfluxDB time series database, and the method includes: Collecting metrics data for each host in the cluster; wherein the metrics data includes the host's own metrics data and the metrics data of components deployed on the host; Storing the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database only allows each host in the cluster to access; Based on a preset aggregation strategy, query first indicator data from the first storage space, perform a periodic aggregation operation on the first indicator data, and obtain aggregated second indicator data; Displaying the second indicator data on the indicator monitoring device interface; The aggregation strategy includes aggregation category, aggregation object identifier and aggregation time; The step of querying the first indicator data from the first storage space based on a preset aggregation strategy and performing a periodic aggregation operation on the first indicator data to obtain aggregated second indicator data includes: According to the aggregation category and the aggregation object identifier, the first storage space is periodically queried using the continuous query statement function of the InfluxDB time series database to obtain first indicator data within the aggregation time; Calculate at least one of the maximum value, the minimum value, and the average value of the first indicator data within the aggregation time to obtain the second indicator data corresponding to the aggregation object identifier.

2. The method according to claim 1, characterized in that The InfluxDB time series database also integrates a Telegraf plug-in; wherein, the Telegraf plug-in is used to collect component indicator data of services outside the Hadoop ecosystem deployed on the host.

3. The method according to claim 1, characterized in that When the aggregation category indicates that the aggregation strategy is aggregation at the host level, the aggregation object identifier includes a host identifier and an indicator identifier; wherein the indicator identifier is an identifier of the host's own indicator and / or a component identifier; When the aggregation category indicates that the aggregation strategy is cluster-level aggregation, the aggregation object identifier includes an indicator identifier.

4. The method according to claim 1, wherein The method further comprises: The second indicator data is stored in a second storage space corresponding to the aggregation strategy; wherein different aggregation strategies correspond to different second storage spaces.

5. The method according to claim 4, characterized in that Displaying the second indicator data on the indicator monitoring device interface includes: At least one second indicator data corresponding to at least one aggregation strategy is obtained from the second storage space, and displayed on the indicator monitoring device interface.

6. The method according to claim 1, characterized in that Storing the indicator data of each host in the first storage space of the InfluxDB time series database includes: The indicator data in JSON format is adjusted to indicator data in a storage format suitable for the InfluxDB time series database.

7. A data monitoring device, applied to an indicator monitoring device, characterized in that: The indicator monitoring device includes an InfluxDB time series database, and the device includes: A collection unit, configured to collect index data of each host in the cluster; wherein the index data includes the host's own index data and the index data of components deployed on the host; A storage unit, configured to store the indicator data of each host in the first storage space of the InfluxDB time series database; wherein the InfluxDB time series database only allows each host in the cluster to access; An aggregation unit is configured to query first indicator data from the first storage space based on a preset aggregation strategy, perform periodic aggregation operations on the first indicator data, and obtain aggregated second indicator data; the aggregation strategy includes an aggregation category, an aggregation object identifier, and an aggregation time; the aggregation unit is specifically configured to use the continuous query statement function of the InfluxDB time series database to periodically query the first storage space based on the aggregation category and the aggregation object identifier to obtain the first indicator data within the aggregation time; and calculate at least one of the maximum value, minimum value, and average value of the first indicator data within the aggregation time to obtain the second indicator data corresponding to the aggregation object identifier; A display unit is configured to display the second indicator data on the indicator monitoring device interface.

8. An indicator monitoring device, characterized in that: The indicator monitoring device includes: a processor and a memory configured to store a computer program that can be run on the processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 6 when running the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images