Method, device and system for monitoring data

Abstract

The present invention discloses a method, device, and system for monitoring data, relating to the field of computer technology. A specific implementation of the method includes a server receiving indicator data collected by a client and utilizing a unified interface to perform read and write operations on a third-party time series database to monitor the storage and access of the indicator data. This overcomes the drawback of existing systems that require one-way writes to third-party time series databases, improves the efficiency of monitoring data, and enhances the liquidity and utilization of indicator data. By determining global business tags, the problem of poor correlation of collected indicator data due to non-standard business tags is resolved. Furthermore, by placing the received indicator data into a message queue, the problem of high concurrency of massive data is resolved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device and system for monitoring data. Background Art

[0002] As a monitoring system, the Prometheus system provides a complete data monitoring solution. Due to its ecological openness and multi-component flexibility, it has been widely deployed and applied in the data monitoring field.

[0003] In the process of implementing the present invention, the inventors discovered that the prior art has at least the following problems:

[0004] The Prometheus system stores metric data received on local storage media. However, storage space limitations pose challenges for storing massive amounts of data. When storing metric data in a third-party database, only writing to the database is supported, not directly reading data from it. This increases the complexity of monitoring read and write operations in multiple third-party databases, resulting in poor data flow and low data utilization.

[0005] The Prometheus system acquires metric data in a complex and diverse manner, such as from target data sources, stored by data gateways, or discovered through service discovery. This leads to a lack of correlation between data due to non-standard labeling of monitored data, making subsequent data correlation analysis difficult. When collecting massive amounts of data, the Prometheus system suffers from the drawbacks of high-concurrency data collection, resulting in data loss during the collection process. Furthermore, the centralized processing of massive amounts of data poses a risk of downtime for the Prometheus system. Summary of the Invention

[0006] In view of this, the embodiments of the present invention provide a method, device and system for monitoring data. The server receives the indicator data collected by the client and uses a unified interface to perform read and write operations on a third-party time series database to monitor the storage and reading of the indicator data. This overcomes the defect of the existing system's one-way writing to the third-party time series database, improves the efficiency of monitoring data, and improves the liquidity and utilization of the indicator data. By determining the global business tag, the problem of poor correlation of the collected indicator data due to non-standard business tags is solved. And by placing the received indicator data into a message queue, the problem of high concurrency of massive data is solved.

[0007] To achieve the above-mentioned purpose, according to one aspect of an embodiment of the present invention, a method for monitoring data is provided, which is applied to a Prometheus system and is characterized in that it includes: receiving indicator data, determining a global business tag corresponding to the indicator data according to a category of the indicator data, and adding the global business tag and the corresponding content to each of the indicator data; forming target indicator data and storing it in a message queue; obtaining the target indicator data from the message queue using a Prometheus server, monitoring the target indicator data according to a set monitoring policy, and storing the target indicator data in a time series database to monitor the storage of the indicator data; receiving a query request, and determining a time series database corresponding to the target indicator data according to the target indicator data in the query request; obtaining the target indicator data from the time series database through a Prometheus server to monitor the reading of the indicator data.

[0008] Optionally, the method for monitoring data is characterized in that:

[0009] According to the format rules of the first format, the data format of the indicator data based on the second format is converted into the first format.

[0010] Optionally, the method for monitoring data is characterized in that:

[0011] When the indicator data includes a non-numeric value, the non-numeric value is converted into a corresponding number according to a predefined correspondence between non-numerics and numbers.

[0012] Optionally, the method for monitoring data is characterized in that:

[0013] A query request for the target indicator data is obtained, and operators included in the query request are converted according to the grammatical rules of the time series database.

[0014] Optionally, the method for monitoring data is characterized in that:

[0015] The indicator data is stored in the time series database based on a remote process call model, and the indicator data in the time series database is read.

[0016] To achieve the above-mentioned purpose, according to the second aspect of an embodiment of the present invention, a method for monitoring data is provided, which is characterized in that it includes: collecting indicator data, and sending the indicator data and the global service tag to the network address according to a set period based on the configured global service tag and network address.

[0017] Optionally, the method for monitoring data is characterized in that:

[0018] The indicator data were collected using a data collection software package.

[0019] Optionally, the method for monitoring data is characterized in that:

[0020] The registration method included in the indicator data acquisition software package is used to add a custom indicator, and the indicator data corresponding to the custom indicator is collected using the data acquisition software package.

[0021] Optionally, the method for monitoring data is characterized in that:

[0022] Use the indicator data collection script to collect indicator data.

[0023] To achieve the above object, according to a third aspect of an embodiment of the present invention, a device for monitoring data is provided, characterized in that it is applied to a Prometheus system and includes: a data processing module and a data reading and writing module; wherein,

[0024] The data processing module is configured to receive indicator data, determine a global service tag corresponding to the indicator data according to the category of the indicator data, add the global service tag and corresponding content to each indicator data, form target indicator data, and store the target indicator data in a message queue;

[0025] The data reading and writing module is used to obtain the target indicator data from the message queue using the Prometheus server, monitor the target indicator data according to the set monitoring policy, and store the target indicator data in the time series database to monitor the storage of the indicator data; receive a query request, and determine the time series database corresponding to the target indicator data based on the target indicator data in the query request; obtain the target indicator data from the time series database through the Prometheus server to monitor the reading of the indicator data.

[0026] Optionally, the device for monitoring data is characterized in that:

[0027] According to the format rules of the first format, the data format of the indicator data based on the second format is converted into the first format.

[0028] Optionally, the device for monitoring data is characterized in that:

[0029] When the indicator data includes a non-numeric value, the non-numeric value is converted into a corresponding number according to a predefined correspondence between non-numerics and numbers.

[0030] Optionally, the device for monitoring data is characterized in that:

[0031] A query request for the target indicator data is obtained, and operators included in the query request are converted according to the grammatical rules of the time series database.

[0032] Optionally, the device for monitoring data is characterized in that:

[0033] The indicator data is stored in the time series database based on a remote process call model, and the indicator data in the time series database is read.

[0034] To achieve the above-mentioned purpose, according to the fourth aspect of an embodiment of the present invention, a device for monitoring data is provided, characterized in that it includes: a data acquisition module; wherein, the data acquisition module is used to collect indicator data, and according to the configured global service tag and network address, sends the indicator data and the global service tag to the network address according to a set period.

[0035] Optionally, the device for monitoring data is characterized in that:

[0036] The indicator data were collected using a data collection software package.

[0037] Optionally, the device for monitoring data is characterized in that:

[0038] The registration method included in the indicator data acquisition software package is used to add a custom indicator, and the indicator data corresponding to the custom indicator is collected using the data acquisition software package.

[0039] Optionally, the device for monitoring data is characterized in that:

[0040] Use the indicator data collection script to collect indicator data.

[0041] To achieve the above objectives, according to a fifth aspect of an embodiment of the present invention, a system 800 for monitoring data is provided, comprising the apparatus 600 for monitoring data provided by the third aspect and the apparatus 700 for monitoring data provided by the fourth aspect.

[0042] To achieve the above-mentioned purpose, according to the sixth aspect of an embodiment of the present invention, an electronic device for monitoring data is provided, characterized in that it includes: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by the one or more processors, the one or more processors implement any of the methods described in the above-mentioned methods for monitoring data.

[0043] To achieve the above-mentioned purpose, according to the seventh aspect of an embodiment of the present invention, a computer-readable medium is provided, on which a computer program is stored, characterized in that when the program is executed by a processor, any method as described in the above-mentioned method for monitoring data is implemented.

[0044] An embodiment of the above invention has the following advantages or beneficial effects: the server receives the indicator data collected by the client, and uses a unified interface to perform read and write operations on a third-party time series database to monitor the storage and reading of the indicator data, overcoming the defect of the existing system's one-way writing to the third-party time series database, improving the efficiency of monitoring data, and improving the liquidity and utilization of the indicator data; by determining the global business tag, the problem of poor correlation of the collected indicator data caused by the non-standard business tag is solved; and by placing the received indicator data into a message queue, the problem of high concurrency of massive data is solved.

[0045] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with specific embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The accompanying drawings are provided for a better understanding of the present invention and are not intended to limit the present invention.

[0047] Figure 1 This is a flow chart of a method for monitoring data provided by one embodiment of the present invention;

[0048] Figure 2 This is a flow chart of a method for collecting indicator data provided by one embodiment of the present invention;

[0049] Figure 3 This is a schematic diagram of a process flow for monitoring data provided by an embodiment of the present invention;

[0050] Figure 4 This is a schematic diagram of the existing Prometheus system;

[0051] Figure 5 Schematic diagram of an improved existing Prometheus system provided by one embodiment of the present invention;

[0052] Figure 6 This is a schematic structural diagram of a device for monitoring data provided by one embodiment of the present invention;

[0053] Figure 7 This is a schematic structural diagram of a device for collecting indicator data provided by one embodiment of the present invention;

[0054] Figure 8 This is a schematic diagram of the structure of a data monitoring system provided by one embodiment of the present invention;

[0055] Figure 9 is an exemplary system architecture diagram in which embodiments of the present invention may be applied;

[0056] Figure 10It is a schematic diagram of the structure of a computer system of a terminal device or server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following description of exemplary embodiments of the present invention is made in conjunction with the accompanying drawings, in which various details of the embodiments of the present invention are included to facilitate understanding. These details should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0058] like Figure 1 As shown, an embodiment of the present invention provides a method for monitoring data on a server, which may include the following steps:

[0059] Step S101: receiving indicator data, determining a global service tag corresponding to the indicator data according to the category of the indicator data, adding the global service tag to each indicator data; forming target indicator data and storing it in a message queue;

[0060] Specifically, the indicator data is the data monitored by the Prometheus monitoring system, for example: including indicator data related to physical machines (for example, the temperature and hardware fault information of routers, switches, and servers; and also including system indicator data running on physical machines, for example, CPU utilization, memory utilization, hard disk utilization, network card traffic, TCP status, number of processes, etc.), service indicator data (for example, the operating indicator data of Nginx, Tomcat, PHP, MySQL, Redis and other services used by the system); business indicator data (indicator data generated by specific business scenarios, for example, how many orders are generated per minute for e-commerce websites, etc.). It can be understood that the present invention is an improvement based on the component framework of the Prometheus system, and the range of the monitored indicator data is similar to the data range processed by the Prometheus system; the format of the indicator data included in the Prometheus system is as follows:

[0061] metric{tagk=tagv,tagk1=tagv1,…}value;

[0062] Among them, metric is the identifier of the monitored indicator, tagk and tagk1 are the names of the parameters associated with the monitored indicator; tagv and tagv1 are the values ​​corresponding to the parameters tagk and tagk1; value is the value corresponding to the metric.

[0063] Furthermore, according to the category of the indicator data, the global business tag corresponding to the indicator data is determined; wherein the category of the indicator data is to distinguish the source of the indicator data, business type, etc., for example, the indicator data of different clusters are divided into different categories; the indicator data from different network address ranges are divided into different categories; the indicator data of different businesses are divided into different categories (for example: the indicator data of logistics and e-commerce are divided into different categories); further, according to the category of the indicator data, the global business tag corresponding to the indicator data is determined, for example, the global business tag corresponding to the indicator data from cluster 1 is determined to be serviceId="cluster11111"; the present invention does not limit the specific content of the category of the indicator data and the specific content of the global business tag.

[0064] Furthermore, the global business tag is added to each of the indicator data; for example, the received indicator data is identified as finishJobAvgTime, which may originate from different services, such as Hbase, Hadoop, Spark, etc., and the global business tag can be used to distinguish the source of the collected indicator data according to the category and when the indicator data identifier is the same. It can be seen that the use of global business tags makes the received indicator data unique and makes the indicator data relevant, overcoming the problem of low data relevance caused by the non-standard indicator data tags; for example, setting the global business tag serviceID = "123456abcdef" to correspond to the indicator data of the HBase service, and the global business tag serviceID = "789010abcdef" to correspond to the indicator data of the Hadoop service, and adding the above global business tags to each indicator data in the corresponding batch of indicator data to form the target indicator data; the above steps solve the problem of poor data relevance caused by the non-uniformity of indicator data tags; preferably, according to the category of the indicator data, the global business tag is determined according to the received global business tag; or according to the customized global business tag configuration rule, the received global business tag is converted into a global business tag that matches the configuration rule. At the same time, if the client sends other business tags at the same time as sending the global business tag, the server can also add the other business tags to each indicator data in a batch of indicator data.

[0065] Furthermore, the target indicator data is stored in a message queue. Specifically, the existing Prometheus system obtains data from the target network unit or data gateway according to a set period, and is unable to complete high-concurrency data collection. In addition, after stress testing, the following example data is obtained: the amount of relevant data generated by 40,000 data processing units can be 219GB / day, the number of indicators generated is: 15 billion / day, and about 173,000 indicators are generated per second. Massive unscheduled data may reduce the stability of the Prometheus server. Therefore, before sending massive indicator data to the Prometheus server, a message queue (for example: Kafka) is used to process the indicator data, and the message queue caching mechanism is used to overcome the concurrent processing problem of massive data. For example: One embodiment of the present invention is that the data gateway pushes the received indicator data to the message queue by setting topic partitions (for example: setting topic1 to indicate HBase-related indicator data, setting topic2 to indicate Hadoop-related indicator data). It can be understood that by setting topic partitions (topic), the data reading and writing of different businesses in the message queue (kafka) can be isolated. Preferably, the Prometheus server can obtain indicator data from the message queue according to a set period, or the message queue can push the indicator data to the Prometheus server according to a set period; thereby partially solving the problem of task blocking or poor server stability caused by directly pushing massive data to the Prometheus server.

[0066] Furthermore, the data format of the indicator data based on the second format is converted into the first format according to the format rules of the first format. Specifically, the first format is a data format supported by the existing Prometheus system, and the second format is a data format inconsistent with the data format supported by the existing Prometheus system. For example, the second format is any data format that can be converted into the first format. The present invention does not limit the specific format of the second format.

[0067] The data format of the indicator data may be based on the data format defined by the Prometheus system (i.e., the first format). An example of the first format is as follows:

[0068] metric{tagk=tagv,tagk1=tagv1,…}value; where metric is the identifier of the monitored metric, tagk and tagk1 are the names of the parameters associated with the monitored metric; tagv and tagv1 are the values ​​corresponding to the parameters tagk and tagk1; and value is the value corresponding to the metric. Furthermore, the data format of the accepted metric data can also be the second format. The second format is an example of the data format defined based on the OpenTSDB database. The second format is as follows:

[0069] [{"metric":"query_info_12345",

[0070] "value":"200",

[0071] "tags":{

[0072] "status":"failed",

[0073] "cluster":"AAAA"

[0074] }}]

[0075] Wherein, metric is the identifier of the monitored indicator. In the above example, the metric is "query_info_12345". tags contains the name of the parameter associated with the indicator, for example, parameter 1 is "status" and the value is "failed". According to the format rules of the first format, the indicator data format based on the second format is converted to the first format. For example, the indicator data example based on the second format is converted to the first format as follows:

[0076] query_info_12345{status="failed", cluster="AAAA"}200

[0077] It can be understood that according to the format rules of the first format, the received indicator data in the second format is converted into the first format according to the corresponding labels or contents, and then the indicator data in the first format is further monitored and analyzed, thereby expanding the scope of the monitored indicator data and reducing the complexity of data processing due to the unification of the data format.

[0078] Step S102: using the Prometheus server to obtain the target indicator data from the message queue, monitoring the target indicator data according to the set monitoring strategy, and storing the target indicator data in the time series database to monitor the storage of the indicator data; receiving a query request, and determining the time series database corresponding to the target indicator data according to the target indicator data in the query request; obtaining the target indicator data from the time series database through the Prometheus server to monitor the reading of the indicator data.

[0079] Specifically, the target indicator data is obtained from the message queue using the Prometheus server (i.e., the monitoring server). Preferably, the indicator data can be obtained from the message queue according to a set period, for example, the set period is set to 30 seconds. It is understandable that the set period is determined according to the specific business or business scenario, as well as the frequency and granularity of the user monitoring data. For example, for indicator data with higher real-time requirements, a shorter period can be set, for example, it can be set to 1 second. According to the set monitoring strategy, the target indicator data is monitored, and the Prometheus server monitors the received indicator data according to the set monitoring strategy. It is understandable that the monitoring strategy is set according to the monitored business scenario and indicator, including setting trigger alarm rules, setting thresholds, setting monitoring periods, setting monitored indicator data, etc. Monitoring the received indicator data according to the set monitoring strategy is a function of the existing Prometheus server, and the present invention does not further explore the functions that the Prometheus server already has.

[0080] Furthermore, according to the description of step S101, the received data is processed to form target indicator data with a consistent format, and the Prometheus server stores the target indicator data in a time series database to monitor the storage of the indicator data; wherein, the HBase-based time series database OpenTSDB is used as the time series database. It can be understood that the indicator data can be stored in multiple time series databases according to the category of the indicator data. Each time series database can be a third-party database. By storing the data in multiple third-party time series databases, the problem of local storage capacity limitation is solved, and this step can monitor the storage of indicator data in multiple time series databases.

[0081] Furthermore, according to the rules of the time series database (OpenTSDB), the storage of non-numeric indicator data is not supported. Therefore, the indicator data value is converted to a corresponding number based on the predefined correspondence between non-numeric and numeric values. For example, the value "active" indicating the service status is mapped to the number 1, and "standby" is mapped to the number 0. That is, when the indicator data contains non-numeric values, the non-numeric values ​​are converted to corresponding numbers based on the predefined correspondence between non-numeric and numeric values.

[0082] Furthermore, a query request is received, and based on the target indicator data in the query request, the time series database corresponding to the target indicator data is determined; the target indicator data is obtained from the time series database to monitor the reading of the indicator data. Specifically, a query request for target indicator data is received, the time series database corresponding to the target indicator data is determined, and the target indicator data is obtained, thereby monitoring the reading of the indicator data; compared with the existing Prometheus system, by supporting the use of the Prometheus server to monitor the reading of the target time series database and obtaining the indicator data in the time series database according to the request, the liquidity and utilization of massive data are increased, and the indicator data can be more conveniently analyzed to generate indicator data analysis information. At the same time, the use of the Prometheus server to directly obtain the indicator data in the time series database improves the efficiency of monitoring and management of third-party time series databases, and overcomes the defect in the existing technology that the Prometheus server cannot read data from third-party databases.

[0083] Since there are inconsistencies between the query syntax of the time series database (OpenTSDB) and the query syntax of the Prometheus system data format, for example, the Prometheus system data format contains four operator symbols: "=", "!=", "=~", and "!~", while the OpenTSDB data format does not contain the above operator symbols; the query request for obtaining the target indicator data is converted according to the grammatical rules of the time series database. For example, the "=" and "!=" in the query request can be converted to the filter condition operator "literal_or" in the grammatical rules of the OpenTSDB format; while the fuzzy matching operator symbols "=~" and "!~" do not have matching filter condition operators in the OpenTSDB format, preferably, the return result corresponding to the query request containing the fuzzy matching operator symbol can be obtained first, and then based on the return result, the "or" operator is used to convert it to the grammatical rules of the time series database defined by OpenTSDB, and further query operations are performed. It can be understood that after the Prometheus server obtains the target indicator data from OpenTSDB, further data calculation and processing are performed in the Prometheus server. That is, a query request for obtaining the target indicator data is converted into operators included in the query request according to the grammatical rules of the time series database.

[0084] Furthermore, the indicator data is stored in the target database and the indicator data of the target database is read based on a remote procedure call model. Specifically, the monitoring server (Prometheus server) performs read and write operations on a time series database (e.g., OpenTSDB) through a remote procedure call model (e.g., gRPC); and monitors the read and write operations on the time series database, wherein gRPC can define an interface through a structured data serialization method (e.g., protobuf), which can serialize data into binary encoding and compress the data, thereby reducing the amount of data transmitted and improving data transmission performance.

[0085] like Figure 2 As shown, an embodiment of the present invention provides a method for a client to collect indicator data, which may include the following steps:

[0086] Step S201: collecting indicator data, and sending the indicator data and the global service tag to the network address according to the configured global service tag and network address at a set period.

[0087] There are two methods for the client to collect indicator data as follows: wherein, the client is a server or computer used to collect indicator data; the present invention does not limit the specific device to which the client belongs.

[0088] The first method: use the data collection software package to collect indicator data.

[0089] Specifically, one embodiment of the present invention is that the client uses an application developed in Java to collect indicator data. The client uses a data collection software package (for example, javaagent.jar). The client can use the following command to add the following parameters to the Java virtual machine parameters: -javaagent:{javaagent.jar}={IP:Port},labels={serviceId:abcd},file={a.yml},jobName={abc}. Specifically, the parameter description of this example is shown in Table 1.

[0090]

[0091]

[0092] Table 1 Java application collection indicator data parameters

[0093] Specifically, the data collection software package includes the following features: support for adding global business tags via startup parameters; and direct push of metric data to the data gateway by configuring the data gateway's network address. Furthermore, after the application starts, the client configures global business tags and dynamically configures the data gateway's network address. This allows the client to scan and collect data using Java's probe technology. The collected metric data is then sent to the configured data gateway via Hypertext Transfer Protocol. Furthermore, the data gateway pushes the received metric data to a message queue (Kafka) via topic information, from which the Prometheus server can retrieve the metric data.

[0094] Furthermore, when the client needs to send indicator data of a custom indicator based on its own business logic, it can use the registration method included in the data collection software package (javaagent.jar) to report the data and complete the data collection. For example, you can use the following method to register a custom metric: metricRegistry.register(name, metricType), where metricRegistry.register is the registration method and metricType is the metric type, including five types: Gauge, Counter, Meter, Timer, and Histogram. Gauge records the instantaneous value of the metric, such as the usage of the current Java virtual machine, including memory utilization, CPU utilization, and thread usage status. Counter generates cumulative metrics through increment and decrement operations, such as the total number of tasks submitted in a cluster. Meter counts the frequency of events, such as the network traffic in the last 1 minute, 5 minutes, and 15 minutes, for aggregate calculation of metrics. Timer counts the distribution of metrics, such as the request frequency and time consumption of a certain interface. Histogram counts the numerical distribution of metric data, such as the minimum, maximum, average, median, 75th percentile, and 90th percentile of the statistical value. The client can define indicators and place them at the location where the logic code throws an exception. When an exception occurs, the logic code of the exception-throwing part is triggered to send the indicator data corresponding to the indicator to the data gateway. It can be seen that the client can call the registration method contained in the data acquisition software package in the logic code part of the exception-throwing part to collect the indicator data corresponding to the custom indicator, that is, use the registration method contained in the indicator data acquisition software package to add a custom indicator, and use the data acquisition software package to collect the indicator data corresponding to the custom indicator.

[0095] The second method: Use the indicator data collection script to collect indicator data.

[0096] Specifically, for applications developed in languages ​​other than Java, the client can use the indicator data collection script to collect indicator data, according to the configured global business tags and network address, and send the customized indicators in the data format set by the data gateway using the Hypertext Transfer Protocol to the configured network address (for example: the network address of the data gateway). The language used to develop the indicator data collection script can be Phython, Perl, etc. The present invention does not limit the specific content and implementation method of the script.

[0097] like Figure 3As shown, an embodiment of the present invention provides a flowchart of monitoring data, and the method may include the following steps:

[0098] Step S301: The client collects indicator data.

[0099] Specifically, the description of the client using a data collection software package or a data collection script to collect indicator data is consistent with step S201 and will not be repeated here. That is, the indicator data is collected and, based on the configured global service tag and network address, the indicator data and the global service tag are sent to the network address according to a set period. Furthermore, the indicator data is collected using the data collection software package. The registration method included in the indicator data collection software package is used to add a custom indicator, and the indicator data corresponding to the custom indicator is collected using the data collection software package. Furthermore, the indicator data corresponding to the service identifier of the indicator data collection script is collected using the indicator data collection script.

[0100] Step S302: The data gateway receives indicator data from the client.

[0101] Specifically, the data gateway receives the indicator data, determines the global business tag corresponding to the indicator data according to the category of the indicator data, and adds the global business tag to each of the indicator data; forms the target indicator data and stores it in the message queue;

[0102] The description of receiving indicator data and processing indicator data is consistent with step S101 and will not be repeated here.

[0103] Step S303: The data gateway puts the target indicator data into the message queue.

[0104] Specifically, the description of the data gateway putting the target indicator data into the message queue is consistent with step S101 and will not be repeated here.

[0105] Step S304-Step S305: Use the Prometheus server to obtain the target indicator data from the message queue, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in the time series database to monitor the storage of the indicator data; receive a query request, and determine the time series database corresponding to the target indicator data according to the target indicator data in the query request; obtain the target indicator data from the time series database through the Prometheus server to monitor the reading of the indicator data.

[0106] The description of the Prometheus server obtaining data from the message queue and monitoring the storage and reading of the time series database is consistent with step S102 and will not be repeated here.

[0107] Figure 4 A schematic diagram of an existing Prometheus system is shown;

[0108] Figure 5 A schematic diagram of an improved Prometheus system provided by an embodiment of the present invention is shown;

[0109] The following comparison Figure 4 and Figure 5 An embodiment of the present invention will be described.

[0110] 1) In the existing Prometheus system, the indicator data is stored locally, and the storage of massive data has the problem of capacity limitation. Although the indicator data can be stored in a third-party database (such as Figure 4 However, the interaction with third-party databases is one-way and only supports one-way writes. That is, the indicator data stored in the third-party database is static, with poor liquidity and low data utilization, which leads to loss of data relevance and possible data format incompatibility.

[0111] contrast Figure 4 ,like Figure 5 As shown, the present invention implements the bidirectional operation of Prometheus server to store and read the time series database and monitor the storage and reading, which is used for monitoring and storing and reading indicator data based on multiple third-party time series databases. For example, the Prometheus server used for monitoring the third-party database (for example, Figure 5 The TSDB shown, namely the OpenTSDB described in this invention, stores and reads indicator data and monitors the storage and reading of third-party databases. It uses the unified interface of the Prometheus server to perform read and write operations on the third-party database, which improves the efficiency of monitoring indicator data, improves the liquidity of indicator data, and improves the utilization rate of indicator data. It also solves the Prometheus performance problem caused by storing massive data on the local hard disk.

[0112] That is, the target indicator data is obtained from the message queue using the Prometheus server, the target indicator data is monitored according to the set monitoring strategy, and the target indicator data is stored in a time series database to monitor the storage of the indicator data; a query request is received, and the time series database corresponding to the target indicator data is determined according to the target indicator data in the query request; the target indicator data is obtained from the time series database through the Prometheus server to monitor the reading of the indicator data.

[0113] 2) If Figure 4As shown in the figure, in the existing Prometheus system, Prometheus periodically obtains the indicator data of statically configured monitoring targets or data gateways (Pushgateway), and can also obtain the indicator data of the data container kubernetes (k8s) through service discovery. It can be seen that the way to collect data is complex and diverse, and the indicator data labels used for collection may be non-standard, resulting in data silos (i.e., lack of data relevance), which makes subsequent data correlation analysis problematic; and due to Figure 4 It can be seen that the k8s machine and the monitoring service are highly coupled.

[0114] like Figure 5 As shown, the Prometheus server can periodically obtain data from the message queue (i.e., target indicator data). Compared with the existing Prometheus system, the data collection method is converted from multiple to a single method of obtaining data from the message queue (kafka), and the target indicator data of the message queue (kafka) is obtained by processing the received indicator data through the data gateway (Pushgateway). The data gateway determines the global business label according to the category of the received indicator data and adds the global business table to the indicator data. Through this technical solution, the indicator data can be obtained uniformly and the relevance and uniqueness of the indicator data can be determined. Moreover, it can also be decoupled from k8s, that is, the indicator data from k8s is not directly collected.

[0115] That is, receiving indicator data, determining the global service tag corresponding to the indicator data according to the category of the indicator data, adding the global service tag to each of the indicator data; forming target indicator data and storing it in a message queue;

[0116] 3) As described in 2), one embodiment of the present invention uses a message queue (Kafka) to address the problem of collecting massive amounts of data with high concurrency. This technical solution is a new component added to the existing Prometheus system. The message queue cache mechanism can partially address the high throughput and low latency issues associated with the monitoring server, while also ensuring good fault tolerance in high-concurrency situations.

[0117] like Figure 6 As shown, the embodiment of the present invention provides a device 600 for monitoring data, including: a data processing module 601 and a data reading and writing module 602; wherein,

[0118] The data processing module 601 is configured to receive indicator data, determine a global service tag corresponding to the indicator data according to the category of the indicator data, and add the global service tag to each indicator data; form target indicator data and store it in a message queue;

[0119] The data reading and writing module 602 is used to use the Prometheus server to obtain the target indicator data from the message queue, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in the time series database to monitor the storage of the indicator data; receive a query request, and determine the time series database corresponding to the target indicator data based on the target indicator data in the query request; obtain the target indicator data from the time series database through the Prometheus server to monitor the reading of the indicator data.

[0120] Optionally, the data processing module 601 is further configured to convert the data format of the indicator data based on the second format into the first format according to the format rules of the first format.

[0121] Optionally, the data processing module 601 is further configured to, when the indicator data includes a non-numeric value, convert the non-numeric value into a corresponding number according to a predefined correspondence between non-numerics and numbers.

[0122] Optionally, the data reading and writing module 602 is further configured to obtain a query request for the target indicator data, and convert operators included in the query request according to grammatical rules of a time series database.

[0123] Optionally, the data reading and writing module 602 is further configured to store the indicator data in the time series database and read the indicator data from the time series database based on a remote process call model.

[0124] like Figure 7 As shown, the embodiment of the present invention provides a device 700 for monitoring data, including: a data acquisition module 701; wherein,

[0125] The data collection module 701 is used to collect indicator data, and according to the configured global service tag and network address, send the indicator data and the global service tag to the network address according to a set period.

[0126] Optionally, the data acquisition module 701 is further configured to acquire indicator data using a data acquisition software package.

[0127] Optionally, the data acquisition module 701 is further configured to use a registration method included in the indicator data acquisition software package to add a custom indicator, and use the data acquisition software package to collect indicator data corresponding to the custom indicator.

[0128] Optionally, the data collection module 701 is further configured to collect indicator data using an indicator data collection script.

[0129] like Figure 8 As shown, an embodiment of the present invention provides a system for monitoring data, including: Figure 6 The device for monitoring data shown, and Figure 7 The device for monitoring data is shown.

[0130] An embodiment of the present invention also provides an electronic device for monitoring data, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method provided in any of the above embodiments.

[0131] An embodiment of the present invention further provides a computer-readable medium having a computer program stored thereon, and when the program is executed by a processor, the method provided in any of the above embodiments is implemented.

[0132] Figure 9 An exemplary system architecture 900 is shown to which the method for monitoring data or the apparatus for monitoring data according to an embodiment of the present invention may be applied.

[0133] like Figure 9 As shown, system architecture 900 may include terminal devices 901, 902, 903, a network 904, and a server 905. Network 904 is used to provide a medium for communication links between terminal devices 901, 902, 903 and server 905. Network 904 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0134] Users can use terminal devices 901, 902, and 903 to interact with server 905 via network 904 to receive or send messages, etc. Terminal devices 901, 902, and 903 can be installed with various client applications, such as web browser applications, search applications, instant messaging tools, and email clients.

[0135] The terminal devices 901 , 902 , and 903 may be various electronic devices having a display screen and supporting web browsing, including but not limited to servers, smart phones, tablet computers, laptop computers, and desktop computers, etc.

[0136] Server 905 may be a server that provides various services, such as a background management server that supports data monitoring requests submitted by users using terminal devices 901, 902, and 903. The background management server may process received data such as data monitoring requests, store received indicator data, and feed back indicator data analysis results to the terminal device.

[0137] It should be noted that the method for monitoring data provided in the embodiment of the present invention is generally executed by the server 905 , and accordingly, the device for monitoring data is generally set in the server 905 .

[0138] It should be understood that Figure 9 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0139] Reference below Figure 10 , which shows a schematic structural diagram of a computer system 1000 of a terminal device suitable for implementing an embodiment of the present invention. Figure 10 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0140] like Figure 10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the system 1000 are also stored in the RAM 1003. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0141] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, and the like; an output section 1007 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 1008 including a hard disk; and a communication section 1009 including a network interface card such as a LAN card or a modem. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 1010 as needed, so that computer programs read therefrom can be installed into the storage section 1008 as needed.

[0142] In particular, according to the embodiments disclosed herein, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed herein include a computer program product comprising a computer program carried on a computer-readable medium, the computer program comprising program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 1009, and / or installed from a removable medium 1011. When the computer program is executed by the central processing unit (CPU) 1001, the above-mentioned functions defined in the system of the present invention are performed.

[0143] It should be noted that the computer-readable medium described in the present invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media can include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In the present invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical fiber cable, RF, or any suitable combination thereof.

[0144] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The modules and / or units involved in the embodiments of the present invention may be implemented in software or in hardware. The modules and / or units described may also be provided in a processor. For example, they may be described as follows: a processor includes a data processing module, a data reading and writing module, and a data acquisition module. The names of these modules do not, in some cases, constitute a limitation on the modules themselves. For example, the data processing module may also be described as a module that "receives indicator data, converts the data format of the indicator data into a first format, and adds a global business tag to the indicator data."

[0146] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs. When the above one or more programs are executed by a device, the device includes: receiving indicator data, determining the global business tag corresponding to the indicator data based on the category of the indicator data, and adding the global business tag to each of the indicator data; forming target indicator data and storing it in a message queue; using a Prometheus server to obtain the target indicator data from the message queue, monitoring the target indicator data according to a set monitoring policy, and storing the target indicator data in a time series database to monitor the storage of the indicator data; receiving a query request, determining the time series database corresponding to the target indicator data based on the target indicator data in the query request; obtaining the target indicator data from the time series database through a Prometheus server to monitor the reading of the indicator data. Collecting indicator data, and sending the indicator data and the global business tag to the network address according to a set period based on the configured global business tag and network address.

[0147] According to the technical solution of the embodiments of the present invention, the server receives the indicator data collected by the client and uses a unified interface to perform read and write operations on a third-party time series database to monitor the storage and reading of the indicator data. This overcomes the defect of the existing system's one-way storage of the third-party time series database, improves the efficiency of monitoring data, and improves the liquidity and utilization of the indicator data. By determining the global business tag, the problem of poor correlation of the collected indicator data due to non-standard business tags is solved. And by placing the received indicator data into a message queue, the problem of high concurrency of massive data is solved.

[0148] The above specific embodiments do not limit the scope of protection of the present invention. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method for monitoring data, characterized in that: Applied to the Prometheus system, including: Receive indicator data, determine the global service tag corresponding to the indicator data according to the category of the indicator data, and add the global service tag to each indicator data; form target indicator data and store it in a message queue; Obtain the target indicator data from the message queue using a Prometheus server, monitor the target indicator data according to a set monitoring strategy, and store the target indicator data in a time series database to monitor the storage of the indicator data; Obtaining a query request for the target indicator data, and converting the operators included in the query request according to the grammatical rules of the time series database; wherein the operators are the parts where the query grammar of the Prometheus system is inconsistent with the query grammar of the time series database; Receive a query request, and determine a time series database corresponding to the target indicator data based on the target indicator data in the query request; obtain the target indicator data from the time series database through a Prometheus server to monitor the reading of the indicator data.

2. The method according to claim 1, characterized in that According to the format rules of the first format, the data format of the indicator data based on the second format is converted into the first format; wherein the first format is a data format supported by the Prometheus system, and the second format is a data format inconsistent with the data format supported by the Prometheus system.

3. The method according to claim 1, characterized in that When the indicator data includes a non-numeric value, the non-numeric value is converted into a corresponding number according to a predefined correspondence between non-numerics and numbers.

4. The method according to claim 1, wherein The indicator data is stored in the time series database based on a remote process call model, and the indicator data in the time series database is read.

5. A device for monitoring data, characterized in that: Applied to the Prometheus system, including: data processing module and data reading and writing module; among them, The data processing module is configured to receive indicator data, determine a global service tag corresponding to the indicator data according to the category of the indicator data, add the global service tag and corresponding content to each indicator data, form target indicator data, and store the target indicator data in a message queue; The data reading and writing module is used to use the Prometheus server to obtain the target indicator data from the message queue, monitor the target indicator data according to the set monitoring strategy, and store the target indicator data in the time series database to monitor the storage of the indicator data; obtain the query request for the target indicator data, and convert the operator contained in the query request according to the grammatical rules of the time series database; wherein the operator is the part where the query grammar of the Prometheus system is inconsistent with the query grammar of the time series database; receive the query request, and determine the time series database corresponding to the target indicator data according to the target indicator data in the query request; obtain the target indicator data from the time series database through the Prometheus server to monitor the reading of the indicator data.

6. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 4.

7. A computer-readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images