Monitoring indicator collection and storage method, system and storage medium based on message queue

Abstract

The present invention discloses a monitoring indicator collection and storage method, system and storage medium based on a message queue, and uses RabbitMQ message queue middleware for data communication. The collection method includes: pushing the monitoring platform indicators to be collected to the message queue; collecting the monitoring platform indicators in the message queue, and pulling the corresponding data source indicator data through the mapping relationship; extracting the indicator value of the data source indicator data, and filling it into the corresponding monitoring platform indicator to form the monitoring platform indicator data. Thus, it is possible to collect the massive monitoring data of each platform without intrusion, standardize and unify the indicator data of multiple business platforms, and store them to support the direct use of data for monitoring operation and maintenance, thereby ensuring the normal and efficient operation of the operation and maintenance business.

Description

Technical Field

[0001] The present invention relates to the field of Internet operation and maintenance technology, and in particular to a monitoring indicator collection and storage method, system and storage medium based on a message queue. Background Art

[0002] The main responsibility of operation and maintenance is to troubleshoot service risks and handle service failures; therefore, it is necessary to use the operation and maintenance platform to monitor the service operation status and resource consumption in real time and receive fault alarms generated by the operation and maintenance platform. Or, in order to improve service performance and optimize service costs, it is necessary to use the operation and maintenance platform to collect service CPU, memory, QPS, response time, traffic data and other information; therefore, the operation and maintenance platform needs to solve the problem of monitoring indicator collection and storage.

[0003] As the businesses carried by the operation and maintenance platform continue to be enriched and improved, more and more components are used in actual scenarios, and the corresponding monitoring data is also increasing sharply. However, the monitoring data standards and formats of various scenarios are not standardized and unified, and most of the indicators obtained from various platforms cannot be used directly. This makes it difficult to achieve the demand for non-invasive collection of massive monitoring data from various platforms and direct use of the collected data on the monitoring and operation and maintenance platform. Summary of the invention

[0004] In view of this, the present application solves the problem of how to non-invasively collect massive monitoring data from various platforms, standardize and store the indicator data of multiple business platforms by providing a monitoring indicator collection and storage method, system and storage medium based on a message queue, so as to support the direct use of data for monitoring operation and maintenance, thereby ensuring the normal and efficient operation of operation and maintenance business.

[0005] On the one hand, a monitoring indicator collection and storage method based on a message queue is provided, which uses RabbitMQ message queue middleware for data communication. The collection method includes:

[0006] Push the monitoring platform indicators to be collected to the message queue;

[0007] Collect monitoring platform indicators in the message queue, and pull pre-configured data source indicator data through mapping relationships. The data source indicator data includes indicator values.

[0008] The indicator value of the data source indicator data is extracted and filled into the corresponding monitoring platform indicator to form the monitoring platform indicator data.

[0009] Furthermore, before pushing the monitoring platform indicators to be collected to the message queue, it also includes: configuring the data source of the monitoring platform; the data source is generated by the business platform;

[0010] Constructing a monitoring platform indicator paradigm; the monitoring platform indicator paradigm at least includes the following attributes: indicator name, indicator value;

[0011] Based on the monitoring scenario requirements, define the monitoring platform indicators to be collected.

[0012] Furthermore, before pulling the corresponding data source indicator data through the mapping relationship, it also includes mapping the configured data source indicator to the monitoring platform indicator, specifically including:

[0013] Get the names of all indicators of the data source;

[0014] Select the data source indicator name that matches the indicator business of the monitoring platform, and map the corresponding data source indicator to the monitoring platform indicator.

[0015] Furthermore, it also includes:

[0016] Based on the non-intrusive principle, configure the data collection method of the data source;

[0017] Configure the collection period of the monitoring platform indicators based on the update frequency of the data source indicator values;

[0018] Furthermore, the data collection method of configuring the data source specifically includes:

[0019] Determine whether the data source provides a query interface; if so, configure the query interface of the data source to the query address of the message queue data source; if not, directly configure the access address of the data source to the query address of the message queue data source.

[0020] Furthermore, based on the data source indicator value update frequency, the collection period of the monitoring platform indicator is configured, which also includes:

[0021] Configure the start and stop status of data source access to control the communication between the data source and the message queue.

[0022] Furthermore, before collecting the monitoring platform indicators in the message queue, it also includes:

[0023] Start multiple collection programs, set the subscription Exchange Type of the message queue for each collection program; set each collection program to access all channels of the message queue through polling.

[0024] Furthermore, the subscription mode of the channel of the message queue is enabled, and the multiple collection programs concurrently collect the channel subscriptions of the message queue and push the monitoring platform indicators to be collected.

[0025] Further, the configured data source is mapped to the monitoring platform indicator;

[0026] Configure the Exchange Type of the message queue based on the correspondence between the monitoring platform indicators and the data source type.

[0027] Further, mapping the configured data source to the monitoring platform indicator specifically includes: constructing a data structure of the monitoring platform indicator corresponding to the data source according to the corresponding relationship between the monitoring platform indicator and the data source;

[0028] Set the data source Tag; write the data source Tag into the structure data of the corresponding monitoring platform indicator.

[0029] Furthermore, the monitoring platform indicators to be collected are pushed to the message queue, specifically including:

[0030] Traverse the constructed data structure;

[0031] Based on the data structure, the monitoring platform indicators associated with the data source are concurrently pushed to the message queue, and the message queue pushes the monitoring platform indicators to the specified message queue channel based on the Exchange Type configured by the data source type.

[0032] Furthermore, the updated indicator value corresponding to the data source indicator is pulled through the mapping relationship, specifically including:

[0033] According to the collection cycle, the collection program pulls the monitoring platform indicators to be collected from the channel of the message queue; according to the query method of the data source corresponding to the monitoring platform indicators, the collection program concurrently and asynchronously pulls the data source indicator data and extracts the updated indicator value of the data source indicator.

[0034] Furthermore, the indicator data of the monitoring platform is persisted to TSDB, and an index is created for the indicator data of the monitoring platform.

[0035] Furthermore, after saving the filled monitoring platform indicator data to TSDB, the following steps are also included:

[0036] The monitoring platform indicator paradigm also includes a timestamp attribute; correspondingly, the monitoring platform indicator data also includes a timestamp;

[0037] Based on the timestamp attribute value, regularly filter out the monitoring platform indicator data in TSDB that has exceeded the validity period; persist the monitoring platform indicator data that has exceeded the validity period to the file system and clear it accordingly in TSDB.

[0038] Furthermore, according to the corresponding relationship between the monitoring platform indicators and the data source, a data structure of the monitoring platform indicators corresponding to the data source is constructed, which specifically includes:

[0039] Regularly detect whether the data source corresponding to the monitoring platform indicator has changed; if changed, update the data source tag corresponding to the monitoring platform indicator;

[0040] If the detection result is that the structural data of the monitoring platform indicators and data sources do not exist, the structural data of the monitoring platform indicators and data sources are newly created.

[0041] Furthermore, it is necessary to regularly detect whether the data source corresponding to the monitoring platform indicator has changed; if not, continue to pull the monitoring platform indicator from the message queue; if so, update the data structure of the monitoring platform indicator and the data source, and then re-pull the monitoring platform indicator from the message queue.

[0042] On the other hand, a monitoring indicator collection and storage system based on a message queue is provided, and the collection system includes: a memory storing a program; and a processor running the program to execute the method described above.

[0043] Yet another aspect provides a computer-readable storage medium containing a computer program, the computer program being operable to cause a computer to execute the method as described above.

[0044] One or more technical solutions provided by this application have at least the following technical effects or advantages:

[0045] 1. Collect the monitoring platform indicators in the message queue, pull the pre-configured data source indicator data through the mapping relationship, extract the indicator value of the data source indicator data, and fill it into the corresponding monitoring platform indicator to form the monitoring platform indicator data. In this way, in the process of collecting the monitoring platform indicators, the monitoring platform indicator data can be collected and escaped in the process of collecting the monitoring platform indicators, so as to obtain the data available to the monitoring platform with unified format and name.

[0046] 2. Use RabbitMQ message queue middleware for data communication, and push the monitoring platform indicators to the corresponding message queue channel according to the configured Exchange Type; thereby pushing the monitoring platform indicators of the same data source to the same message queue channel to achieve concurrent collection of multiple data sources. Due to the concurrent collection of multiple data sources, that is, the access load balancing of the data source is controlled from the front end, thus avoiding access congestion of a single data source.

[0047] 3. By setting up multiple collection programs, concurrent collection of message queues can be achieved, thereby improving the efficiency of monitoring platform indicator collection.

[0048] 4. By setting the indicator paradigm, the monitoring platform indicators can be freely adapted to the data storage formats of most TSDBs (such as RRD, M3DB, VM).

[0049] 5. When mapping indicators, you can flexibly configure the simple indicators provided by the data source, and you can configure the query statements provided by the data source to achieve aggregate indicators (complex indicators) that are close to monitoring needs.

[0050] 6. In order to ensure that the acquisition program pulls the latest value from the specified data source when pulling data, another possible implementation method is: by flexibly configuring the pulling cycle of different data sources, the most recent value of the mapped data source indicator is obtained, effectively ensuring the timeliness and reliability of the monitoring platform indicators.

[0051] 7. By configuring different data sources, the monitoring platform indicators map the indicators of different data sources, pull the data to the local TSDB, and pull the key indicators from different data sources, so as to achieve the rapid upward convergence of key indicators and important data. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is an overall flow chart of a method for collecting and storing monitoring indicators based on a message queue according to the first embodiment of the present invention.

[0053] Figure 2 It is a flowchart of the process of constructing indicators and pushing them to RabbitMQ in a method for collecting and storing monitoring indicators based on a message queue in the first embodiment of the present invention.

[0054] Figure 3 It is a flowchart of the process from obtaining indicators to collecting data in a method for collecting and storing monitoring indicators based on a message queue in the first embodiment of the present invention.

[0055] Figure 4 It is a flowchart of the process from index storage to index building in a method for collecting and storing monitoring indicators based on a message queue in the first embodiment of the present invention. DETAILED DESCRIPTION

[0056] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0057] Please refer to Figure 1 , Figure 1This is the overall flow chart of the monitoring indicator collection and storage method based on the message queue of the first embodiment. As can be seen from the figure, the application environment of the monitoring data collection system includes a monitoring platform, a RabbitMQ message queue, multiple business platforms and data sources generated by the business platform. The so-called message queue middleware is a communication method between applications in a distributed system, which is mainly used to solve problems such as application decoupling, asynchronous messages, and traffic peak shaving. The two ends of the RabbitMQ message queue are producers and consumers. The producer connects to the RabbitMQ server and sends messages to the queue of the RabbitMQ server, which belongs to the sender of the message; the consumer connects to RabbitMQ to consume messages in the queue and belongs to the receiver of the message. The message exchange (Exchange) is used to receive messages from producers and forward messages to the bound queues according to the routing key. Exchange Type: exchange type; usually includes four types: Fanout, Direct, Topic, Headers. Fanout: message broadcast mode, regardless of the routing key or routing mode, the message will be sent to all queues bound to it. Direct: direct mode, queues with completely matched routing keys are required to receive messages. Topic: Topic matching mode. As long as the routing key meets the wildcard condition, it will be routed to the corresponding queue. Headers: Negotiation mode.

[0058] The monitoring platform pushes the monitoring platform indicators to be collected through the RabbitMQ message queue. The collection program pulls the monitoring platform indicators from the message queue, and accesses the data source corresponding to the indicator to pull the corresponding data source indicator, extracts the data source indicator value and fills it into the monitoring platform indicator, generates the monitoring platform indicator data, and persists it to TSDB (time series database) for the monitoring platform to retrieve and use. The first embodiment of the present application defines the indicator paradigm at the monitoring end. When collecting data source data, only the corresponding indicator value is extracted and filled into the standardized indicator, and then persisted to TSDB, so that the indicator data stored in TSDB is formed based on a uniformly defined indicator paradigm. And it can be achieved in the process of collecting the data to be monitored, the data source indicator data is "escaped", and the monitoring data with a unified format and name that can be directly used by the monitoring platform is obtained.

[0059] Please refer to Figures 1 to 4 The first embodiment of the present invention provides a monitoring indicator collection and storage method based on a message queue. The method uses RabbitMQ message queue middleware (hereinafter referred to as message queue) for data communication. The collection method includes the following steps:

[0060] S1. Pre-configure the data source and sampling period of the monitoring platform.

[0061] S2. Build a monitoring platform indicator paradigm.

[0062] S3. Based on the monitoring scenario requirements, define the monitoring platform indicators to be collected;

[0063] S4. Map the configured data source indicators to the monitoring platform indicators.

[0064] S5. Push the monitoring platform indicators to be collected to the message queue.

[0065] S6. Collect the monitoring platform indicators in the message queue, and pull the corresponding data source indicator data through the mapping relationship.

[0066] S7. Extract the indicator value of the data source indicator data, and fill it into the corresponding monitoring platform indicator to form monitoring platform indicator data.

[0067] S8. Persist the monitoring platform indicator data to TSDB and create an index for the monitoring platform indicator data.

[0068] The following is a further description of each key step, please refer to Figure 1 , S1. Configure the data source of the monitoring platform.

[0069] S1.1 Configure the data source of the monitoring platform:

[0070] According to the corresponding relationship between the monitoring platform indicators and the data source, the data structure of the monitoring platform indicators corresponding to the data source is constructed; the data source tag is set; and the data source tag is written into the structural data of the corresponding monitoring platform indicators. The monitoring platform indicators mentioned here, for example, for monitoring the traffic platform, the indicators can be traffic, bandwidth, packet loss rate, etc.

[0071] One possible implementation method is to construct a data structure of monitoring platform indicators and data sources, and also include periodically detecting whether the data source corresponding to the monitoring platform indicators has changed; if changed, updating the data source tag corresponding to the monitoring platform indicators; if the detection result is that the structural data of the monitoring platform indicators and data sources does not exist, then newly creating the structural data of the monitoring platform indicators and data sources.

[0072] S1.2 Configure the query method of the data source:

[0073] Based on the principle of non-intrusive indicator collection, configure the monitoring platform to collect data from the data source indicator data. The specific method includes: first, determine whether the data source provides a query interface; if so, configure the query interface of the data source to the data source query address of the monitoring platform indicator. For example, most data sources provide query API interfaces or scripts, such as prometheus, opentsdb, etc., which provide query interfaces for all indicators and specified indicators; if not, configure the access address of the data source directly to the query address of the message queue data source. For example, Zabbix, influxDB, etc., in this case, the indicator value of the data source indicator can be obtained by querying the database. The so-called indicator value here, for example, is the traffic value: 123MB / s.

[0074] S1.3 Configure the data source collection cycle:

[0075] Based on the update frequency of the data source indicator values, configure the collection period of the monitoring platform indicators. In addition, it is also necessary to configure the start and stop status of the data source access to control the communication between the data source and the message queue. Specifically, the update frequency of the indicator values ​​of different data sources is detected, and the sampling period of the monitoring platform indicators is synchronously configured according to the update frequency of the indicator values ​​of different data sources. This can avoid the data update frequency and the data collection frequency being different, which may lead to the omission of monitoring data or the loss of server performance due to frequent pulling of indicator data. In addition, a possible implementation method is to configure the start and stop status of the data source access for the collection program, so as to control whether to pull indicator data from the data source. In order to ensure that when the collection program pulls data, it pulls the latest value from the specified data source, another possible implementation method is: by flexibly configuring the pulling period of different data sources, the most recent value of the mapped data source indicator is obtained, effectively ensuring the timeliness and reliability of the monitoring platform indicators.

[0076] S1.4 Configure the Exchange Type of the message queue data source:

[0077] According to the mapping of monitoring platform indicators to data source indicators, build and maintain the corresponding data structure of data source types and monitoring platform indicators. According to the corresponding data structure of data source types and monitoring platform indicators, configure the Exchange Type of the message queue. Subsequently, according to different data source types, the associated monitoring platform indicators will be grouped and pushed to the message queue in staggered time periods according to the Exchange Type to avoid a large number of indicators pushing and blocking the message queue at the same time, and the collection program will not consume and pull in time.

[0078] S2. Construct the indicator paradigm of the monitoring platform:

[0079] Please refer to Figure 4, since multiple business platforms generate their own data sources respectively, the monitoring data standards and formats of each business scenario cannot be standardized and unified on the business platform side, resulting in the indicator data pulled from the data source cannot be directly used on the monitoring platform. Therefore, a possible implementation method is to first build a monitoring platform indicator paradigm to be collected on the monitoring side. The so-called monitoring platform indicator paradigm is a standardized format based on indicator attributes. For example, the monitoring platform indicator paradigm may include attributes such as indicator name, indicator value, indicator identifier, timestamp, and data source tag. Of course, in view of actual monitoring needs, the attributes defined by the indicator paradigm can be adjusted, changed, increased or decreased, and there is no limitation here.

[0080] Please refer to Figure 2 , S4. Map the configured data source indicators to the monitoring platform indicators. The specific steps include:

[0081] S41. Get the full index name of the data source;

[0082] S42. Select a data source indicator name that matches the indicator business of the monitoring platform, and map the corresponding data source indicator to the monitoring platform indicator.

[0083] Specifically, after the monitoring platform defines the monitoring platform indicators according to the monitoring scenario requirements, it queries all the indicator names in the data source through the configured data source query method. The monitoring platform selects the data source indicators that are consistent with the monitoring platform indicator business to be collected for mapping. The same monitoring platform indicator can map indicators of multiple data sources, that is, it can be a one-to-many mapping relationship, thereby overcoming the problem that the same type of monitoring platform indicators have inconsistent names due to being taken from different data sources. In addition, in order to achieve flexible configuration of simple indicators provided by the data source when mapping indicators, the query statement provided by the data source can be configured to achieve aggregated indicators (complex indicators) that are close to monitoring requirements. Therefore, another possible implementation method is that when the monitoring platform indicator maps the data source indicator, a single data source indicator can be specified, or it can be mapped to an aggregated data source indicator. For example: if the data source type is Prometheus, a single indicator of Prometheus can be specified during mapping, or the data source indicator can be aggregated by summing, averaging, etc. through PromQL statements.

[0084] Please refer to Figure 1 , S5. Specific steps for pushing the monitoring platform indicators to be collected to the message queue:

[0085] S51. Traverse the constructed data structure;

[0086] S52. Based on the data structure, the monitoring platform indicators associated with the data source are concurrently pushed to the message queue. The message queue pushes the monitoring platform indicators to the specified message queue channel based on the Exchange Type configured by the data source type for collection by the backend collection program. The monitoring platform indicators are cyclically pushed to the message queue according to the collection cycle of the associated data source, realizing efficient and large-volume data push.

[0087] Please refer to Figure 3 , S6. Collect the monitoring platform indicators in the message queue, and pull the corresponding data source indicator data through the mapping relationship:

[0088] S61 starts multiple acquisition programs;

[0089] Before collecting the monitoring platform indicators in the message queue, you need to start multiple collection programs, set the subscription Exchange Type of the message queue for each collection program, set each collection program to access all channels of the message queue through polling, and enable the subscription mode of the message queue channel.

[0090] S62 Multiple collection programs concurrently collect the channel subscriptions of the message queue and push the monitoring platform indicators to be collected. The specific implementation steps are as follows:

[0091] S621, allocating multiple collection programs to all Exchange Type channels to avoid excessive load on a single collection program, and allowing multiple collection programs to evenly share data collection tasks in the message queue channel;

[0092] S622. Obtain the monitoring platform indicator data to be collected in the message queue channel, traverse the structural data of the monitoring platform indicator, find the data source query method recorded in the structural data, and according to the query method, asynchronously and concurrently pull the corresponding data source indicator data from the data source to achieve efficient pulling of large-scale indicator data.

[0093] S7. The data source indicator data includes indicator values; extract the indicator values ​​of the data source indicator data, and fill them into the corresponding monitoring platform indicators to form monitoring platform indicator data.

[0094] One possible implementation method is to pull the updated indicator value corresponding to the data source indicator through a mapping relationship, specifically including:

[0095] S71. According to the collection cycle, the collection program pulls the monitoring platform indicators to be collected from the channel of the message queue; according to the query method of the data source corresponding to the monitoring platform indicators, the collection program concurrently and asynchronously pulls the data source indicator data and extracts the updated indicator value of the data source indicator.

[0096] S72. Fill the pulled data source indicator data with values ​​according to the indicator definition indicator paradigm of the monitoring platform, such as the indicator name, indicator value, timestamp, and data source tag. The indicator data mentioned here refers to the indicator data containing complete attributes and attribute values, for example: traffic indicator: indicator name: traffic; indicator value: 123MB / s; timestamp: 1653657842; data source tag: traffic monitoring platform. Furthermore, the monitoring platform indicators can be freely adapted to most TSDB (such as RRD, M3DB, vm) storage data formats.

[0097] Please refer to Figure 4 , S8.Persist the monitoring platform indicator data to TSDB and create an index for the monitoring platform indicator data.

[0098] One possible implementation method is to save the filled monitoring platform indicator data to TSDB, and further include: regularly filtering out the monitoring platform indicator data in TSDB that has exceeded the time limit according to the timestamp attribute value; persisting the monitoring platform indicator data that has exceeded the time limit to the file system, and clearing it accordingly in TSDB.

[0099] In addition, building the data structure of monitoring platform indicators and data sources also includes regularly detecting whether the data source corresponding to the monitoring platform indicators has changed; if not, continue to pull the indicators from the message queue; if so, update the data structure of the monitoring platform indicators and data sources, and then pull the indicators from the message queue again.

[0100] One possible implementation is that when the mapping relationship between the data source and the monitoring platform indicator has not changed, steps S5 to S8 are executed cyclically, i.e., the next cycle of data collection is started. If the mapping relationship between the data source and the monitoring platform indicator has changed, the execution starts from step S4, i.e., the mapping relationship between the data source and the monitoring platform indicator is rebuilt, and the collection cycle of the monitoring platform is reconfigured according to the latest configured data source, and the monitoring data is collected according to the updated configuration.

[0101] The first implementation mode of the present application configures different data sources, maps the monitoring platform indicators to the indicators of different data sources, pulls data to the local TSDB, and pulls key indicators from different data sources, thereby realizing the rapid upward convergence of key indicators and important data.

[0102] The specific working process of the first embodiment of the present invention is introduced below.

[0103] Use the collection program to pull data from the Prometheus data source, pull the indicator value of the data source mapped by the monitoring platform indicator through the query API open to Prometheus, extract the specified value to fill the platform indicator, save it to TSDB and create an index. The specific steps include the following:

[0104] Step 1: Configure the Prometheus address and collection cycle, enable Prometheus collection status, and call the http request to determine whether the API address is connected successfully;

[0105] Step 2: Monitor platform indicators (indicator name, indicator identifier, Chinese meaning, data source tag, etc.). For monitoring platform indicators, select the data source (Prometheus), call the Prometheus indicator query interface, obtain the full amount of indicator name data stored in the Prometheus database, and select the indicator provided by Prometheus for mapping;

[0106] Step 3: Before pushing the indicator, first build the data structure of the monitoring platform indicator corresponding to the data source, traverse the data structure, and push the monitoring platform indicator to the message queue concurrently. After the message queue receives the data, it matches the data source type associated with the monitoring platform indicator with the set Exchange Type, and pushes the matching indicator to the specified channel. The indicator is pushed to the message queue cyclically according to the collection cycle configured by the data source;

[0107] Step 4: Define multiple collection programs. The collection programs receive indicators from all channels. The collection programs that subscribe to the channels evenly distribute the indicator data in the channels. The collection programs concurrently obtain the indicators in the message queue distribution channel and call the indicator query API interface provided by Prometheus. According to the fields specified by the monitoring platform indicators, extract the values ​​of the mapping indicators and fill in the monitoring platform indicators. Save the monitoring platform indicators to TSDB, and persist the monitoring indicator data that is more than 3 days old to the file (of course, the time limit setting can be changed according to needs and is not limited to 3 days. It can be configured manually). Clear the data that exceeds the time limit in tsdb and establish the index of the indicator to realize the process of collecting, escaping and storing the monitoring operation and maintenance data based on the message queue.

[0108] Another possible implementation is: a monitoring indicator collection and storage system based on a message queue, the collection system includes multiple servers: each server includes a memory and a processor, the memory stores a program; the processor cooperates to run the above program to execute the method as described above.

[0109] Yet another possible implementation is: a computer-readable storage medium containing a computer program, wherein the computer program is operable to cause a computer to execute the method described above.

[0110] The above description shows and describes the preferred embodiments of the present invention. As mentioned above, it should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention concept of this article through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art do not deviate from the design spirit and scope of the present invention, and should be within the scope of protection of the claims attached to the present invention.

Claims

1. A monitoring indicator collection and storage method based on a message queue, characterized in that: RabbitMQ message queue middleware is used for data communication, and the collection and storage method includes: Based on the principle of non-intrusive indicator collection, configure the monitoring platform to collect data from the data source indicator data, including: determine whether the data source provides a query interface; if so, configure the query interface of the data source to the data source query address of the monitoring platform indicator; if not, directly configure the access address of the data source to the query address of the message queue data source; Constructing a monitoring platform indicator paradigm, wherein the monitoring platform indicator paradigm at least includes attributes: indicator name, indicator value; Based on the monitoring scenario requirements, define the monitoring platform indicators to be collected; Mapping the configured data source indicator data to the monitoring platform indicator, specifically including: obtaining the full indicator name of the data source; selecting the full indicator name of the data source that is consistent with the monitoring platform indicator business, and mapping the corresponding data source indicator data to the monitoring platform indicator; the monitoring platform pushes the monitoring platform indicator to be collected to the message queue; Collect the monitoring platform indicators in the message queue, and pull the pre-configured data source indicator data through the mapping relationship, wherein the data source indicator data includes the indicator value; and pull the updated indicator value corresponding to the data source indicator through the mapping relationship; The indicator value of the data source indicator data is extracted and filled into the corresponding monitoring platform indicator to form monitoring platform indicator data.

2. The monitoring indicator collection and storage method based on the message queue according to claim 1 is characterized in that: Before pushing the monitoring platform indicators to be collected to the message queue, the method further includes: Configure the data source of the monitoring platform; the data source is generated by the business platform.

3. The monitoring indicator collection and storage method based on the message queue according to claim 1 is characterized in that: Also includes: Based on the data source indicator value update frequency, configure the collection period of the monitoring platform indicator.

4. The monitoring indicator collection and storage method based on the message queue according to claim 3 is characterized in that: The data collection method of configuring the data source specifically includes: Determine whether the data source provides a query interface; if so, configure the query interface of the data source to the query address of the message queue data source; if not, directly configure the access address of the data source to the query address of the message queue data source.

5. The monitoring indicator collection and storage method based on message queue according to claim 3 is characterized in that: The step of configuring the collection period of the monitoring platform indicator based on the data source indicator value update frequency further includes: The start and stop states of the data source access are configured to control the on and off communication between the data source and the message queue.

6. The monitoring indicator collection and storage method based on the message queue according to claim 1 is characterized in that: Before the monitoring platform indicators in the acquisition message queue are collected, it also includes: Start multiple collection programs, and set a subscription Exchange Type of the message queue for each collection program; Each acquisition program is set to access all channels of the message queue in a polling manner.

7. The monitoring indicator collection and storage method based on the message queue according to claim 6 is characterized in that: The subscription mode of the channel of the message queue is enabled, and the multiple collection programs concurrently collect the channel subscription of the message queue and push the monitoring platform indicators to be collected.

8. The monitoring indicator collection and storage method based on the message queue according to claim 1 is characterized in that: Also includes: Mapping the configured data source to the monitoring platform indicator; According to the correspondence between the monitoring platform indicators and the data source types, the ExchangeType of the message queue is configured.

9. The monitoring indicator collection and storage method based on message queue according to claim 8 is characterized in that: The mapping of the configured data source to the monitoring platform indicator specifically includes: According to the corresponding relationship between the monitoring platform indicator and the data source, construct a data structure corresponding to the data source of the monitoring platform indicator; Set the data source Tag; write the data source Tag into the structure data of the corresponding monitoring platform indicator.

10. The monitoring indicator collection and storage method based on message queue according to claim 9 is characterized in that: The pushing of the monitoring platform indicators to be collected to the message queue specifically includes: Traversing the constructed data structure; Based on the data structure, the monitoring platform indicators associated with the data source are concurrently pushed to the message queue, and the message queue pushes the monitoring platform indicators to the specified message queue channel based on the Exchange Type configured by the data source type.

11. The monitoring indicator collection and storage method based on message queue according to claim 1 is characterized in that: Pulling the updated indicator value corresponding to the data source indicator through the mapping relationship specifically includes: According to the collection cycle, the collection program pulls the monitoring platform indicators to be collected from the channel of the message queue; According to the query method of the data source corresponding to the monitoring platform indicator, the acquisition program concurrently and asynchronously pulls the data source indicator data and extracts the updated indicator value of the data source indicator.

12. The monitoring indicator collection and storage method based on message queue according to claim 1 is characterized in that: Also includes: The monitoring platform indicator data is persisted to TSDB, and an index is created for the monitoring platform indicator data.

13. The monitoring indicator collection and storage method based on message queue according to claim 12 is characterized in that: After persisting the filled monitoring platform indicator data to TSDB, the following steps are also included: The monitoring platform indicator paradigm also includes a timestamp attribute; correspondingly, the monitoring platform indicator data also includes a timestamp; According to the timestamp attribute value, regularly filter out the monitoring platform indicator data in the TSDB that exceeds the time limit; The monitoring platform indicator data that has exceeded the validity period is persisted to the file system and cleared accordingly in the TSDB.

14. The monitoring indicator collection and storage method based on message queue according to claim 9 is characterized in that: According to the correspondence between the monitoring platform indicators and the data source, constructing a data structure corresponding to the monitoring platform indicators and the data source specifically includes: Regularly detect whether the data source corresponding to the monitoring platform indicator has changed; if changed, update the data source Tag corresponding to the monitoring platform indicator; If the detection result is that the structural data of the monitoring platform indicators and the data source does not exist, the structural data of the monitoring platform indicators and the data source are newly created.

15. The monitoring indicator collection and storage method based on message queue according to claim 14 is characterized in that: The periodic detection of whether the data source corresponding to the monitoring platform indicator has changed also includes: If no change occurs, continue to pull the monitoring platform indicator from the message queue; if so, update the data structure of the monitoring platform indicator and the data source, and then re-pull the monitoring platform indicator from the message queue.

16. A monitoring indicator collection and storage system based on a message queue, characterized in that: The acquisition and storage system comprises: A memory storing programs; A processor runs the program to execute the method according to any one of claims 1 to 15.

17. A computer-readable storage medium containing a computer program, wherein the computer program is operable to cause a computer to execute the method according to any one of claims 1 to 15.

Images