Index acquisition system

By working together with the distributed coordination module and the monitoring agent module, the indicator collection method is automatically adjusted, which solves the cumbersome and error-prone problems caused by manual script distribution in the existing technology and realizes a simplified monitoring indicator collection process.

CN118916238APending Publication Date: 2024-11-08CHINA TELECOM CORP LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411087946.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-08
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Current technologies require manual distribution and uploading of scripts to collect monitoring indicators, making the adjustment process cumbersome and prone to errors.

Method used

A distributed coordination module is used to store the indicator collection method scripts and configuration files, while a monitoring agent module listens to and adjusts the indicator collection process to achieve automated configuration.

Benefits of technology

The elimination of the need for manual script distribution simplifies the process of adjusting the monitoring indicator collection method and reduces the error rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916238A_ABST
    Figure CN118916238A_ABST
Patent Text Reader

Abstract

The invention discloses an index acquisition system. The system comprises a monitoring server, a distributed coordination module, a plurality of monitoring agent modules and a plurality of index acquisition processes, the distributed coordination module is used for receiving a configuration instruction sent by the monitoring server and configuring a first type of nodes and a second type of nodes in the distributed coordination module according to the configuration instruction, the first type of nodes are used for storing index acquisition mode scripts and index acquisition mode configuration files, and the second type of nodes are used for storing monitoring index information; the target monitoring agent module is used for monitoring a second type of nodes; and determining a target first type node according to the monitoring result, and adjusting an index collection mode of an index collection process connected with the target monitoring agent module according to the target first type node. According to the method and the device, the technical problems of tedious adjustment process and error proneness caused by the fact that a monitoring index acquisition mode is configured by adopting a mode of manually distributing and uploading scripts in related technologies are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of data collection, and in particular, to an indicator collection system. Background Art

[0002] In the related art, when collecting monitoring indicators for various data, in order to modify the monitoring indicators, the method usually adopted is to set the modified monitoring indicators through custom scripts. The problem with this method is that the user needs to upload the custom script to the corresponding host (the device whose monitoring indicators are collected). For example, uploading the indicator collection script related to the redis middleware to the host of the redis cluster requires manual pre-classification of resources and distribution of scripts. As a result, the adjustment process in the related art when the monitoring indicators need to be adjusted is more cumbersome, and it is easy to make mistakes caused by human negligence.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiment of the present application provides an indicator collection system to at least solve the technical problem that the adjustment process is cumbersome and error-prone due to the use of manual distribution and uploading of scripts to configure the monitoring indicator collection method in the related technology.

[0005] According to one aspect of an embodiment of the present application, an indicator collection system is provided, including a monitoring server, a distributed coordination module, multiple monitoring agent modules, and multiple indicator collection processes, wherein the distributed coordination module is used to receive configuration instructions sent by the monitoring server, and configure the first type of nodes and the second type of nodes in the distributed coordination module according to the configuration instructions, wherein the first type of nodes are used to store indicator collection method scripts and indicator collection method configuration files, and the second type of nodes are used to store monitoring indicator information corresponding to various types of data resources, and the monitoring indicator information includes monitoring indicator codes and resource type codes corresponding to various types of data resources determined by the configuration management database; the target monitoring agent module is used to monitor the second type of nodes in the distributed coordination module, wherein the target monitoring agent module is multiple Any monitoring agent module in the monitoring agent module; determine the target first-class node in the distributed coordination module based on the monitoring result, and determine the target indicator collection method script and the target indicator collection method configuration file based on the target first-class node to adjust the indicator collection method of the indicator collection process connected to the target monitoring agent module, wherein the indicator collection process is set in the target device and is used to collect monitoring indicators in the target device, and the indicator collection method includes the monitoring indicator coding of the monitoring indicators collected by the indicator collection process; the distributed coordination module is also used to monitor the health status of multiple monitoring agent modules, and determine the main monitoring agent module among the multiple monitoring agent modules based on the health status, wherein the main monitoring agent module is used to control the working status of all indicator collection processes in the multiple indicator collection processes.

[0006] Optionally, the first type of node includes a script node for storing an indicator collection method script, and a configuration file node for storing an indicator collection method configuration file; the preset values ​​corresponding to the second type of node include a first preset value and a second preset value, the first preset value is used to indicate that the monitoring indicator corresponding to the second type of node is a newly added monitoring indicator within a preset time period, and the second preset value is used to indicate that the monitoring indicator corresponding to the second type of node is an updated monitoring indicator.

[0007] Optionally, the configuration instructions include custom script configuration instructions and plug-in configuration instructions; configuring the first type of nodes and the second type of nodes in the distributed coordination module according to the configuration instructions includes: when the configuration instructions are custom script configuration instructions, the distributed coordination module generates an indicator collection script and an indicator collection method configuration file corresponding to the custom script configuration instructions according to the custom script configuration instructions, and stores the indicator collection script in the script node, and stores the indicator collection method configuration file in the configuration file node; when the configuration instructions are plug-in definition configuration instructions, the distributed coordination module generates a custom plug-in corresponding to the plug-in definition configuration instructions and a plug-in configuration file corresponding to the custom plug-in according to the plug-in definition configuration instructions, and stores the plug-in configuration file in the configuration file node.

[0008] Optionally, the script node also stores the monitoring indicator code and resource type code corresponding to the indicator collection method script, and the configuration file node also stores the monitoring indicator code corresponding to the indicator collection method configuration file, the unique identification code of the resource instance indicated by the resource type code and the monitoring indicator code, and the IP address of the resource instance. The unique identification code is determined by the configuration management database.

[0009] Optionally, the target monitoring agent module is also used to: after the target monitoring agent module monitors a newly added second-type node or a modified second-type node, obtain the resource type code, monitoring indicator code, unique identification code and IP address corresponding to the newly added or modified second-type node, and when it is determined that the target device corresponding to the IP address is the device corresponding to the target monitoring agent module, determine the script node and configuration file node corresponding to the resource type code, monitoring indicator code, unique identification code and IP address; obtain the indicator collection method script from the corresponding script node, and obtain the indicator collection method configuration file from the corresponding configuration file node.

[0010] Optionally, the target monitoring agent module is also used to: after the target monitoring agent module monitors the deleted second-type node, obtain the resource type code, monitoring indicator code, unique identification code and IP address corresponding to the deleted second-type node, and when it is determined that the target device corresponding to the IP address is the device corresponding to the target monitoring agent module, delete the indicator collection method script and indicator collection method configuration file in the indicator collection process connected to the target monitoring agent module, wherein the indicator collection method script in the indicator collection process is the indicator collection method script corresponding to the resource type code, monitoring indicator code and unique identification code, and the indicator collection method configuration file in the indicator collection process is the indicator collection method configuration file corresponding to the resource type code, monitoring indicator code and unique identification code.

[0011] Optionally, the distributed coordination module is further used to: determine the target script node in which the stored indicator collection method script is modified, and after the indicator collection method script in the target script node is modified, determine the unique identification code and IP address corresponding to the monitoring indicator code in the configuration management database according to the monitoring indicator code in the target script node, and set the preset value of the second type of node corresponding to the monitoring indicator code to the second preset value;

[0012] The target configuration file node corresponding to the script node is determined based on the unique identification code and IP address corresponding to the monitoring indicator code in the target script node, and the preset value of the second type of node corresponding to the monitoring indicator code is set as the second preset value.

[0013] Optionally, monitoring the health status of multiple monitoring agent modules includes: determining a failover controller corresponding to each monitoring agent module among the multiple monitoring agent modules, wherein there is a one-to-one correspondence between the monitoring agent modules and the failover controllers, and a session is maintained between each pair of monitoring agent modules and the failover controllers; and periodically sending health monitoring instructions to each monitoring agent module through the failover controller to obtain the health status of each monitoring agent module.

[0014] Optionally, a temporary node corresponding to the main monitoring agent node is set in the distributed coordination module, and the temporary node is used as a lock for the main monitoring agent node selection process; determining the main monitoring agent module among multiple monitoring agent modules based on the health status includes: after the working state of the main monitoring agent module is abnormal, deleting the temporary node, triggering the lock grabbing process of multiple monitoring agent modules; determining the monitoring agent module that grabs the lock among multiple monitoring agent modules as the new main monitoring agent module.

[0015] Optionally, the target monitoring agent module is also used to: continuously obtain heartbeat packets sent by the target indicator collection process connected to the target monitoring agent module, and if no heartbeat packets sent by the target indicator collection process are received within a preset time period, detect the communication network between the indicator collection process and the monitoring indicator collection process; if it is detected that the communication network is disconnected, the target indicator collection process backs up the collected monitoring indicators; if it is detected that the communication network is normal and the target indicator collection process is not detected, restart the target indicator collection process.

[0016] Optionally, after collecting the monitoring indicators, the indicator collection process sends the monitoring indicators to the data lake and stores the monitoring indicators in a local backup on the target device.

[0017] In an embodiment of the present application, an indicator collection system is provided, including a monitoring server, a distributed coordination module, multiple monitoring agent modules, and multiple indicator collection processes, wherein the distributed coordination module is used to receive configuration instructions sent by the monitoring server, and configure the first type of nodes and the second type of nodes in the distributed coordination module according to the configuration instructions, wherein the first type of nodes are used to store indicator collection method scripts and indicator collection method configuration files, and the second type of nodes are used to store monitoring indicator information corresponding to various types of data resources, and the monitoring indicator information includes monitoring indicator codes and resource type codes corresponding to various types of data resources determined by the configuration management database; the target monitoring agent module is used to monitor the second type of nodes in the distributed coordination module, wherein the target monitoring agent module is a plurality of monitoring agent modules any one of the monitoring agent modules in; determining the target first-class node in the distributed coordination module according to the monitoring result, and determining the target indicator collection method script and configuration file node target indicator collection method configuration file according to the target first-class node to adjust the indicator collection method of the indicator collection process connected to the target monitoring agent module, wherein the indicator collection process is set in the target device and is used to collect monitoring indicators in the target device, and the indicator collection method includes the monitoring indicator coding of the monitoring indicators collected by the indicator collection process; the distributed coordination module is also used to monitor the health status of multiple monitoring agent modules, and determine the main monitoring agent module in the multiple monitoring agent modules according to the health status, wherein the main monitoring agent module is used to control the working status of all indicator collection processes in the multiple indicator collection processes. In the indicator collection system provided in the embodiment of the present application, by allowing the first type of nodes in the distributed coordination module to store the indicator collection method script and the indicator collection method configuration file, and allowing the second type of nodes in the distributed coordination module to store the monitoring indicator information corresponding to each type of data resources, and allowing each monitoring agent module to listen to the second type of nodes, the target first type of node is determined according to the listening result to determine the target indicator collection method script and the target indicator collection method configuration file to adjust the indicator collection method of the indicator collection process connected to the target monitoring agent module, thereby achieving the purpose of the monitoring agent module automatically adjusting the indicator collection method in each indicator collection process, thereby achieving the technical effect of not having to manually distribute and upload configuration files and scripts to each target device, thereby solving the technical problem that the adjustment process is cumbersome and error-prone due to the use of manual distribution and uploading of scripts to configure the monitoring indicator collection method in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1It is a structural diagram of an indicator collection system provided according to an embodiment of the present application;

[0020] Figure 2 It is a flow chart of a monitoring indicator change process in an indicator collection system provided according to an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of the determination process of the main indicator collection process in the indicator collection system provided in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0025] In order to better understand the embodiments of the present application, the technical terms involved in the embodiments of the present application are explained as follows:

[0026] CMDB (Configuration Management Database): is a logical database that contains information about the entire life cycle of configuration items and the relationships between configuration items (including physical relationships, real-time communication relationships, non-real-time communication relationships, and dependency relationships).

[0027] Telegraf: is an open source plug-in-based indicator collection tool. It is mainly used in conjunction with a time-series database for performance monitoring. Usually Telegraf captures a batch of indicator data at regular intervals and sends the data to a time-series database or other custom Output.

[0028] Monitoring is an important part of daily operation and maintenance work and the entire life cycle of products. It aims to timely warn and discover faults in advance, locate problems in conjunction with monitoring data during the process, and provide data for problem analysis afterwards. In particular, indicator monitoring runs through the entire life cycle of a fault and is the inherent nature of the fault appearance and the essence of the alarm: if there is a fault, there must be abnormal indicators, and if there is no fault, the indicator value is normal.

[0029] Therefore, in order to more accurately determine the status of each host (device to be monitored), it is necessary to cover as many monitoring indicators as possible and collect real-time data on the monitoring indicators during system operation.

[0030] There are many methods for collecting monitoring indicators in the related art, but these methods all have certain limitations, resulting in the inability to fully cover all types of monitoring indicators. For example, the related art provides an indicator collection method in which the knowledge module area (KM) in the system pre-defines the collectible monitoring indicators, and the user selects the required monitoring indicators from the KM when setting the monitoring indicators. The problem with this method is that the scope of indicator collection is limited, and only the monitoring indicators defined in the KM can be collected. It cannot support the personalized needs of users, and it cannot be well adapted to some systems or middleware.

[0031] In order to achieve customized changes in monitoring indicators, a method for developing customized monitoring indicators is also provided in the related art, which specifically includes determining the monitoring indicator information to be developed, importing the monitoring indicator information into the corresponding field according to the field information in the monitoring indicator template, and developing the collection script using the new monitoring indicator template and collection script development specification, thereby achieving customized development of monitoring indicators and improving the flexibility of monitoring indicators. The problem with this method is that the script needs to be uploaded to the corresponding host, for example, uploading the indicator collection script related to the redis middleware to the host of the redis cluster, and it is necessary to manually classify the resources in advance and distribute the scripts.

[0032] In order to solve the above problems, relevant solutions are provided in the embodiments of the present application, which are described in detail below.

[0033] According to an embodiment of the present application, a system embodiment of an indicator collection system is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0034] The indicator collection system provided in the embodiment of the present application is as follows: Figure 1 As shown, it includes a monitoring server 10, a distributed coordination module 20, multiple monitoring agent modules 30, and multiple indicator collection processes 40, wherein the distributed coordination module 20 is used to receive the configuration instructions sent by the monitoring server 10, and configure the first type of nodes and the second type of nodes in the distributed coordination module 20 according to the configuration instructions, wherein the first type of nodes are used to store indicator collection method scripts and indicator collection method configuration files, and the second type of nodes are used to store monitoring indicator information corresponding to various types of data resources, and the monitoring indicator information includes the monitoring indicator codes and resource type codes corresponding to various types of data resources determined by the configuration management database; the target monitoring agent module 30 is used to monitor the second type of nodes in the distributed coordination module 20, wherein the target monitoring agent module 30 is any one of the multiple monitoring agent modules 30. Agent module 30; determines the target first-class node in the distributed coordination module 20 according to the monitoring result, and determines the target indicator collection method script and the target indicator collection method configuration file according to the target first-class node to adjust the indicator collection method of the indicator collection process 40 connected to the target monitoring agent module 30, wherein the indicator collection process 40 is set in the target device and is used to collect monitoring indicators in the target device, and the indicator collection method includes the monitoring indicator coding of the monitoring indicators collected by the indicator collection process 40; the distributed coordination module 20 is also used to monitor the health status of multiple monitoring agent modules 30, and determines the main monitoring agent module 30 among the multiple monitoring agent modules 30 according to the health status, wherein the main monitoring agent module 30 is used to control the working status of all indicator collection processes 40 in the multiple indicator collection processes 40.

[0035] It should be noted that the indicator collection process 40 (telegarf) corresponding to each monitoring agent module 30 (agent) can be set by the user. Figure 1 In the manner described above, each monitoring agent module 30 corresponds to all indicator collection processes 40, so that when a monitoring agent module 30 fails, it will not affect the collection of monitoring indicators and the configuration and status confirmation of the indicator collection process 40.

[0036] In addition, after collecting the monitoring indicators, the indicator collection process 40 will send the monitoring indicators to the data lake and store the monitoring indicators in the local backup of the target device. Figure 1As shown in the figure, the collected monitoring indicators are stored in the data lake through the message middleware Kafka.

[0037] In some embodiments of the present application, the first type of node includes a script node for storing an indicator collection method script, and a configuration file node for storing an indicator collection method configuration file. The preset value corresponding to the second type of node includes a first preset value and a second preset value, the first preset value is used to indicate that the monitoring indicator corresponding to the second type of node is a newly added monitoring indicator within a preset time period, and the second preset value is used to indicate that the monitoring indicator corresponding to the second type of node is an updated monitoring indicator. The first preset value and the second preset value can be set by yourself, for example, the first preset value can be 1, and the second preset value can be 2.

[0038] As an optional implementation, the configuration instructions include custom script configuration instructions and plug-in configuration instructions; configuring the first type of nodes and the second type of nodes in the distributed coordination module 20 according to the configuration instructions includes: when the configuration instruction is a custom script configuration instruction, the distributed coordination module 20 generates an indicator collection script and an indicator collection method configuration file corresponding to the custom script configuration instruction according to the custom script configuration instruction, and stores the indicator collection script in the script node, and stores the indicator collection method configuration file in the configuration file node; when the configuration instruction is a plug-in definition configuration instruction, the distributed coordination module 20 generates a custom plug-in corresponding to the plug-in definition configuration instruction and a plug-in configuration file corresponding to the custom plug-in according to the plug-in definition configuration instruction, and stores the plug-in configuration file in the configuration file node.

[0039] Specifically, Figure 2 As shown, the monitoring server 10 is mainly used to configure custom indicators according to the relevant operations of the user, and the configuration methods include custom scripts and custom plug-ins. Taking the new indicator as an example, when configuring the custom indicator, the user needs to select the resource type to which the new indicator belongs. After that, the monitoring server 10 will save the new indicator and the corresponding resource type to the configuration management database, and the configuration management database will assign the corresponding indicator code to the new indicator.

[0040] In addition from Figure 2 As can be seen, the configurations that the monitoring server 10 can perform include adding script indicators, adding resource instances, modifying script content, etc. in the distributed coordination module 20. When the relevant data stored in the distributed coordination module 20 changes, the monitoring agent module 30 that monitors the distributed coordination module 20 will issue the corresponding configuration file to the connected indicator collection process 40 according to the monitoring result.

[0041] As an optional implementation, when a custom script is used to define the indicator collection method, the monitoring server 10 generates a Telegraf configuration file according to the user's relevant operations, where the Telegraf configuration file can be written in the Telegraf standard format, the input uses the exec mode to execute the custom script, and the output is defined in json format. Then, the script content and the configuration file are stored in the corresponding node in the distributed coordination module 20 (zookeeper). The storage path and stored content of each node are as follows:

[0042] [Script node]: / [resource type code] / [indicator code] / script, stores script content

[0043] [Configuration file node]: / [Resource type code] / [Indicator code] / conf, storage configuration file

[0044] When using the plug-in to define the collection method of indicators, you can use Telegraf native plug-ins or custom Input plug-ins. Telegraf is written in Go language and has good scalability. When writing a custom Input plug-in, you need to implement the telegraf.Input interface, call the inputs.Add method in the init function, and register it under plugins / inputs / all. After registration, it can be used like a native plug-in. The system can generate a configuration file based on the selected plug-in type and store it in the distributed coordination module 20.

[0045] It should be noted that when the plug-in is used to define the indicator, the script file corresponding to the indicator may not be modified. At this time, the storage path of the configuration file in the distributed coordination module 20 is as follows:

[0046] [Configuration file node]: / [Resource type code] / [Indicator code] / conf, which stores configuration files.

[0047] Specifically, the IP, instanceName and the indicator code bound to all resource instances under the above resource type can be obtained by querying the data resources in the configuration management database, and then stored in the distributed coordination module 20. The IP corresponding to the resource instance includes the IP addresses of all target devices with the resource instance. In addition, the storage path of the relevant information of the changed monitoring indicators corresponding to the collection method of the plug-in defined indicators in the distributed coordination module 20 is as follows:

[0048] [Second type node]: / metric / [IP] / [resource type code] / [instanceName] / [metric code]

[0049] In some embodiments of the present application, the script node also stores the monitoring indicator code and resource type code corresponding to the indicator collection method script, and the configuration file node also stores the monitoring indicator code corresponding to the indicator collection method configuration file, the unique identification code of the resource instance indicated by the resource type code and the monitoring indicator code, and the IP address of the resource instance, and the unique identification code is determined by the configuration management database. Specifically, when the preset value of the second type of node is 1, it indicates that the resource corresponding to the node has added a new collection indicator and started to collect the indicator, where instanceName represents the unique identification of the resource instance.

[0050] As an optional implementation, the distributed coordination module 20 is also used to: determine the target script node in which the stored indicator collection method script is modified, and after the indicator collection method script in the target script node is modified, determine the unique identification code and IP address corresponding to the monitoring indicator code in the target script node in the configuration management database; determine the target configuration file node corresponding to the script node based on the unique identification code and IP address corresponding to the monitoring indicator code in the target script node, and set the preset value corresponding to the target configuration file node to the second preset value.

[0051] Specifically, when the user modifies the script content in the defined indicator, the content in the script node can be modified, and the resource type bound to the collection indicator can be queried through the indicator code, and the instanceName and IP of each resource instance under this type can be queried through the configuration management database interface. Then, the preset value of the corresponding second-type node is modified to 2, indicating that the collection indicator is updated for the resource.

[0052] In some embodiments of the present application, when a new monitoring resource is added, the resource information of the new monitoring resource will be synchronized to the configuration management database, and the configuration management database will determine the resource type of the new monitoring resource and the indicator code bound to the resource type, and then generate a corresponding node in the distributed coordination module 20 to store this information.

[0053] When a resource instance is removed, the resource information of the resource instance will also be removed from the configuration management database. The monitoring server 10 will automatically obtain the IP, instanceName, resource type and indicator code bound to the resource instance, and then delete the node corresponding to the resource instance in the distributed coordination module 20.

[0054] In some embodiments of the present application, monitoring the health status of multiple monitoring agent modules 30 includes: determining a failover controller corresponding to each monitoring agent module 30 in the multiple monitoring agent modules 30, wherein there is a one-to-one correspondence between the monitoring agent module 30 and the failover controller, and a session is maintained between each pair of the monitoring agent module 30 and the failover controller; and periodically sending health monitoring instructions to each monitoring agent module 30 through the failover controller, thereby obtaining the health status of each monitoring agent module 30.

[0055] As an optional implementation, a temporary node corresponding to the main monitoring agent node is provided in the distributed coordination module 20, and the temporary node is used as a lock for the main monitoring agent node selection process; determining the main monitoring agent module 30 among multiple monitoring agent modules 30 based on the health status includes: after the working state of the main monitoring agent module 30 is an abnormal state, deleting the temporary node, triggering the lock grabbing process of multiple monitoring agent modules 30; determining the monitoring agent module 30 that grabs the lock among multiple monitoring agent modules 30 as the new main monitoring agent module 30.

[0056] It should be noted that, in addition to storing the above-mentioned scripts, configuration files, monitoring indicators and resource instance information, the distributed coordination service module in the embodiment of the present application is also used to elect and switch the main monitoring agent module 30 (agent leader).

[0057] As an optional implementation, in order to solve the problem of single point failure of the monitoring agent, the monitoring agent module 30 in the present application adopts a distributed deployment mode. And one monitoring agent module 30 among the multiple monitoring agent modules 30 will be selected as the main monitoring agent module 30, which is responsible for managing the control of the status of the indicator collection process 40 in the entire cluster.

[0058] In some embodiments of the present application, Figure 3 As shown, each monitoring agent module 30 corresponds to a ZKFC (ZKFailOverController) in the distributed coordination service module, and there is a one-to-one correspondence between the monitoring agent module 30 and the ZKFC. Among them, ZKFC is responsible for monitoring the status of the agent and writing the information into ZK. ZKFC can periodically obtain the status of the monitoring agent module 30 through an independent thread, and is the core of realizing the failover function. Among them, the status of the monitoring agent module 30 includes normal and abnormal, and the failover function refers to the timely selection of a new main monitoring agent module 30 after determining that the main monitoring agent module 30 is abnormal.

[0059] As an optional implementation, in addition to health monitoring, the main function of ZKFC can also be responsible for the selection of the main monitoring agent module 30. Specifically, when each monitoring agent module 30 is in a healthy state, ZKFC will maintain a session in the distributed coordination service module; if the monitoring agent module 30 is the main monitoring agent module 30, ZKFC will occupy a temporary node znode in the distributed coordination service module as a lock. When the status of the monitoring agent module 30 becomes abnormal, the znode will be deleted, and the election of the main monitoring agent module 30 will be triggered. Each monitoring agent module 30 starts to grab the lock, and the monitoring agent module 30 that successfully grabs the lock is upgraded to the main monitoring agent module 30.

[0060] In some embodiments of the present application, the target monitoring agent module 30 is also used to: continuously obtain heartbeat packets sent by the target indicator collection process 40 connected to the target monitoring agent module 30, and when the heartbeat packet sent by the target indicator collection process 40 is not received within a preset time period, detect the communication network between the indicator collection process 40 and the monitoring indicator collection process 40; when it is detected that the communication network is disconnected, the target indicator collection process 40 backs up the collected monitoring indicators; when it is detected that the communication network is normal and the target indicator collection process 40 is not detected, restart the target indicator collection process 40.

[0061] Specifically, the monitoring agent module 30 is provided with a registration center and an indicator monitoring center. Among them, the registration center is used to monitor the heartbeat packet of the indicator collection process 40 received by the monitoring agent, and reply ACK to give a response. The relevant information of the indicator collection process 40 received by the monitoring agent module 30 can be saved in a key-value pair, and the time of the last heartbeat packet received is updated when the heartbeat packet is received. When the heartbeat packet sent by a certain indicator collection process 40 is not received for a period of time, it is considered that the target device where the process is located is offline, which may be caused by the disconnection of the network connection or the non-existence of the indicator collection process 40. If the agent detects that the indicator collection process 40 does not exist, it issues an instruction to start the indicator collection process 40 of the corresponding target device. If the heartbeat packet cannot be received due to the disconnection of the network connection, it is necessary to back up and process the collected data, which is implemented by the indicator collection tool (that is, the indicator collection process 40). When the process or network connection is restored, the time of the last heartbeat packet received is reset. The heartbeat packet information may include the heartbeat packet ID, the heartbeat packet sending time, the instance name of the indicator collection process 40, the IP, etc.

[0062] The indicator monitoring center is used to monitor the second type of nodes, and perform corresponding operations on the indicator collection process 40 according to the changes of the second type of nodes. For example, after the target monitoring agent module 30 monitors the newly added or modified second type of nodes, the resource type code, monitoring indicator code, unique identification code and IP address corresponding to the newly added or modified second type of nodes are obtained, and when it is determined that the target device corresponding to the IP address is the device corresponding to the target monitoring agent module 30, the script node and configuration file node corresponding to the resource type code, monitoring indicator code, unique identification code and IP address are determined; the indicator collection method script is obtained from the corresponding script node, and the indicator collection method configuration file is obtained from the corresponding configuration file node.

[0063] After monitoring the deleted second-type node, obtain the resource type code, monitoring indicator code, unique identification code and IP address corresponding to the deleted second-type node, and when it is determined that the target device corresponding to the IP address is the device corresponding to the target monitoring agent module 30, delete the indicator collection method script and indicator collection method configuration file in the indicator collection process 40 connected to the target monitoring agent module 30, wherein the indicator collection method script in the indicator collection process 40 is the indicator collection method script corresponding to the resource type code, monitoring indicator code and unique identification code, and the indicator collection method configuration file in the indicator collection process 40 is the indicator collection method configuration file corresponding to the resource type code, monitoring indicator code and unique identification code.

[0064] Specifically, when monitoring the addition or modification of the second-type node, the monitoring agent module 30 can confirm whether the indicator collection process 40 in the target device corresponding to the IP of the newly added or modified second-type node is responsible for itself, and after confirmation, it determines the script node and configuration file node corresponding to the node according to the changed second-type node to update the indicator collection method in the indicator collection process 40. When monitoring the deletion of the second-type node, after confirming that action is required according to the IP, delete the script and configuration file in the indicator collection process 40 related to the IP, and end the collection of the indicator.

[0065] In summary, an embodiment of the present application provides an indicator collection system, including a monitoring server, a distributed coordination module, multiple monitoring agent modules, and multiple indicator collection processes, wherein the distributed coordination module is used to receive configuration instructions sent by the monitoring server, and configure the first type of nodes and the second type of nodes in the distributed coordination module according to the configuration instructions, wherein the first type of nodes are used to store indicator collection method scripts and indicator collection method configuration files, and the second type of nodes are used to store monitoring indicator information corresponding to various types of data resources, and the monitoring indicator information includes monitoring indicator codes and resource type codes corresponding to various types of data resources determined by the configuration management database; the target monitoring agent module is used to monitor the second type of nodes in the distributed coordination module, wherein the target monitoring agent module is multiple monitoring agents. any monitoring agent module in the module; determine the target first-class node in the distributed coordination module based on the monitoring result, and determine the target indicator collection method script and configuration file node target indicator collection method configuration file based on the target first-class node to adjust the indicator collection method of the indicator collection process connected to the target monitoring agent module, wherein the indicator collection process is set in the target device and is used to collect monitoring indicators in the target device, and the indicator collection method includes the monitoring indicator coding of the monitoring indicators collected by the indicator collection process; the distributed coordination module is also used to monitor the health status of multiple monitoring agent modules, and determine the main monitoring agent module in multiple monitoring agent modules based on the health status, wherein the main monitoring agent module is used to control the working status of all indicator collection processes in multiple indicator collection processes. In the indicator collection system provided in the embodiment of the present application, by allowing the first type of nodes in the distributed coordination module to store the indicator collection method script and the indicator collection method configuration file, and allowing the second type of nodes in the distributed coordination module to store the monitoring indicator information corresponding to each type of data resources, and allowing each monitoring agent module to listen to the second type of nodes, the target first type of node is determined according to the listening result to determine the target indicator collection method script and the target indicator collection method configuration file to adjust the indicator collection method of the indicator collection process connected to the target monitoring agent module, thereby achieving the purpose of the monitoring agent module automatically adjusting the indicator collection method in each indicator collection process, thereby achieving the technical effect of not having to manually distribute and upload configuration files and scripts to each target device, thereby solving the technical problem that the adjustment process is cumbersome and error-prone due to the use of manual distribution and uploading of scripts to configure the monitoring indicator collection method in the related technology.

[0066] In some embodiments of the present application, the structure of the target device is as follows: Figure 4 As shown, it can be a computer terminal or a mobile device. Figure 4As shown, the computer terminal 40 (or mobile device 40) may include one or more (402a, 402b, ..., 402n are used to illustrate) processors 402 (the processor 402 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 404 for storing data, and a transmission module 406 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 4 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 4 More or fewer components as shown, or with Figure 4 Different configurations are shown.

[0067] It should be noted that the one or more processors 402 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuit may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 40 (or mobile device). As involved in the embodiments of the present application, the data processing circuit acts as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0068] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 40 (or mobile device).

[0069] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0070] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0071] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0072] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0073] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the relevant technology or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0074] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An indicator collection system, characterized in that: It includes a monitoring server, a distributed coordination module, multiple monitoring agent modules, and multiple indicator collection processes, among which: The distributed coordination module is used to receive the configuration instructions sent by the monitoring server, and configure the first type of nodes and the second type of nodes in the distributed coordination module according to the configuration instructions, wherein the first type of nodes are used to store the indicator collection method script and the indicator collection method configuration file, and the second type of nodes are used to store the monitoring indicator information corresponding to each type of data resources, and the monitoring indicator information includes the monitoring indicator code and resource type code corresponding to the various types of data resources determined by the configuration management database; The target monitoring agent module is used to monitor the second-class node in the distributed coordination module, wherein the target monitoring agent module is any one of the multiple monitoring agent modules; the target first-class node is determined in the distributed coordination module according to the monitoring result, and the target indicator collection method script and the target indicator collection method configuration file are determined according to the target first-class node to adjust the indicator collection method of the indicator collection process connected to the target monitoring agent module, wherein the indicator collection process is set in the target device and is used to collect monitoring indicators in the target device, and the indicator collection method includes the monitoring indicator code of the monitoring indicator collected by the indicator collection process; The distributed coordination module is also used to monitor the health status of the multiple monitoring agent modules, and determine the main monitoring agent module among the multiple monitoring agent modules based on the health status, wherein the main monitoring agent module is used to control the working status of all indicator collection processes in the multiple indicator collection processes.

2. The indicator collection system according to claim 1, characterized in that: The first type of nodes include script nodes for storing the indicator collection method script, and configuration file nodes for storing the indicator collection method configuration file; The preset values ​​corresponding to the second type of nodes include a first preset value and a second preset value. The first preset value is used to indicate that the monitoring indicator corresponding to the second type of nodes is the monitoring indicator newly added within a preset time period, and the second preset value is used to indicate that the monitoring indicator corresponding to the second type of nodes is the updated monitoring indicator.

3. The indicator collection system according to claim 2, characterized in that: The configuration instructions include custom script configuration instructions and plug-in configuration instructions; configuring the first type of nodes and the second type of nodes in the distributed coordination module according to the configuration instructions includes: In the case where the configuration instruction is a custom script configuration instruction, the distributed coordination module generates an indicator collection script and an indicator collection method configuration file corresponding to the custom script configuration instruction according to the custom script configuration instruction, and stores the indicator collection script in the script node, and stores the indicator collection method configuration file in the configuration file node; In the case where the configuration instruction is the plug-in definition configuration instruction, the distributed coordination module generates a custom plug-in corresponding to the plug-in definition configuration instruction and a plug-in configuration file corresponding to the custom plug-in based on the plug-in definition configuration instruction, and stores the plug-in configuration file in the configuration file node.

4. The indicator collection system according to claim 2, characterized in that: The script node also stores the monitoring indicator code and the resource type code corresponding to the indicator collection method script, and the configuration file node also stores the monitoring indicator code corresponding to the indicator collection method configuration file, the unique identification code of the resource instance indicated by the resource type code and the monitoring indicator code, and the IP address of the resource instance, and the unique identification code is determined by the configuration management database.

5. The indicator collection system according to claim 4, characterized in that: The target monitoring agent module is also used for: After the target monitoring agent module monitors the newly added second-type node or the modified second-type node, the resource type code, the monitoring indicator code, the unique identification code and the IP address corresponding to the newly added or modified second-type node are obtained, and when it is determined that the target device corresponding to the IP address is the device corresponding to the target monitoring agent module, the script node and the configuration file node corresponding to the resource type code, the monitoring indicator code, the unique identification code and the IP address are determined; The indicator collection method script is obtained from the corresponding script node, and the indicator collection method configuration file is obtained from the corresponding configuration file node.

6. The indicator collection system according to claim 4, characterized in that: The target monitoring agent module is also used for: After the target monitoring agent module monitors the deleted second-type node, the resource type code, the monitoring indicator code, the unique identification code and the IP address corresponding to the deleted second-type node are obtained, and when it is determined that the target device corresponding to the IP address is the device corresponding to the target monitoring agent module, the indicator collection method script and the indicator collection method configuration file in the indicator collection process connected to the target monitoring agent module are deleted, wherein the indicator collection method script in the indicator collection process is the indicator collection method script corresponding to the resource type code, the monitoring indicator code and the unique identification code, and the indicator collection method configuration file in the indicator collection process is the indicator collection method configuration file corresponding to the resource type code, the monitoring indicator code and the unique identification code.

7. The indicator collection system according to claim 4, characterized in that: The distributed coordination module is also used for: Determine a target script node in which the stored indicator collection method script is modified, and after the indicator collection method script in the target script node is modified, determine the unique identification code and the IP address corresponding to the monitoring indicator code in the configuration management database according to the monitoring indicator code in the target script node, and set the preset value of the second type of node corresponding to the monitoring indicator code as the second preset value; Determine the target configuration file node corresponding to the script node based on the unique identification code corresponding to the monitoring indicator code in the target script node and the IP address, and set the preset value of the second type of node corresponding to the monitoring indicator code to the second preset value.

8. The indicator collection system according to claim 1, characterized in that: Monitoring the health status of the multiple monitoring agent modules includes: Determine a failover controller corresponding to each of the plurality of monitoring agent modules, wherein the monitoring agent modules correspond to the failover controllers one-to-one, and each pair of the monitoring agent modules and the failover controllers maintains a session; The failover controller periodically sends health monitoring instructions to each monitoring agent module, thereby obtaining the health status of each monitoring agent module.

9. The indicator collection system according to claim 1, characterized in that: The distributed coordination module is provided with a temporary node corresponding to the main monitoring agent node, and the temporary node is used as a lock for the main monitoring agent node selection process; Determining a main monitoring agent module among the plurality of monitoring agent modules according to the health status includes: After the working state of the main monitoring agent module is abnormal, the temporary node is deleted to trigger the lock grabbing process of the multiple monitoring agent modules; The monitoring agent module that grabs the lock among the multiple monitoring agent modules is determined to be the new main monitoring agent module.

10. The indicator collection system according to claim 1, characterized in that: The target monitoring agent module is also used for: Continuously obtain the heartbeat packet sent by the target indicator collection process connected to the target monitoring agent module, and if the heartbeat packet sent by the target indicator collection process is not received within a preset time period, detect the communication network between the indicator collection process and the indicator collection process, and detect the indicator collection process; When it is detected that the communication network is disconnected, the target indicator collection process performs backup processing on the collected monitoring indicators; When it is detected that the communication network is normal and the target indicator collection process is not detected, the target indicator collection process is restarted.

11. The indicator collection system according to claim 1, characterized in that: After collecting the monitoring indicators, the indicator collection process sends the monitoring indicators to the data lake, and backs up and stores the monitoring indicators locally on the target device.