Collection probe management system and method

Abstract

The embodiment of the invention provides an acquisition probe management system and method. The method can be used in distributed fields. The system comprises an acquisition probe deployment page management end, an acquisition proxy server end and an acquisition proxy end, the acquisition probe deployment page management end is used for providing a man-machine interaction function so as to create an operation task corresponding to operation task information based on the operation task information for an acquisition probe obtained by man-machine interaction; and sending the operation task to a collection agent server corresponding to the node, and displaying an execution state of the operation task, the acquisition agent server side is used for triggering the node to execute the operation task, and the acquisition agent side is deployed in the node; and the acquisition agent end is used for executing the operation task aiming at the acquisition probe. The method is used for achieving the effect of quickly deploying the corresponding acquisition probes to a plurality of servers and realizing unified management of the acquisition probes.

Description

Technical Field

[0001] The present application relates to the field of distributed systems, and in particular to a collection probe management system and method. Background Art

[0002] With the popularization of distributed cloud native technology, enterprises generally use Prometheus as a system-level monitoring tool to collect and report monitoring indicators of monitoring objects such as operating systems, middleware, and databases. Prometheus is an open source system monitoring and alarm system that supports multiple collection probes (exporters) to collect data and push gateways (pushgateway) to report data. The principle of Prometheus monitoring is to deploy various collection probes as data collectors on the monitored servers. The collected indicators are exposed to the Uniform Resource Locator (URL) for the Prometheus server to pull and then store in the time series database.

[0003] In the related art, relevant personnel manually deploy collection probes on the monitored servers. However, in large enterprises such as banks, the scale of servers deployed in an application system is huge, which makes it impossible to quickly deploy corresponding collection probes to multiple servers. Summary of the invention

[0004] The embodiments of the present application provide a collection probe management system and method for achieving the effect of quickly deploying corresponding collection probes to multiple servers.

[0005] In a first aspect, an embodiment of the present application provides a collection probe management system, including: a collection probe deployment page management end, a collection agent server end, and a collection agent end, wherein:

[0006] The collection probe deployment page management terminal is used to provide a human-computer interaction function, so as to create an operation task corresponding to the operation task information based on the operation task information for the collection probe obtained through human-computer interaction, and the operation task information includes the node identifier where the monitoring object is located; and send the operation task to the collection agent server corresponding to the node; and display the execution status of the operation task;

[0007] The collection agent server is used to trigger the node to perform operation tasks. The collection agent is deployed in the node;

[0008] The collection agent is used to perform operation tasks for the collection probes, including batch installation of collection probes, batch startup of collection probes, batch stop of collection probes, batch update of collection probe configurations, and collection probe status monitoring.

[0009] Optionally, the collection proxy server is deployed in a cluster, and the collection proxy server includes:

[0010] A response module, used for responding to received operation tasks and storing the operation tasks in a database;

[0011] The task scheduling module is used to pull operation tasks from the database in a preemptive manner and record the task pulling results in the database;

[0012] The task execution module is used to read the nodes of the operation tasks to be executed in batches, and trigger the nodes to execute the pulled operation tasks through remote procedure calls; and obtain the execution results of the operation tasks and record the execution results in the database.

[0013] Optionally, the task scheduling module is specifically used for:

[0014] In the process of pulling the operation tasks of the region in the database in a preemptive manner, determine whether the number of tasks pulled has reached the upper limit of the number of tasks;

[0015] When the number of tasks that have been pulled reaches the upper limit of the number of tasks, stop preemption.

[0016] Optionally, if the operation task is to install collection probes in batches, the operation task carries the collection probe type and the configuration version of the collection probe. The collection agent is specifically used for:

[0017] Get the collection probe type and configuration version carried in the operation task;

[0018] According to the collection probe type, pull the target collection probe installation package of the corresponding collection probe type from the file server. The file server is used to store the collection probe installation packages of different types of monitoring objects.

[0019] According to the configuration version of the collection probe, the target configuration version parameters of the configuration version of the corresponding collection probe are pulled from the database, and the database is used to store the configuration version parameters of the configuration version;

[0020] Install and start collection probes in batches according to the target collection probe installation package and target configuration version parameters;

[0021] Return the execution result to the collection agent server.

[0022] Optionally, when the collection agent starts the collection probe, it is used to:

[0023] Get the target group that should be registered in the Prometheus cluster for the collection pointer to be started. The target group is the group with the smallest load among all groups in the Prometheus cluster.

[0024] Register the collection pointer to be started to the target group.

[0025] Optionally, the target group is determined by the collection proxy server in the following way:

[0026] According to the collection pointer type of the collection pointer to be started, determine, in each group of the Prometheus cluster, a candidate group in which the collection pointer type of the collection pointer to be started can be registered;

[0027] When there are multiple candidate groups, determine the group corresponding to the minimum load among the candidate groups as the target group;

[0028] When the number of candidate groups is one, the candidate group is determined to be the target group.

[0029] Optionally, when the collection agent registers the collection pointer to be started to the target group, it is specifically used for:

[0030] Register the interface address exposed by the monitoring indicator to be collected by the collection pointer to be started to the registration center of the Prometheus cluster, so that the target group can discover the interface address to obtain the indicator data of the monitoring indicator.

[0031] Optionally, the collection agent is also used to:

[0032] Regularly report the status information of the collection agent and the status information of the collection probes managed by the collection agent to the collection agent server;

[0033] Correspondingly, the collection proxy server is also used to return the acquired status information to the collection probe deployment page management end for display by the collection probe deployment page management end.

[0034] Optionally, the collection probe management system also includes:

[0035] The configuration information synchronization module is used to synchronize node information and store the node information in the database so that the collection probe deployment page management end can create operation tasks corresponding to the operation task information.

[0036] In a second aspect, an embodiment of the present application provides a collection probe management method, which is applied to a collection agent end of a collection probe management system. The collection probe management system also includes a collection probe deployment page management end and a collection agent service end. The collection probe management method includes:

[0037] Receive operation tasks, which include batch installation of collection probes, batch startup of collection probes, batch stop of collection probes, batch update of collection probe configuration and collection probe status monitoring. The operation tasks are operation task information for collection probes obtained by the collection probe deployment page management end based on human-computer interaction, and create operation tasks corresponding to the operation task information. The operation task information includes the node identifier where the monitoring object is located; and send the operation task to the collection agent server corresponding to the node, which is triggered by the collection agent server to execute the node, and deploy the collection agent in the node;

[0038] Perform operational tasks on the collection probe.

[0039] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the collection probe management method of the second aspect as described above.

[0040] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the collection probe management method of the second aspect above.

[0041] The collection probe management system and method provided in the embodiment of the present application are developed by developing a collection probe deployment page management terminal, a collection agent server terminal and a collection agent terminal. Among them, through the collection probe deployment page management terminal, users can create operation tasks through the human-computer interaction interface. These tasks can include batch installation, start, stop, update of collection probes and other operations, which can avoid manual repetitive operations on each server and greatly improve the deployment efficiency. Through the collection agent terminal, it is possible to interact with the collection agent server terminal in a unified manner to realize functions such as collection probe management and completion of various operation tasks. Through the collection agent server terminal, as a bridge for the interaction between the collection probe deployment page management terminal and the collection agent terminal, it is possible to receive user tasks for collection probe operations and send operation and maintenance instructions to the collection agent terminal, etc. Therefore, the system supports batch operation functions, allowing the same operation tasks to be performed on multiple nodes at the same time. The collection agent server is used to centrally manage and trigger the operation tasks of each node, so that in a large-scale server environment, the collection probes can be quickly deployed to multiple servers, and multiple server nodes can be managed on one platform without having to log in to each server one by one to operate. In general, through automation and batch operations of the system, the time and effort of manual operations are significantly reduced, and the efficiency of collection probe deployment and management is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] Figure 1 The structure diagram of the acquisition probe management system provided for this application Figure 1 ;

[0044] Figure 2 The interface of the collection probe management system provided for this application Figure 1 ;

[0045] Figure 3 The interface of the collection probe management system provided for this application Figure 2 ;

[0046] Figure 4 The structure diagram of the acquisition probe management system provided for this application Figure 2 ;

[0047] Figure 5 The structure diagram of the acquisition probe management system provided for this application Figure 3 ;

[0048] Figure 6 Schematic diagram of the process of the acquisition probe management method provided in this application Figure 1 ;

[0049] Figure 7 Schematic diagram of the process of the acquisition probe management method provided in this application Figure 2 ;

[0050] Figure 8 Schematic diagram of the process of the acquisition probe management method provided in this application Figure 3 .

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0053] First, the terms involved in this application are explained:

[0054] Collection probe: refers to an exporter in Prometheus, which is used to collect and report monitoring indicators of monitoring objects such as operating systems and middleware. The Prometheus community has many open source exporters, such as node exporter for monitoring Linux operating systems and ES exporter for monitoring ES databases.

[0055] In the prior art, the collection probes are manually deployed on the monitored servers by relevant personnel. At present, the Ansible tool can also be used to push the collection probes in batches, define the system status by writing a descriptive language, and communicate with the target machine to achieve agentless automated management. However, in large enterprises such as banks, the scale of servers deployed in an application system is huge, and manual deployment is inefficient. The Ansible tool needs to execute tasks from scratch each time it is run, which makes it impossible to quickly deploy the corresponding collection probes to multiple servers.

[0056] Based on this, the present application proposes a collection probe management system and method. By developing a unified collection center, namely, a collection probe management system, including a collection probe deployment page management terminal, a collection agent server terminal and a collection agent terminal, a variety of functions are provided, which can support users to select the type of probe to be deployed and the configuration of the probe by clicking on the page, and bind a group of server nodes. The system will automatically generate corresponding tasks according to the operation requirements of the probe, and execute the nodes to be executed in the task one by one, complete the deployment of the probe, configuration update, start and stop and other operations, improve the efficiency of monitoring deployment, and realize the unified management of monitoring collection probes.

[0057] It should be noted that the collection probe management system and method provided in this application can be used in distributed fields, and can also be used in any field other than distributed fields. This application does not limit the application field of the provided collection probe management system and method.

[0058] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0059] Figure 1 The structure diagram of the acquisition probe management system provided for this application Figure 1 .like Figure 1 As shown, the system may include a collection probe deployment page management end, a collection agent server end, and a collection agent end, wherein:

[0060] The collection probe deployment page management end is used to provide human-computer interaction functions, so as to create operation tasks corresponding to the operation task information for the collection probe obtained based on the human-computer interaction, and the operation task information includes the node identifier where the monitored object is located; and send the operation task to the collection agent server corresponding to the node; and display the execution status of the operation task.

[0061] In this embodiment, the collection probe deployment page management terminal may refer to the portal layer in the software architecture, which can provide a unified access interface for users to access different applications and services through this interface. At the same time, it can integrate information from different sources so that users can access various applications and data.

[0062] Nodes can refer to servers or computing devices, which are physical or virtual machines that actually run applications or services. In the collection probe management system, nodes refer to places where collection probes need to be deployed and managed.

[0063] As an example, the interface of the collection probe deployment page management terminal is as follows Figure 2 and Figure 3 Specifically, as shown Figure 2 As shown, this interface is a visual interface provided by the collection probe system. In this visual interface, users can click on node management, select batch operation, and then select the product and operation type according to their needs, and select the type of collection probe to be installed and the version of the collection probe configuration. Users can also select to filter the selectable hosts from the range of hosts currently existing in the configuration management database (CMDB) in this interface to perform batch operations on the collection probes. Among them, a product can refer to an application system that provides a class of functions or services to the outside world, and a product can include multiple microservices. CMDB is used to record the deployment resource information of the product, such as which servers are deployed under each product, which operating system and which middleware are deployed on each server, etc. This information can be used to filter servers by server role when deploying collection probes.

[0064] Further, such as Figure 3 As shown in the figure, after filtering the hosts, the selected host list will be displayed on the current batch operation page. In this visual interface, the selectable regions and availability zones are automatically displayed according to the selected product for users to select. The region can refer to the city where the data center belongs, and the availability zone can refer to different data centers in the same region.

[0065] As a result, the background will generate corresponding batch operation tasks in the database. For example, according to the region to which the selected IP belongs, a task for batch installation of collection probes is created in each region.

[0066] The operation task information may include the task name, operation type (such as batch installation of collection probes), collection probes (including the type of collection probes), probe configuration version, execution IP, task status (created, pending, executing, completed, etc.), and executor (records which instance in the collection agent server has claimed the current task. This field will be filled in after the collection agent server schedules the task later).

[0067] The task information corresponding to the created task will be stored in the TDSQL-task management database, and the probe configuration related information (including configuration name, collection probe, configuration content) will be stored in the TDSQL-probe configuration management database. The probe configuration version in the TDSQL-task management database is associated with the configuration name in the TDSQL-probe configuration management database. The configuration information in the TDSQL-probe configuration management database mainly includes parameters such as environment variables, startup parameters, probe collection frequency, and resource usage limit. Each collection probe configuration supports multiple configuration versions. You can select the desired configuration version when deploying the collection probe.

[0068] It should be noted that the above-mentioned databases are all distributed databases, and other types of databases may also be used, which is not limited in this application.

[0069] In this embodiment, the operation task information includes a clear node identifier to ensure that the task can be correctly assigned to the corresponding server or device. The node identifier can help the system know which specific machine to execute the task on.

[0070] It should be noted that the collection probe deployment page management end can provide a variety of batch operation functions, such as batch installation of collection probes, batch start / stop of probes, batch update of probe configurations, and probe status monitoring. The client also provides functions to display the execution status of each operation task, so that users can view whether the task is successfully executed, whether there are errors, or the current progress of the task through the interface.

[0071] It can be understood that the collection probe deployment page management terminal is a bridge between the user and the collection probe management system. It not only allows users to create and manage operation tasks, but also provides real-time feedback so that users can monitor the execution of tasks. This design improves the system's usability and user experience, ensuring that users can effectively manage and monitor large-scale server environments.

[0072] The collection agent server is used to trigger the node to perform operation tasks. The collection agent is deployed in the node.

[0073] In this embodiment, the collection agent server may refer to the server of the unified collection agent (OneAgentManager, OAM), and the collection agent may refer to the unified collection agent (oneAgent).

[0074] The collection agent server is the core component of the system. It is responsible for receiving operation task requests from the collection probe deployment page management end, coordinating and managing the execution of these operation tasks, and further determining which nodes need to perform which specific operation tasks based on the task requests, and distributing these operation tasks to the corresponding nodes for execution on these nodes.

[0075] The collection agent server is clustered and can be deployed in multiple areas at the same time (such as area A and area B). Each area has multiple instances to receive tasks and send the task instructions to the collection agent of a node in the monitored server.

[0076] The collection agent is a local agent program that is responsible for performing specific operation tasks on the node. When the collection agent receives instructions from the collection agent server, it will perform these tasks locally, such as installing, starting, stopping or updating the collection probe.

[0077] It is understandable that through the architectural design, the collection agent server and the collection agent jointly realize the mode of centralized management and distributed execution. The collection agent server, as the central control point, is responsible for the distribution and coordination of tasks, while the collection agent performs specific tasks on each node. This design not only improves the scalability and management efficiency of the system, but also reduces the need for manual intervention and reduces the possibility of errors.

[0078] The collection agent is used to perform operation tasks for the collection probes, including batch installation of collection probes, batch startup of collection probes, batch stop of collection probes, batch update of collection probe configurations, and collection probe status monitoring.

[0079] Batch updating of collection probe configurations may refer to updating configuration parameters of multiple collection probes to adapt to new monitoring requirements or strategies.

[0080] The collection probe status monitoring may refer to monitoring the running status of the collection probe, ensuring that it works normally, and collecting running data.

[0081] In this embodiment, the main responsibility of the collection agent is to execute the operation tasks from the collection agent server on the local node. These tasks are distributed by the collection agent server based on the task requests defined by the user on the collection probe deployment page management end.

[0082] It should be noted that when the collection agent performs the task of installing or updating the collection probe, it needs to pull the collection probe installation package from the file server (FTP server) for installation.

[0083] It is understandable that the collection agent supports batch operations and can perform the same task on multiple collection probes at the same time, which can significantly improve operational efficiency in a large-scale server environment.

[0084] Optionally, the collection agent is also used to periodically report status information of the collection agent and status information of the collection probes managed by the collection agent to the collection agent server.

[0085] Correspondingly, the collection proxy server is also used to return the acquired status information to the collection probe deployment page management end for display by the collection probe deployment page management end.

[0086] The state information of the collection agent may refer to the running state, resource usage, version information, etc. of the agent itself. The state information of the collection probe may refer to the running state, performance index, error information, etc. of each collection probe.

[0087] In this embodiment, after the collection agent completes the deployment of the collection probe, during daily operation, the collection agent on each monitored server will regularly report the heartbeat to the collection agent server, including the state of the collection agent itself and the state of each collection probe managed by the collection agent, so as to perform status monitoring and alarm. Among them, the heartbeat signal is a simple message sent regularly to indicate that a component (such as the collection agent) is still running normally. By receiving the heartbeat signal, the collection agent server can determine whether the collection agent is online and healthy.

[0088] The collection agent server receives status information from multiple collection agents, and can summarize, analyze and store this information for subsequent use, such as sending the status information to the collection probe deployment page management end, so that the client can display the status information of each node and collection probe in real time.

[0089] It is understandable that regular reporting of status information helps ensure the transparency and monitorability of the system. Through this feedback mechanism, the status of the collection agent and collection probe can be understood, and users can promptly discover and resolve potential problems to ensure the stable operation of the system.

[0090] Optionally, the collection probe management system further includes: a configuration information synchronization module for synchronizing node information and storing the node information in a database so that the collection probe deployment page management end can create operation tasks corresponding to the operation task information.

[0091] In this embodiment, the collected node information is stored in the TDSQL-node management database, so that the system can easily access and manage the node information. The node information includes the node IP, the product, the region, and the deployed middleware.

[0092] The configuration information synchronization module is responsible for regularly synchronizing the server node (IP) information of each product, as well as the operating system and middleware information deployed on the node from the CMDB, parsing batch files and storing the above information in the database, so that when users deploy collection probes, they can filter the servers to be bound according to certain conditions.

[0093] It can be understood that the synchronization process of the configuration information synchronization module ensures that all node information is up-to-date and remains consistent in the system. By accessing the node information in the database, the user can accurately select the target node and create corresponding operation tasks in the collection probe deployment page management terminal.

[0094] The collection probe management system provided in the embodiment of the present application realizes the automated deployment and centralized management of collection probes through the developed collection probe deployment page management terminal, collection agent server terminal and collection agent terminal, so that users can perform batch operations on collection probes and monitor the status in real time, thereby improving the convenience and controllability of overall operation and maintenance, and solving the problem of inefficiency in manual deployment of collection probes in large enterprises (such as banks).

[0095] Based on the above embodiments, Figure 4 The structure diagram of the acquisition probe management system provided for this application Figure 2 .like Figure 4 As shown, in this embodiment Figure 1 Based on the embodiment, the acquisition agent server is described in detail, including:

[0096] In this embodiment, the collection agent server is deployed in a cluster. Specifically, a collection agent server cluster is deployed in each region, and the cluster includes multiple instances.

[0097] The response module is used to respond to the received operation tasks and store the operation tasks in the database.

[0098] It can be understood that the response module is responsible for receiving operation task requests from the collection probe deployment page management end, and storing the received operation tasks in the database for subsequent processing, thereby ensuring that all task requests are reliably received and stored, laying the foundation for task scheduling and execution, and providing persistent storage of task requests to prevent task loss.

[0099] The task scheduling module is used to pull operation tasks from the database in a preemptive manner and record the task pulling results in the database.

[0100] In this embodiment, multiple instances in the acquisition proxy server take over the tasks of the local domain in a preemptive manner, and record the task pulling results in the database to maintain the consistency and traceability of the task status.

[0101] It is understandable that preemptive scheduling ensures that tasks can be processed in a timely manner and avoids task backlogs. At the same time, recording the task pull status can help the system manage the life cycle and status tracking of tasks.

[0102] The task execution module is used to read the nodes of the operation tasks to be executed in batches, and trigger the nodes to execute the pulled operation tasks through remote procedure calls (RPC); and obtain the execution results of the operation tasks and record the execution results in the database.

[0103] It is understandable that efficient task distribution and execution of nodes can be achieved through batch processing and RPC calls, and the recorded execution results can provide feedback on the completion of the task, facilitating subsequent analysis and troubleshooting.

[0104] Optionally, the task scheduling module is specifically used to: in the process of pulling operation tasks for the area in the database in a preemptive manner, determine whether the number of tasks pulled has reached the upper limit of the number of tasks; when the number of tasks pulled has reached the upper limit of the number of tasks, stop preemption.

[0105] In this embodiment, in order to ensure that the tasks preempted by each acquisition proxy server instance are balanced and avoid starvation, the system sets an upper limit on the number of tasks preempted by each instance. The upper limit can be parameterized according to information such as the actual number of tasks and the available resources of the instance. Once the number of tasks currently received reaches the upper limit, tasks will no longer be preempted to prevent some instances from being overloaded due to too many tasks. Among them, starvation refers to the situation where some processes or tasks cannot obtain the required resources for a long time under the condition of improper or unfair resource allocation, resulting in their inability to execute or complete.

[0106] In some embodiments, a dynamic adjustment mechanism can also be implemented to dynamically adjust the upper limit of the number of tasks based on the overall load of the system and the performance of each instance. For example, the upper limit can be appropriately increased during peak task periods and lowered during low load periods to optimize resource utilization.

[0107] It is understandable that by performing task scheduling and quantity control of each instance in the acquisition proxy server through the task scheduling module, resource utilization can be optimized and the overall performance of the system can be improved.

[0108] The collection probe management system provided in the embodiment of the present application receives operation task requests from the collection probe deployment page management end through the response module in the collection agent server, which can prevent the loss of task requests, pulls the operation tasks to be executed and schedules them through the task scheduling module, which can ensure that the tasks are processed normally, and reads the node information of the operation tasks to be executed in batches through the task execution module and triggers the tasks, which can realize efficient task distribution and execution of nodes.

[0109] Based on the above embodiments, Figure 5 The structure diagram of the acquisition probe management system provided for this application Figure 3 .like Figure 5 As shown, in this embodiment Figure 1 Based on the embodiment, the collection probe management system is further described, including:

[0110] The Prometheus server is deployed in clusters. Each region has a Prometheus cluster. There are multiple groups in a cluster. Different groups are responsible for obtaining indicators of different types of monitoring objects. For example, group 1 is used to collect monitoring indicators of all middleware, and group 2 is used to obtain indicators of all operating systems. There are two prometheus servers in each group, which are high availability (HA) master and slave to achieve high availability. The two prometheus server instances collect the same exporter target.

[0111] It can be understood that these two instances collect the same collection probe target, ensuring that when one instance fails, the other instance can seamlessly take over the task and avoid data collection interruption.

[0112] The group information under each Prometheus cluster is recorded in the TDSQL-Prometheus cluster group information database. The database table records the role of each group (for example, which type of monitoring object it is responsible for), the service instance under the group (the specific IP address of the server), the number of collection probes registered for the current group, and the maximum number of registrations.

[0113] The collection agent server is used to collect probe registration group calculation and configuration delivery.

[0114] Registry (consul), used for service discovery.

[0115] As an example, when you need to obtain middleware indicators, all nodes that deploy middleware collection probes (such as ES exporter, nginx exporter, etc.) will use tags to identify that the current collection probe belongs to group 1 when the collection probe is registered with the registration center. That is, the exposed indicators will be subsequently pulled by the prometheus server of group 1.

[0116] The collection probe management system provided in the embodiment of the present application ensures that the system can still operate normally in the event of a single point failure through multiple groups of the Prometheus cluster and multiple instances under the groups. Through the management of groups and the maximum number of registrations, load balancing of each group is ensured to prevent overload. At the same time, new groups can be added or the configuration of existing groups can be adjusted as needed to adapt to different monitoring requirements. Finally, the group information is recorded in the database to achieve centralized management and monitoring of the entire monitoring system.

[0117] Figure 6 Schematic diagram of the process of the acquisition probe management method provided in this application Figure 1 .like Figure 6 As shown, the method is applied to the collection agent end of the collection probe management system. The collection probe management system also includes a collection probe deployment page management end and a collection agent service end. The collection probe management method specifically includes the following steps:

[0118] S601. Receive operation tasks, which include batch installation of collection probes, batch startup of collection probes, batch stop of collection probes, batch update of collection probe configurations and collection probe status monitoring. The operation tasks are operation task information for collection probes obtained by the collection probe deployment page management end based on human-computer interaction, and the operation tasks corresponding to the operation task information are created. The operation task information includes the node identifier where the monitored object is located; and, send the operation tasks to the collection agent server corresponding to the node, which is triggered by the collection agent server to execute the node, and deploy the collection agent in the node.

[0119] S602: Execute an operation task on the collection probe.

[0120] In this embodiment, the collection agent receives the operation task instruction from the collection agent server, downloads the necessary probe software package on the node and configures the initial parameters, thereby installing multiple collection probes in batches.

[0121] Start the collection probes in batches, load the configuration files, initialize the operating environment, start related processes, etc., so that they can start to perform monitoring tasks.

[0122] When necessary, stop running collection probes in batches, such as during maintenance, updates, or troubleshooting. Ensure that resources are released after the probes are stopped and record the stop status for subsequent analysis.

[0123] Adapt to new monitoring requirements or strategies by modifying configuration files, restarting collection probes to apply new configurations, etc. Continuously monitor the running status of collection probes, collect performance indicators, error logs and other information, perform status monitoring, help identify abnormal conditions in probe operation, and provide data support to optimize collection probe performance.

[0124] The collection probe management method provided in the embodiment of the present application is applied to the collection agent end. Through batch processing, the time and complexity of manual operations are significantly reduced, and management efficiency is improved. Through continuous status monitoring, problems in probe operation can be discovered and resolved in a timely manner, ensuring the stable operation of the system and flexible adaptation to monitoring needs and environmental changes. The automated and standardized operating procedures reduce the possibility of human errors and reduce operational risks.

[0125] Based on the above embodiments, Figure 7 Schematic diagram of the process of the acquisition probe management method provided in this application Figure 2 .like Figure 7 As shown, if the operation task is to install collection probes in batches, the operation task carries the collection probe type and the configuration version of the collection probe, which is applied to the collection agent. The method includes:

[0126] S701: Obtain the type of collection probe and the configuration version of the collection probe carried in the operation task.

[0127] The operation task is usually a data structure containing multiple fields, including relevant information of the collection probe, such as the collection probe type and the configuration version of the collection probe. The above information can be obtained by parsing and extracting the operation task.

[0128] The collection probe type is used to indicate the function or purpose of the probe, such as monitoring databases, middleware, operating systems, etc.

[0129] The configuration version of a collection probe refers to a specific configuration set, including parameter settings, operating environment requirements, etc., which is used to ensure that the correct configuration is applied during the installation process so that the collection probe can run normally and collect the required monitoring data.

[0130] It is understandable that by acquiring various information carried by the operation task, the system can collect sufficient information to complete the subsequent installation and configuration process, ensuring that the collection probe can be deployed according to specific needs and environment.

[0131] S702. Pull a target collection probe installation package of the corresponding collection probe type from a file server according to the collection probe type.

[0132] Among them, the file server is used to store the collection probe installation packages of different types of monitoring objects. The file server is usually organized into a directory structure. Each type of collection probe corresponds to a specific directory or file path, so that the installation package path on the file server is determined according to the collection probe type.

[0133] The file server may be a local server, a remote server, or a cloud-based storage service. In this embodiment, the file server is an FTP server.

[0134] In some embodiments, in order to ensure the integrity and security of the target acquisition probe installation package, the integrity of the downloaded file may also be verified through a verification process.

[0135] It can be understood that by pulling the target collection probe installation package of the corresponding collection probe type from the file server according to the collection probe type, the system can obtain the correct target collection probe installation package and prepare for subsequent installation and configuration steps, ensuring the consistency of the collection probe type and the installation package, thereby avoiding operational problems caused by installing the wrong package.

[0136] S703: According to the configuration version of the collection probe, pull the target configuration version parameters corresponding to the configuration version of the collection probe from the database.

[0137] The database is used to store the configuration version parameters of the configuration version, and can be a relational database (such as MySQL, PostgreSQL) or a NoSQL database (such as MongoDB), depending on the system design. In this embodiment, it is a TDSQL database.

[0138] The target configuration version parameters include all necessary parameters such as collection frequency, data storage path, network settings, authentication information, etc., so as to be applied to the configuration of the collection probe in subsequent steps.

[0139] S704. According to the target collection probe installation package and target configuration version parameters, batch install and start the collection probes.

[0140] In this embodiment, through the pre-deployed collection agent, the system can automatically install and start collection probes in batches, and each collection probe is connected to its data source (Prometheus server).

[0141] S705: Return the execution result to the collection agent server.

[0142] In this embodiment, the collection agent server sends the registration group result information to the collection agent on each monitored server respectively, indicating that the selected Promethues group name is passed to the collection agent as a tag, so that the collection agent starts the collection probe to be started on the current node according to the grouping result and registers it with consul. The tag is also used to identify the Promethues group during registration. Then the collection agent server updates the number of collection probe registrations in the TDSQL-Prometheus cluster group information database table, that is, adds the number of collection probes newly registered to the current group, so as to facilitate subsequent monitoring and auditing, and provide data support for future analysis.

[0143] Optionally, when starting the collection probe, the following steps are specifically included:

[0144] Get the target group where the collection pointer to be started should be registered in the Prometheus cluster. The target group is the group with the smallest load among all groups in the Prometheus cluster. Register the collection pointer to be started to the target group.

[0145] The load can be measured by a variety of indicators, such as the number of currently registered collection probes, the number of processed indicators, CPU usage, etc. Based on the evaluation results, the group with the smallest load is selected as the target group, thereby achieving load balancing and preventing some groups from being overloaded.

[0146] The collection probe to be started can be registered in the selected target group by adding the relevant information of the collection probe (such as indicator type, collection frequency, data source, etc.) to the configuration of the target group. After the registration is completed, the group information database of the Prometheus cluster is updated and the number of registered probes in the current group is increased to reflect the new collection probe registration.

[0147] It can be understood that by registering the collection probe to the target group with the smallest load, the system can ensure that the collection probe is registered and run in the Prometheus cluster in the best way, which not only improves the resource utilization efficiency of the system, but also enhances the overall stability and scalability.

[0148] Optionally, when registering the collection pointer to be started to the target group, the following steps are specifically included:

[0149] Register the interface address exposed by the monitoring indicator to be collected by the collection pointer to be started to the registration center of the Prometheus cluster, so that the target group can discover the interface address to obtain the indicator data of the monitoring indicator.

[0150] Prometheus supports multiple service discovery methods, including static configuration, DNS service discovery, Consul, Kubernetes, etc. In this embodiment, the interface address is registered to Consul of the Prometheus cluster, so that the prometheus server in the target group discovers the interface address and obtains the indicator data.

[0151] The collection probe management method provided in the embodiment of the present application, after the collection agent receives the operation instruction issued by the collection agent server, takes the batch deployment of collection probes as an example, pulls the collection probes to be deployed from the FTP server, combines the configuration information of the collection probes, installs and starts the collection probes, and feeds back the execution results to the collection agent server, which can ensure that the batch installation process of the collection probes is efficient and controllable, and through the automated process, can reduce manual intervention and improve deployment efficiency and accuracy.

[0152] Based on the above embodiments, Figure 8 Schematic diagram of the process of the acquisition probe management method provided in this application Figure 3 .like Figure 8 As shown, in this embodiment Figure 7 Based on the embodiment, the process of determining the target group by the acquisition agent server is described in detail, which specifically includes the following steps:

[0153] S801. According to the collection pointer type of the collection pointer to be started, determine, in each group of the Prometheus cluster, a candidate group in which the collection pointer type of the collection pointer to be started can be registered.

[0154] The candidate group may refer to a group that can support this type of collection pointer. Each group may be configured to handle a specific type of monitoring object (such as middleware, operating system, etc.), so it is necessary to ensure that the type of collection probe to be started matches the role of the group and filter out those groups that do not support this type of collection pointer.

[0155] S802: When there are multiple candidate groups, determine a group corresponding to the smallest load among the candidate groups as the target group.

[0156] In this embodiment, the load is measured by the number of registered collection probes in the group. The registration status of the collection probes can be queried by calling the registration center interface and comparing the load status of each group, thereby determining the target group with the minimum load in the candidate groups.

[0157] In some embodiments, real-time load data may also be obtained through a Prometheus application programming interface (API) or a monitoring system.

[0158] S803: When the number of candidate groups is one, determine the candidate group as a target group.

[0159] The collection probe management method provided in the embodiment of the present application can ensure that the collection probe is registered in the most appropriate Prometheus group by selecting the target group to complete the registration of the collection probe according to the collection probe type and the load conditions of each group, optimize the use of system resources through load balancing, and improve the efficiency and stability of the system.

[0160] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0161] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0162] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0163] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0164] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0165] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0166] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0167] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0168] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0169] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A collection probe management system, characterized in that: include: The collection probe deploys the page management end, the collection agent server end, and the collection agent end, among which: The acquisition probe deployment page management terminal is used to provide a human-computer interaction function, to create an operation task corresponding to the operation task information based on the operation task information for the acquisition probe obtained by human-computer interaction, wherein the operation task information includes the node identifier where the monitored object is located; and to send the operation task to the acquisition agent server corresponding to the node; and to display the execution status of the operation task; The acquisition agent server is used to trigger the node to perform the operation task, and the acquisition agent is deployed in the node; The collection agent is used to execute the operation tasks for the collection probes, and the operation tasks include batch installation of collection probes, batch startup of collection probes, batch stop of collection probes, batch update of collection probe configurations and collection probe status monitoring.

2. The collection probe management system according to claim 1, characterized in that: The acquisition agent server is deployed in a cluster, and includes: A response module, used for responding to received operation tasks and storing the operation tasks in a database; A task scheduling module, used to pull operation tasks from the database in a preemptive manner, and record the task pulling results in the database; The task execution module is used to read the nodes of the operation tasks to be executed in batches, and trigger the nodes to execute the pulled operation tasks through remote procedure calls; and obtain the execution results of the operation tasks and record the execution results in the database.

3. The collection probe management system according to claim 2, characterized in that: The task scheduling module is specifically used for: In the process of pulling the operation tasks of the area in the database in a preemptive manner, determining whether the number of tasks pulled has reached an upper limit on the number of tasks; When the number of tasks that have been pulled reaches the upper limit of the number of tasks, stop preemption.

4. The collection probe management system according to any one of claims 1 to 3, characterized in that: If the operation task is to install collection probes in batches, the operation task carries the collection probe type and the configuration version of the collection probe. The collection agent is specifically used for: Get the collection probe type and configuration version carried in the operation task; According to the collection probe type, pull the target collection probe installation package of the corresponding collection probe type from the file server, where the file server is used to store collection probe installation packages of different types of monitored objects; According to the configuration version of the collection probe, pull the target configuration version parameters of the configuration version corresponding to the collection probe from the database, where the database is used to store the configuration version parameters of the configuration version; According to the target collection probe installation package and the target configuration version parameters, batch install and start the collection probes; The execution result is returned to the acquisition agent server.

5. The collection probe management system according to claim 4, characterized in that: When the collection agent starts the collection probe, it is specifically used to: Obtain a target group that should be registered in the Prometheus cluster for the collection pointer to be started, wherein the target group is the group with the smallest load among the groups of the Prometheus cluster; Register the to-be-started collection pointer to the target group.

6. The collection probe management system according to claim 5, characterized in that: The target group is determined by the acquisition agent server in the following manner: According to the collection pointer type of the collection pointer to be started, determining, in each group of the Prometheus cluster, a candidate group in which the collection pointer type of the collection pointer to be started can be registered; When there are multiple candidate groups, determine a group corresponding to the minimum load among the candidate groups as the target group; When the number of the candidate groups is one, the candidate group is determined to be the target group.

7. The collection probe management system according to claim 5, characterized in that: When the collection agent registers the to-be-started collection pointer to the target group, it is specifically used to: The interface address exposed by the monitoring indicator to be collected by the collection pointer to be started is registered in the registration center of the Prometheus cluster, so that the target group discovers the interface address to obtain the indicator data of the monitoring indicator.

8. The collection probe management system according to any one of claims 1 to 3, characterized in that: The acquisition agent is also used for: Regularly reporting the status information of the collection agent and the status information of the collection probes managed by the collection agent to the collection agent server; Correspondingly, the collection agent server is also used to return the acquired status information to the collection probe deployment page management end for display by the collection probe deployment page management end.

9. The collection probe management system according to any one of claims 1 to 3, characterized in that: The acquisition probe management system also includes: The configuration information synchronization module is used to synchronize node information and store the node information in the database so that the acquisition probe deployment page management end can create an operation task corresponding to the operation task information.

10. A collection probe management method, characterized in that: The acquisition agent end is applied to the acquisition probe management system, the acquisition probe management system also includes the acquisition probe deployment page management end and the acquisition agent service end, and the acquisition probe management method includes: Receive operation tasks, the operation tasks include batch installation of collection probes, batch startup of collection probes, batch stop of collection probes, batch update of collection probe configurations and collection probe status monitoring, the operation tasks are operation task information for collection probes obtained by the collection probe deployment page management end based on human-computer interaction, create operation tasks corresponding to the operation task information, the operation task information includes the node identifier where the monitoring object is located; and send the operation tasks to the collection agent server corresponding to the node, the collection agent server triggers the execution of the node, and the collection agent is deployed in the node; Perform operational tasks on the collection probe.

Images