Processing method and system of data center adaptive monitoring and collecting system, and medium

Abstract

The application provides a processing method and system of a data center adaptive monitoring collection system and a medium. The processing method is applied to the data center adaptive monitoring collection system, and the monitoring collection system comprises a monitoring data acquisition module, a monitoring scheduling module and a monitoring collection module. The monitoring collection scheduler is used to acquire variable data information pre-grouped in a registration center. When the instruction operation information is determined to be new instruction operation information, a KNN scheduling algorithm is used to assign a target monitoring server to a to-be-processed monitoring collector identified by the to-be-processed monitoring collector, and the attribute information of the to-be-processed monitoring collector or the attribute information of the target monitoring server is updated in the pre-grouping in the registration center. A monitoring collection executor is triggered to generate configuration information identifiable by the target monitoring server for the to-be-processed monitoring collector, so as to complete the monitoring collection scheduling processing. The processing method provided by the application can improve the stability of the data center and the efficiency of the monitoring collection scheduling processing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data monitoring, and particularly relates to a processing method and system of a data center adaptive monitoring collection system and a medium. BACKGROUND

[0002] As a key content of "new infrastructure", a data center carries the most core and key assets of information systems in various industries and is also the cornerstone and engine of digital economy. Therefore, the reliability of the data center is crucial. Real-time monitoring of the operation state of the data center and generation and pushing of abnormal alarm information are particularly important for ensuring stable operation of the data center.

[0003] In the prior art, the monitoring of the data center usually needs to be manually operated and processed, that is, a technical personnel needs to pre-establish matching rules of a monitoring server and a monitoring collector and store the matching rules in a monitoring system configuration management library, synchronizes the matching rules to a registration center based on a data synchronization tool, and enables the monitoring collection executor to monitor data of the data center according to the matching rules.

[0004] However, the matching rules in the prior art are relatively mechanical and can only be monitored and processed according to preset matching rules. When there is a new business change, the technical personnel needs to re-set the matching rules, which leads to slow processing efficiency. In addition, the collection resources of the monitoring server can only be monitored and processed according to preset resource configurations, which lacks flexibility. Furthermore, the synchronization of the matching rules between the monitoring server and the monitoring collector is in the monitoring server, which makes it impossible to automatically generate a topology diagram of the monitoring server and the monitoring collector, leading to the fact that when a host where a certain monitoring server is located is down, the affected monitoring business cannot be quickly located, and thus the stability of the data center is poor. SUMMARY

[0005] The present application provides a processing method, system and medium of a data center adaptive monitoring collection system to solve the problems of slow processing efficiency, poor flexibility and poor stability of the monitoring system of the data center in the prior art.

[0006] In a first aspect, the present application provides a processing method of a data center adaptive monitoring collection system. The processing method is applied to the data center adaptive monitoring collection system. The monitoring collection system includes a monitoring data acquisition module, a monitoring scheduling module and a monitoring collection module. The monitoring scheduling module includes a monitoring collection scheduler, a monitoring collection executor and a registration center.

[0007] The processing method includes the following steps.

[0008] The monitoring collection scheduler acquires the changed data information pre-grouped in the registry center by the data synchronization tool in the monitoring data acquisition module, and the changed data includes instruction operation information and to-be-processed monitoring collector identifier;

[0009] The monitoring collection scheduler judges whether the instruction operation information in the changed data information is new instruction operation information, if yes, uses KNN scheduling algorithm to allocate a target monitoring server for the to-be-processed monitoring collector corresponding to the to-be-processed monitoring collector identifier, and updates attribute information of the to-be-processed monitoring collector or attribute information of the target monitoring server in the registry center pre-grouping;

[0010] The monitoring collection scheduler triggers the monitoring collection executor, so that the monitoring collection executor generates configuration information identifiable by the target monitoring server for the to-be-processed monitoring collector according to attribute information of the to-be-processed monitoring collector or the target monitoring server and configuration parameters of the configuration module center of the monitoring scheduling module, acquires key_value key-value pair relationship between the to-be-processed monitoring collector and the target monitoring server, and synchronizes the key_value key-value pair relationship to the registry center running group, so as to complete monitoring collection scheduling processing.

[0011] In a preferred embodiment, the monitoring scheduling module further includes a preprocessing probe, and the monitoring collection scheduler judges whether the instruction operation information in the changed data information is new instruction operation information, if yes, uses KNN scheduling algorithm to allocate a target monitoring server for the to-be-processed monitoring collector, and updates attribute information of the to-be-processed monitoring collector or the target monitoring server in the registry center pre-grouping, including:

[0012] The monitoring collection scheduler triggers the preprocessing probe to acquire collection amount information of the to-be-processed monitoring collector, and acquires central characteristic value of the to-be-processed monitoring collector and generates a first feature vector in combination with attribute information of the to-be-processed monitoring collector;

[0013] The monitoring collection scheduler acquires central characteristic value of the monitoring server in the registry center and generates a second feature vector;

[0014] The monitoring collection scheduler respectively calculates central distance between the first feature vector and the second feature vector, acquires a central distance vector, and calculates a product of a pre-configured weight vector and the central distance vector to acquire a weighted distance value;

[0015] The monitoring collection scheduler judges whether the weighted distance value is greater than a preconfigured weighted distance threshold value, and if yes, newly creates attribute information of a target monitoring server in a pre-grouping in the registration center.

[0016] In a preferred embodiment, the method further comprises:

[0017] The monitoring collection scheduler judges whether the weighted distance value is less than the preconfigured weighted distance threshold value, and if yes, selects a monitoring server corresponding to the highest priority weighted distance value in the registration center as the target monitoring server.

[0018] The monitoring collection scheduler updates attribute information of the to-be-processed monitoring collector according to attribute information of the target monitoring server.

[0019] In a preferred embodiment, the method further comprises:

[0020] If the monitoring collection scheduler judges that the instruction operation information in the change data information is a deletion instruction operation information, the monitoring collection scheduler triggers the monitoring collection executor to determine target attribute information corresponding to the to-be-processed monitoring collector identifier in a running grouping in the registration center according to the to-be-processed monitoring collector identifier.

[0021] The monitoring collection scheduler triggers the monitoring collection executor to generate deletion configuration information related to the to-be-processed monitoring collector according to the target attribute information, and synchronizes the deletion configuration information to a monitoring server having a binding relationship with the to-be-processed monitoring collector identifier.

[0022] The monitoring collection scheduler triggers the monitoring server having the binding relationship with the to-be-processed monitoring collector identifier to automatically hot load the deletion configuration information, so as to complete a deletion processing operation on the to-be-processed monitoring collector.

[0023] The monitoring collection scheduler triggers the monitoring collection executor to perform a deletion processing operation on the to-be-processed monitoring collector in the running grouping in the registration center when it is judged that the monitoring server having the binding relationship with the to-be-processed monitoring collector identifier completes the deletion processing operation within a preset time.

[0024] In a preferred embodiment, the method further comprises:

[0025] The monitoring collection scheduler triggers the monitoring collection executor to acquire a daily average resource usage rate of each monitoring server at a designated time.

[0026] The monitoring collection scheduler triggers the monitoring collection executor to determine whether the daily average resource usage meets the condition of automatic shrinkage processing for each of the daily average resource usage.

[0027] If the condition of automatic shrinkage processing is met, the monitoring collection scheduler triggers the monitoring collection executor to adjust the pre-occupied resource configuration of the monitoring service end corresponding to the daily average resource usage that meets the condition of automatic shrinkage processing, so as to complete the automatic shrinkage processing.

[0028] In a preferred embodiment, the condition of automatic shrinkage processing includes:

[0029] The daily average resource usage is less than a pre-configured low-load threshold, and the number of consecutive days in which the monitoring service end corresponding to the daily average resource usage is less than the pre-configured low-load threshold is greater than a pre-configured low-load day number.

[0030] In a preferred embodiment, the method further includes:

[0031] If the condition of automatic shrinkage processing is not met and the daily average resource usage is greater than a pre-configured high-load threshold, the monitoring collection scheduler triggers the monitoring collection executor to perform corresponding automatic expansion processing according to the remaining collection amount quota in the attribute information of the monitoring service end corresponding to the daily average resource usage.

[0032] In a preferred embodiment, the method further includes:

[0033] If the remaining collection amount quota is greater than zero, the monitoring collection scheduler triggers the monitoring collection executor to adjust the pre-occupied resource configuration of the monitoring service end corresponding to the daily average resource usage, so as to complete the automatic expansion processing.

[0034] In a preferred embodiment, the method further includes:

[0035] If the remaining collection amount quota is less than zero, the monitoring collection scheduler triggers the monitoring collection executor to adjust the field value of the operation field in the attribute information of the monitoring collection device corresponding to the daily average resource usage to a clearing processing, and performs the clearing processing according to the field value of the operation field to obtain a monitoring collection device after the clearing processing.

[0036] The monitoring collection scheduler triggers the monitoring collection executor to synchronize the attribute information of the monitoring collection device after the clearing processing to the registration center pre-grouping.

[0037] In a preferred embodiment, the method further includes:

[0038] The monitoring collection scheduler displays the association topology graph of the running host, the monitoring service end and the monitoring collector on the console interface of the monitoring scheduling module according to the attribute information of all the monitoring collectors and monitoring service ends of the registration center and the association information of each monitoring service end and the running host carrying the monitoring service end.

[0039] In a second aspect, the application provides a data center adaptive monitoring collection system, which comprises a monitoring data acquisition module, a monitoring scheduling module and a monitoring collection module, wherein the monitoring scheduling module comprises a monitoring collection scheduler, a monitoring collection executor and a registration center.

[0040] The monitoring collection scheduler is configured to acquire variable data information pre-grouped by the registration center and synchronized by a data synchronization tool in the monitoring data acquisition module, wherein the variable data comprises instruction operation information and a to-be-processed monitoring collector identifier.

[0041] The monitoring collection scheduler is further configured to determine whether the instruction operation information in the variable data information is new instruction operation information, and if so, to assign a target monitoring service end to a to-be-processed monitoring collector corresponding to the to-be-processed monitoring collector identifier by using a KNN scheduling algorithm, and to update attribute information of the to-be-processed monitoring collector or attribute information of the target monitoring service end in the registration center pre-grouping.

[0042] The monitoring collection scheduler is further configured to trigger the monitoring collection executor to generate configuration information identifiable by the target monitoring service end for the to-be-processed monitoring collector according to attribute information of the to-be-processed monitoring collector or the target monitoring service end and configuration parameters of a configuration module center of the monitoring scheduling module, to acquire a key_value key-value pair relationship between the to-be-processed monitoring collector and the target monitoring service end, and to synchronize the key_value key-value pair relationship to the registration center running grouping, so as to complete monitoring collection scheduling processing.

[0043] In a third aspect, the application provides a monitoring collection scheduler, which comprises:

[0044] The acquisition module is configured to acquire variable data information pre-grouped by the registration center and synchronized by a data synchronization tool in the monitoring data acquisition module, wherein the variable data comprises instruction operation information and a to-be-processed monitoring collector identifier.

[0045] The processing module is used to determine whether the instruction operation information in the changed data information is a newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the monitoring collector corresponding to the monitoring collector to be processed, and updates the attribute information of the monitoring collector to be processed or the attribute information of the target monitoring server in the pre-grouping of the registration center.

[0046] The processing module is further configured to trigger the monitoring and acquisition executor, so that the monitoring and acquisition executor generates configuration information recognizable by the target monitoring server for the monitoring and acquisition device to be processed according to the attribute information of the monitoring and acquisition device to be processed or the target monitoring server, and the configuration parameters of the configuration module center of the monitoring scheduling module, obtains the key-value pair relationship between the monitoring and acquisition device to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the registration center running group to complete the monitoring and acquisition scheduling processing.

[0047] Fourthly, this application provides a monitoring and acquisition scheduler, including: a processor, and a memory communicatively connected to the processor;

[0048] The memory stores computer-executed instructions;

[0049] The processor executes computer execution instructions stored in the memory to implement the method as described in the preceding one.

[0050] Fifthly, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the method described in any of the preceding claims.

[0051] This application provides a processing method, system, and medium for a data center adaptive monitoring and acquisition system. The processing method is applied to the data center adaptive monitoring and acquisition system, which includes a monitoring data acquisition module, a monitoring scheduling module, and a monitoring acquisition module. The monitoring scheduling module includes a monitoring acquisition scheduler, a monitoring acquisition executor, and a registration center. The processing method includes: the monitoring acquisition scheduler acquiring pre-grouped change data information synchronized to the registration center by a data synchronization tool in the monitoring data acquisition module; the change data including: instruction operation information and a monitoring acquisition device identifier to be processed; the monitoring acquisition scheduler determining whether the instruction operation information in the change data information is newly added instruction operation information; if so, using the KNN scheduling algorithm to process the newly added instruction operation information. The monitoring and data acquisition scheduler identifies the monitoring and data acquisition device to be processed and assigns a target monitoring server to it. It then updates the attribute information of the monitoring and data acquisition device to be processed or the attribute information of the target monitoring server in the pre-grouping of the registration center. The monitoring and data acquisition scheduler triggers the monitoring and data acquisition executor, which generates configuration information recognizable by the target monitoring server for the monitoring and data acquisition device according to the attribute information of the monitoring and data acquisition device to be processed or the target monitoring server, and the configuration parameters of the configuration module center of the monitoring and scheduling module. It obtains the key-value pair relationship between the monitoring and data acquisition device to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the running group of the registration center to complete the monitoring and data acquisition scheduling process. Compared with the prior art, this application, when real-time monitoring detects new instruction operation information in the registration center, uses the KNN scheduling algorithm to reasonably and efficiently determine whether to assign a new monitoring server to the newly added monitoring and data acquisition device. If not, it selects the monitoring server with the highest priority as the server bound to the newly added monitoring and data acquisition device according to the KNN scheduling algorithm. This avoids the need for manual modification of the mapping table between monitoring collectors and monitoring servers every time a new monitoring collector is added, improving processing efficiency. At the same time, it provides a reasonable method for assigning monitoring servers to new monitoring collectors, avoiding the occurrence of associated host crashes due to overload of monitoring server collection volume, thereby improving the stability of the data center. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1A schematic diagram of the structure of a data center adaptive monitoring and acquisition system provided in this application is as follows:

[0054] Figure 2 A schematic diagram of the processing method flow of a data center adaptive monitoring and acquisition system provided in this application;

[0055] Figure 3 A schematic diagram of the processing method flow of a data center adaptive monitoring and acquisition system provided in this application;

[0056] Figure 4 A schematic diagram of the processing method flow of a data center adaptive monitoring and acquisition system provided in this application;

[0057] Figure 5 A schematic diagram of the processing method flow of a data center adaptive monitoring and acquisition system provided in this application;

[0058] Figure 6 This is a schematic diagram of a monitoring and data acquisition scheduler provided in this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments made by those skilled in the art under the guidance of these embodiments are within the scope of protection of this application.

[0060] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0061] As a key component of "new infrastructure," data centers house the most core and critical assets of information systems across various industries, serving as the cornerstone and engine of the digital economy. Therefore, the reliability of data centers is paramount. Real-time monitoring of data center operational status and the generation and dissemination of anomaly alarms are crucial for ensuring their stable operation.

[0062] In existing technologies, during the operation and maintenance of data centers, technicians need to establish matching rules between monitoring collectors and monitoring servers in advance based on the needs of business systems. This is so that when the data collected by the connected monitoring collectors exceeds the carrying capacity of the monitoring server, the monitoring collection scheduler can perform scheduling processing according to the matching rules to maintain the stability of the data center.

[0063] However, with existing technologies, when the services hosted by the data center become more complex, the pre-established matching rule tables cannot meet the current business needs. This requires shuffling the existing matching rules and reallocating them, leading to increased monitoring system maintenance costs and slower processing efficiency. Furthermore, the monitoring server cannot automatically adjust resources based on the allocated monitoring collectors, which can easily result in resource waste or server crashes, leading to poor data center stability. In addition, since the matching rules for the monitoring server and monitoring collectors are currently configured on the monitoring server itself and are not managed uniformly, the topology diagram of the monitoring server and monitoring collectors cannot be automatically generated, requiring manual maintenance. When a host hosting a monitoring server goes down, it is impossible to quickly locate the affected monitoring services, thus impacting the stability of the data center.

[0064] Based on the above-mentioned technical problems, the technical concept of this application is: how to realize a data center adaptive monitoring and acquisition system that can improve both the stability of the data center and the scheduling and processing efficiency and flexibility of the data center.

[0065] Figure 1 This application provides a schematic diagram of the structure of a data center adaptive monitoring and acquisition system, as shown below. Figure 1 As shown, the system 1 includes: a monitoring data acquisition module 101, a monitoring scheduling module 102, and a monitoring acquisition module 103.

[0066] The monitoring data acquisition module 101 includes a monitoring configuration management library 1011 and a data synchronization tool. The monitoring configuration management library is used to store the configuration information of the data center monitoring collector. The data synchronization tool is used to subscribe to the monitoring configuration management library change data and automatically synchronize it to the registration center 1021 of the monitoring scheduling module 102 so that the monitoring scheduler 1022 can perform scheduling processing.

[0067] The monitoring and scheduling module 102 includes: a registration center 1021, used to store a key-value pair database of monitoring collectors and monitoring servers, as well as runtime information and attribute information of monitoring collectors and monitoring servers; a monitoring collection invoker 1022, used to subscribe to the changed data of the registration center 1021, and based on the runtime data of the monitoring collectors and monitoring servers, to complete the automatic allocation and scheduling of monitoring collectors using a weighted KNN scheduling algorithm, realizing the automatic creation and scaling scheduling of monitoring servers; a preprocessing probe 1023, used to subscribe to the changed data of the registration center 1021, pre-detect the amount of data collected by newly added (not yet connected) monitoring collectors, and update the runtime information of newly added monitoring collectors in the registration center 1021; and a real-time probe 1024, used to periodically detect runtime information such as the resource usage of the monitoring server and the amount of monitoring data collected, and update the registration center. 1021 can also be used to periodically detect the data volume of connected monitoring collectors and update the registration center 1021; the console interface 1025 is used to configure parameters such as the maximum collection volume and startup template of the monitoring server, as well as the user console interface for configuring the central feature value of the weighted KNN scheduling algorithm, and the interface for displaying the topology diagram of the host, monitoring server and monitoring collector in real time; the configuration center 1026 is used to store the key-value pair library of the monitoring collector and the monitoring server, as well as to store configuration information such as the maximum collection volume and configuration template of the monitoring server, and to store configuration such as the central feature value of the weighted KNN scheduling algorithm; the monitoring collection executor 1027 is used to generate configuration files of monitoring collectors that can be recognized by the monitoring server based on the runtime data of the registration center 1021 and the template of the configuration center 1026, or to create new monitoring servers or scale up or down the old ones.

[0068] The monitoring and acquisition module 103 includes a monitoring router 1031, which can be used to associate with multiple monitoring servers and provide a unified entry point for monitoring data query. To ensure high availability, the monitoring router is configured in a dual-machine cluster system mode (Highly Available, HA), for example... Figure 1 10311 and 1032 are two nodes in HA mode, connected to the same monitoring server, such as... Figure 1The monitoring server 10311 uses a solid line connection, the same as the monitoring server 10312 which uses a dashed line connection, to ensure that if one router fails, the other router can perform the configuration work. A virtual IP address (VIP) provides a unified router query entry point, facilitating downstream systems to query monitoring data. Monitoring collector 1032 specifically refers to an application that provides monitoring sample data for monitoring data samples. Monitoring server 1033 periodically retrieves and stores monitoring data from the configured data source of monitoring collector 1032. Furthermore, to ensure high availability of the monitoring acquisition system, the monitoring server is also configured in HA mode, such as... Figure 1 Monitoring servers 10331 and 10332 are two nodes in HA mode, and they are configured with the same monitoring collector to ensure that if one node fails, the other node can continue to provide monitoring services. Figure 1 The monitoring data acquisition unit 10331 uses a solid line connection, which is the same as the monitoring data acquisition unit 10332 uses a dashed line connection.

[0069] The technical solution of this application will now be described in detail through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0070] Figure 2 This application provides a schematic diagram of a processing method for an adaptive monitoring and acquisition system for a data center, as shown below. Figure 2 As shown, the method includes:

[0071] Step 201: The monitoring and acquisition scheduler obtains the changed data information synchronized to the registration center by the data synchronization tool in the monitoring data acquisition module. The changed data includes: instruction operation information and the identifier of the monitoring acquisition device to be processed.

[0072] It should be noted that the processing method provided in this embodiment can be applied to, for example, Figure 1 The system shown is used to monitor the stability of the data center monitoring server and adjust the matching relationship between the monitoring server and the monitoring collector in real time according to changes in the complexity of data center business, so as to keep the data center in a stable state.

[0073] In this embodiment, the monitoring and acquisition scheduler monitors whether there is any changed data information in the registration center. Optionally, the changed data information in the registration center is obtained by the data synchronization tool in the monitoring data acquisition module, that is, the data synchronization tool synchronizes the information obtained from the configuration management library to the registration center. The registration center includes both real-time groups for storing the operation information of the monitoring collectors and the monitoring server, and pre-groups for storing pending monitoring collectors generated due to the complexity of system business, such as pending monitoring collectors generated due to increased business demand or pending monitoring collectors generated due to reduced business demand.

[0074] Correspondingly, the change data information of the pre-grouped registration center includes: instruction operation information and the identifier of the monitoring collector to be processed. Among them, the instruction operation information includes new instruction operation information and deleted instruction operation information.

[0075] Step 202: The monitoring and acquisition scheduler determines whether the instruction operation information in the changed data information is a newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the corresponding monitoring acquisition device to be processed, and updates the attribute information of the monitoring acquisition device to be processed or the attribute information of the target monitoring server in the pre-grouping of the registration center.

[0076] In this embodiment, the monitoring and data acquisition scheduler determines whether the received instruction operation information is a new instruction information, that is, whether there is a new business requirement in the current system. When there is a new business requirement in the system, such as collecting and monitoring a new type of data.

[0077] It should be noted that the monitoring and data acquisition scheduler has a pre-stored KNN scheduling algorithm, which was pre-established by relevant technical personnel in this field.

[0078] When a new instruction operation is detected, the monitoring and acquisition scheduler invokes its internal KNN scheduling algorithm to schedule and allocate the monitoring acquisition device to be processed. Specifically, the monitoring and acquisition scheduler first obtains the acquisition volume information of the monitoring acquisition device through a preprocessing probe, and combines this information with other attribute information of the monitoring acquisition device, such as level, system, and region, to generate a first feature vector. Similarly, it obtains the attribute information of each monitoring server from the running groups in the registry center, such as level, system, and region, and generates a second feature vector for each monitoring server. The KNN algorithm is then used to calculate the center vectors of the first and second feature vectors, thereby obtaining a weighted distance value. The target monitoring server is then assigned to the monitoring acquisition device to be processed based on this weighted distance value.

[0079] Accordingly, once the target monitoring server is assigned, the monitoring acquisition scheduler will write the target monitoring server's attribute information, such as its identifier, into the location information of the monitoring acquisition server's attribute information. At the same time, it will synchronize the target monitoring server's attribute information to the pre-grouping in the registration center, that is, update the location information in the attribute information of the server to be processed and synchronize the target server's attribute information, so as to facilitate subsequent scheduling and allocation processing.

[0080] Step 203: The monitoring and acquisition scheduler triggers the monitoring and acquisition executor so that the monitoring and acquisition executor generates new configuration information that can be recognized by the target monitoring server for the monitoring and acquisition device to be processed according to the attribute information of the monitoring and acquisition device to be processed or the target monitoring server and the configuration parameters of the configuration module center of the monitoring and scheduling module. It obtains the key-value pair relationship between the monitoring and acquisition device to be processed and the target monitoring server and synchronizes the key-value pair relationship to the running group in the registration center to complete the monitoring and acquisition scheduling process.

[0081] In this embodiment, once the monitoring and acquisition scheduler completes the scheduling and allocation process for the monitoring and acquisition collector to be processed, it can trigger the monitoring and acquisition executor to perform scheduling processing. Optionally, when the monitoring and acquisition executor detects a change in the value of the location field of the monitoring and acquisition collector to be processed, it generates a configuration file based on the configuration center template and registration center data and automatically synchronizes it to the target monitoring server. It then waits for the target monitoring server to complete the hot loading of the configuration, which generates new configuration information for the target monitoring server that can identify the monitoring and acquisition collector to be processed.

[0082] Then, the monitoring and data acquisition executor moves the monitoring and data acquisition devices to be processed in the registry center from the pre-group to the running group, and updates the attribute information of the target monitoring server in the running group of the registry center, thereby completing the automatic allocation and processing of the monitoring and data acquisition devices to be processed.

[0083] It should be noted that, in this embodiment, the running group of the registration center pre-stores key-value pairs between the monitoring collectors to be processed and the target monitoring server, as well as key-value pairs between other monitoring collectors and the target monitoring server. These key-value pairs represent the attribute information of the monitoring collectors and the monitoring server; each attribute is represented by a key, and each key has a corresponding value. Since the relationship between the monitoring collectors and the monitoring server is many-to-one, there may be multiple monitoring collectors whose location fields correspond to the same key value. The specific implementation of these key-value pairs includes, but is not limited to, methods set by those skilled in the art based on experience.

[0084] Accordingly, during allocation and scheduling, the monitoring and acquisition scheduler needs to synchronize the key-value pair relationship between the target monitoring server and the monitoring acquisition device to be processed to the running group in the registry center, so that the monitoring and acquisition executor can generate a configuration file based on the key-value pair relationship, thereby enabling the target monitoring server to complete the configuration hot loading and complete the configuration processing of the monitoring acquisition device to be processed.

[0085] In this embodiment, a processing method for a data center adaptive monitoring and acquisition system is provided. The method is applied to the data center adaptive monitoring and acquisition system, which includes a monitoring data acquisition module, a monitoring scheduling module, and a monitoring acquisition module. The monitoring scheduling module includes a monitoring acquisition scheduler, a monitoring acquisition executor, and a registration center. The processing method uses the monitoring acquisition scheduler to acquire pre-grouped change data information synchronized to the registration center by the data synchronization tool in the monitoring data acquisition module. The change data includes: instruction operation information and the identifier of the monitoring acquisition device to be processed. The monitoring acquisition scheduler determines whether the instruction operation information in the change data is newly added instruction operation information; if so, it uses the KNN scheduling algorithm to... The pending monitoring collector identifier assigns a target monitoring server to the corresponding pending monitoring collector, and updates the attribute information of the pending monitoring collector or the target monitoring server in the pre-group of the registration center. The monitoring collection scheduler triggers the monitoring collection executor so that the monitoring collection executor generates configuration information that can be recognized by the target monitoring server for the pending monitoring collector according to the attribute information of the pending monitoring collector or the target monitoring server and the configuration parameters of the configuration module center of the monitoring scheduling module. It obtains the key-value pair relationship between the pending monitoring collector and the target monitoring server, and synchronizes the key-value pair relationship to the running group of the registration center to complete the monitoring collection scheduling process.

[0086] Compared to existing technologies, the method provided in this embodiment can monitor the runtime information of the data center's monitoring collectors and monitoring servers in real time. It can also make reasonable adjustments to the association between the monitoring collectors and monitoring servers in real time when the business expands or shrinks, so as to maintain the stability of the data center without the need for manual re-changing of the matching relationship. In addition, the method of this embodiment avoids the monitoring server's inability to automatically adjust resources according to the allocated monitoring collectors, which can easily lead to resource waste or monitoring server crashes.

[0087] Figure 3 This is a schematic diagram of a processing method for a data center adaptive monitoring and acquisition system provided in this application. Based on the above embodiments, this embodiment specifically illustrates the specific processing method of step 202, which includes:

[0088] Step 301: The monitoring and data acquisition scheduler determines whether the instruction operation information in the changed data information is a newly added instruction operation information. If so, proceed to steps 302 to 305; otherwise, proceed to steps 308 to 3011.

[0089] Step 302: The monitoring and acquisition scheduler triggers the preprocessing probe to obtain the acquisition quantity information of the monitoring acquisition device to be processed, and combines it with the attribute information of the monitoring acquisition device to be processed to obtain the central feature value of the monitoring acquisition device to be processed and generate the first feature vector.

[0090] In this embodiment, after the monitoring and acquisition scheduler determines that there is a new instruction operation, it will use the KNN scheduling algorithm to assign a target monitoring server to the monitoring and acquisition device to be processed.

[0091] First, the monitoring and acquisition scheduler will immediately trigger the preprocessing probe to detect and process the acquisition amount information of the monitoring acquisition device to be processed, so as to obtain the acquisition information of the monitoring acquisition device to be processed, that is, the amount of information data that the monitoring acquisition device to be processed needs to monitor and collect. Based on the acquisition information and other attribute information of the monitoring acquisition device to be processed, such as the central feature value, which includes information such as level, system and region, a 1*4 first feature vector is generated.

[0092] The central feature value can be customized based on the service type provided by the data center. For example, the monitoring server and the monitoring collector to be processed have three default attributes, such as level, system, and region. The central feature value can also be customized based on the subdivided service type, such as the business module to which it belongs. For example, the monitoring server has one default central feature value based on its runtime parameters, such as the remaining monitoring collection volume as a custom central feature value. The monitoring collector also has one default central feature value based on its runtime parameters, such as the monitoring collection volume as a custom central feature value.

[0093] For ease of explanation, optionally, unchanging attributes can be treated as static attributes and variable attributes as dynamic attributes. Based on this, the first feature vector in this embodiment can include, in order of static first and dynamic later, the following: level, system, region and collection volume.

[0094] Step 303: The monitoring and acquisition scheduler obtains the central feature value of the monitoring server from the registration center and generates the second feature vector.

[0095] Accordingly, in order to ensure the rationality of scheduling, the monitoring and acquisition scheduler needs to obtain the second feature vector of each monitoring server in the registration center running group, which is similar to the first feature vector of the monitoring acquisition device to be processed. The second feature vector includes the following in the order of static first and dynamic: level, system, region and remaining acquisition volume.

[0096] Step 304: The monitoring and acquisition scheduler calculates the center distance between the first feature vector and the second feature vector respectively, obtains the center distance vector, and calculates the product of the pre-configured weight vector and the center distance vector to obtain the weighted distance value.

[0097] In this embodiment, the monitoring and acquisition scheduler needs to obtain the center distance between the first feature vector and each of the second feature vectors. This center distance refers to the distance from the center feature value of the monitoring acquisition device to be processed to the center feature value of the monitoring server. For static attributes, the center distance can be characterized by whether the center feature value of the monitoring acquisition device to be processed matches the center feature value of the monitoring server. For example, if the center feature value of the monitoring acquisition device to be processed matches the center feature value of the monitoring server, the distance is defined as 1; otherwise, the distance is defined as 1. For dynamic attributes, the center distance can be characterized by calculating the difference between the remaining acquisition amount of the monitoring server and the acquisition amount of the monitoring acquisition device to be processed. For the center distance of dynamic attributes, the corresponding results can be arranged in descending order, and the sorting order can represent the ascending order of priority of each monitoring server.

[0098] After obtaining the center distance between the first feature vector and the second feature vector, the monitoring and acquisition scheduler can obtain the corresponding center distance vector, such as (1,1,1,0.2), based on the center distance of the static attribute and the center distance of the dynamic attribute.

[0099] The monitoring and data acquisition scheduler pre-stores a weight vector, which characterizes the magnitude of the influence factors on the distance to each center. This weight vector is set by those skilled in the art based on experience. Optionally, it can be set according to the rule that the higher the priority of the center feature value, the smaller the weight value. For example, for four center feature values, a 4-element weight vector needs to be created.

[0100] Considering that weighting different center distances can improve classification accuracy, the monitoring and data acquisition scheduler also needs to calculate the product of the center distance vector and the weight vector to obtain the weighted distance between the monitoring data acquisition device to be processed and each monitoring server.

[0101] Step 305: The monitoring and data collection scheduler determines whether the weighted distance value is greater than the pre-configured weighted distance threshold. If yes, proceed to step 306; otherwise, proceed to step 307.

[0102] In this embodiment, the monitoring and data acquisition scheduler determines whether to re-establish the monitoring server for a data acquisition device to be processed based on the obtained weighted distance value.

[0103] Optionally, the monitoring and acquisition scheduler has a pre-configured weighted distance threshold, and the pre-configured weighted distance threshold changes with the changes in system services. The specific implementation of the pre-configured weighted distance threshold includes, but is not limited to, being set by those skilled in the art based on the actual system services.

[0104] When the weighted distance is greater than the pre-configured threshold, it indicates that the remaining collection volume of the existing monitoring server is insufficient to meet the collection volume of the monitoring collector to be processed, and step 306 can be executed; otherwise, it indicates that the remaining collection volume of the existing monitoring server is sufficient to meet the collection volume of the monitoring collector to be processed, and step 307 can be executed.

[0105] Step 306: The monitoring and data acquisition scheduler creates the attribute information of the target monitoring server in the pre-grouping of the registration center.

[0106] In this embodiment, the monitoring and acquisition scheduler creates key-value pairs for the target monitoring server in the pre-grouping of the registration center, which contain the attribute information of the target monitoring server, such as level, system, and region information. When the triggered monitoring and acquisition executor runs the newly created target server, the monitoring and acquisition scheduling device jumps back to steps 302 to 305 to continue the automatic allocation and processing of the monitoring and acquisition server to be processed.

[0107] Step 307: The monitoring and data collection scheduler selects the monitoring server corresponding to the highest priority weighted distance value in the registration center as the target monitoring server, and updates the attribute information of the monitoring data collector to be processed according to the attribute information of the target monitoring server.

[0108] In this embodiment, the monitoring and data acquisition scheduler needs to select the monitoring server with the highest priority from the existing monitoring servers. For example, when the center distances of static attributes are all the same, the monitoring server with the largest weighted distance value and the largest center distance of the dynamic attribute corresponding to the largest weighted distance can be selected as the target server.

[0109] Specifically, the monitoring acquisition retrieval device updates the identification information of the target monitoring server into the location attribute information of the monitoring acquisition device, and triggers the monitoring acquisition executor to generate a configuration file that can be recognized by the target monitoring server based on the attribute information of the monitoring acquisition device to be processed and the configuration information of the configuration center. The configuration file information is then synchronized to the target monitoring server so that the target monitoring server can perform hot loading of the configuration file to complete the system addition operation of the monitoring acquisition device to be processed.

[0110] Step 308: If the monitoring and acquisition scheduler determines that the operation instruction information in the changed data information is a deletion operation instruction information, it triggers the monitoring and acquisition executor to determine the target attribute information corresponding to the monitoring and acquisition identifier to be processed in the registration center running group according to the identifier of the monitoring and acquisition device to be processed.

[0111] In this embodiment, when the monitoring and data acquisition scheduler detects a delete instruction, it may indicate a reduction in business demand or that the current data collection volume of the pending monitoring data acquisition device has reached its limit. During the delete instruction processing, the monitoring and data acquisition executor is triggered to retrieve the target attribute information of the monitoring data acquisition device corresponding to the pending monitoring data acquisition device identifier from the registry's running group. This information may include details such as level, system, and region.

[0112] Step 309: The monitoring and acquisition scheduler triggers the monitoring and acquisition executor to generate deletion configuration information related to the monitoring and acquisition device to be processed based on the target attribute information, and synchronizes the deletion configuration information to the monitoring server that is bound to the identifier of the monitoring and acquisition device to be processed.

[0113] Specifically, the monitoring and acquisition scheduler triggers the monitoring and acquisition executor to obtain the configuration information from the configuration center and the target attribute information of the monitoring and acquisition device to be processed, and generates deletion configuration information related to the monitoring and acquisition device to be processed. In order for the monitoring server to respond to the deletion command operation information in a timely manner, the monitoring and acquisition executor synchronizes the generated deletion configuration information to the monitoring server that is bound to the identifier of the monitoring and acquisition device to be processed.

[0114] Step 3010: The monitoring and data acquisition scheduler triggers the automatic hot-loading and deletion of configuration information on the monitoring server that is bound to the identifier of the monitoring data acquisition to be processed, so as to complete the deletion processing operation of the monitoring data acquisition to be processed.

[0115] Specifically, the monitoring server that is bound to the monitoring collection identifier to be processed will immediately start hot reloading after receiving the deletion configuration information, that is, it will execute the deletion process immediately after receiving the deletion configuration information.

[0116] Step 3011: When the monitoring and acquisition scheduler determines that the monitoring server that is bound to the identifier of the monitoring and acquisition device to be processed has completed the deletion operation within a preset time, it triggers the monitoring and acquisition executor to perform the deletion operation on the monitoring and acquisition device to be processed in the running group of the registration center.

[0117] Specifically, to prevent the monitoring server from crashing, the monitoring collection scheduler needs to determine whether the monitoring server has completed the deletion operation within a preset time. If so, the information synchronization operation can be performed, that is, the monitoring collection executor needs to be triggered to delete the monitoring collectors to be processed in the running group of the registration center. Otherwise, an information operation error message is generated and the message is fed back to the console interface.

[0118] It should be noted that when adding or deleting command information, the monitoring and data acquisition executor needs to check whether the target monitoring server has completed the configuration update process within a preset time to prevent downtime or endless waiting.

[0119] Correspondingly, when the target monitoring server completes the addition or processing operation within the preset time, the monitoring acquisition executor will transfer the monitoring acquisition devices to be processed from the pre-grouping in the registration center to the running group, and update the attribute information of the target monitoring server.

[0120] In an optional embodiment, when the acquired instruction operation information is neither a new instruction operation information nor a delete instruction operation information, the monitoring and acquisition scheduler needs to generate an error message for the operation information and feed the message back to the console interface. At the same time, the change data of the monitoring acquisition device to be processed where the instruction operation information is located is deleted.

[0121] This embodiment specifically illustrates several possible scenarios for instruction operation information in changing data, as well as the specific processing methods under different scenarios. This improves the flexibility of the data center monitoring system and is more conducive to maintaining the stability of the data center.

[0122] Figure 4 This diagram illustrates the processing flow of a data center adaptive monitoring and acquisition system provided in this application. Considering that as business changes occur, connected monitoring acquisition devices may need to be taken offline, leaving some resources idle on the monitoring server. If these resources are pre-occupied without being used for an extended period, it results in resource waste. Therefore, it is necessary to automatically scale down the connected monitoring server. Figure 4 As shown, the automatic scaling down process includes:

[0123] Step 401: The monitoring and data collection scheduler triggers the monitoring and data collection executor to obtain the average daily resource utilization rate of each monitoring server at the calibrated time.

[0124] In this embodiment, the monitoring and data collection scheduler triggers the monitoring and data collection executor to obtain the average daily resource utilization rate of each server in the registration center running group in real time or periodically. Optionally, the monitoring and data collection executor calculates the average daily resource utilization rate at 0:00 every day based on the monitoring data collected by the monitoring server and writes it into the resource field of the registration center monitoring server attribute information.

[0125] Step 402: The data collection scheduler triggers the monitoring data collection executor to determine whether the daily average resource utilization rate meets the conditions for automatic scaling down.

[0126] In this embodiment, the monitoring and data acquisition scheduler pre-stores a pre-configured low-load threshold. This low-load threshold is used to determine whether the monitoring server is in a resource-idle state. The specific implementation of the pre-configured low-load threshold includes, but is not limited to, values ​​set by those skilled in the art based on experience. Furthermore, the monitoring and data acquisition scheduler also stores a pre-configured number of low-load days, which serves as another criterion for determining whether the monitoring server is in a resource-idle state. This avoids misjudgments due to resource limitations at a certain moment. This number of low-load days is set by those skilled in the art based on the actual business scenario, but is not limited to other limiting methods.

[0127] Optionally, the automatic scaling-down processing conditions in this embodiment may be: the average daily resource utilization rate is less than the pre-configured low load threshold, and the number of consecutive days that the monitoring server corresponding to the average daily resource utilization rate is less than the pre-configured low load threshold is greater than the pre-configured low load days.

[0128] Specifically, the triggered monitoring and data collection executor also needs to determine whether the average daily resource utilization of each monitoring server meets the conditions for automatic scaling down, so as to determine whether the current monitoring server needs to perform automatic scaling down.

[0129] Step 403: If the conditions for automatic scaling down are met, the monitoring and data acquisition scheduler triggers the monitoring and data acquisition executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the daily average resource utilization rate that meets the conditions for automatic scaling down, so as to complete the automatic scaling down process.

[0130] Specifically, if the current monitoring server is determined to meet the conditions for automatic scaling down, the monitoring collector triggers the monitoring collection executor to adjust the pre-allocated resource configuration of the monitoring server and restarts the monitoring server. After the monitoring server successfully restarts, its low load field is set to 0, and the process jumps to step 401 to restart the loop. If the restart fails, an error message is returned, notifying manual intervention.

[0131] In an optional embodiment, if there is a situation where the average daily resource utilization rate is not less than the pre-configured low load threshold, the monitoring collector triggers the monitoring collection executor to adjust the pre-occupied resource configuration of the monitoring server and restarts the monitoring server. After the monitoring server is successfully restarted, its low load field is set to 0, and the process jumps to step 401 to repeat the loop.

[0132] In another optional embodiment, if there is a situation where the average daily resource utilization rate is not less than the pre-configured low load threshold, and the number of consecutive days that the monitoring server corresponding to the average daily resource utilization rate is less than the pre-configured low load threshold is less than the pre-configured low load days, then the process jumps to step 401 for looping.

[0133] This embodiment specifically illustrates the handling of situations where monitoring server resources are idle, thus avoiding resource waste and adjusting the configuration of data center resources in a timely manner to achieve optimal resource scheduling, thereby improving the stability of the data center.

[0134] Figure 5 This diagram illustrates the processing flow of a data center adaptive monitoring and acquisition system provided in this application. Considering that as business volume increases, the data collection volume of the connected monitoring collectors may also increase, even exceeding the capacity of the monitoring server, leading to instability in the monitoring server's operation. To ensure the stable and reliable operation of the monitoring server and improve system resource utilization, automatic expansion of the existing monitoring server is necessary. For example... Figure 5 As shown, the automatic capacity expansion process includes:

[0135] Step 501: If the conditions for automatic scaling down are not met, and the average daily resource utilization rate is greater than the pre-configured high load threshold, the monitoring and acquisition scheduler triggers the monitoring and acquisition executor to perform the corresponding automatic scaling up process based on the remaining acquisition quota in the attribute information of the monitoring server corresponding to the average daily resource utilization rate.

[0136] In this embodiment, the monitoring and data collection scheduler also has a pre-configured high load threshold, which is used to determine whether the monitoring server is experiencing an overload of data collection. This high load threshold is set by those skilled in the art based on the actual business scenario, but is not limited to other specific implementation methods.

[0137] When the monitoring and data collection scheduler determines that the current monitoring server does not meet the conditions for automatic scaling down, and determines that the average daily resource utilization rate of the current monitoring server is greater than the pre-configured high load threshold, it can be preliminarily determined that the current monitoring server needs to be scaled up. However, the specific subsequent processing depends on the remaining data collection quota of the monitoring server. The specific processing steps are as follows.

[0138] Step 502: Determine if the remaining collection quota is greater than 0. If yes, proceed to step 503; otherwise, proceed to steps 504 to 505.

[0139] The monitoring and data acquisition scheduler needs to determine whether the remaining data acquisition quota of the current monitoring server is greater than 0, that is, whether the current monitoring server is fully loaded or overloaded. If it is not fully loaded or overloaded, proceed to step 503; if it is overloaded or fully loaded, proceed to steps 504 to 505.

[0140] Step 503: The monitoring and data collection scheduler triggers the monitoring and data collection executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the daily average resource utilization rate in order to complete the automatic expansion process.

[0141] Specifically, the monitoring data acquisition executor adjusts the pre-allocated resource configuration of the monitoring server and restarts the monitoring server. After a successful restart, the high load field is set to 0, and the process proceeds to step 501 to repeat the loop. If the restart fails, an error message is returned, notifying manual intervention.

[0142] Step 504: The monitoring and data collection scheduler triggers the monitoring and data collection executor to adjust the field value of the operation field in the attribute information of the monitoring and data collection device corresponding to the daily average resource utilization rate to clear, and performs the clearing process according to the field value of the operation field to obtain the monitoring and data collection device after clearing.

[0143] Step 505: The monitoring and acquisition scheduler triggers the monitoring and acquisition executor to synchronize the cleared attribute information of the monitoring and acquisition device to the pre-grouping of the registration center.

[0144] In this embodiment, when the monitoring and data collection scheduler determines that the remaining data collection quota of the monitoring server is less than or equal to 0, a new monitoring server needs to be created to share the data collection pressure. This will involve adjusting the automatic balancing of the monitoring data collectors.

[0145] Specifically, the monitoring data acquisition executor identifies the monitoring data acquisition devices corresponding to monitoring servers with a data acquisition quota of 0 and designates them as monitoring data acquisition devices to be migrated. Simultaneously, it clears the values ​​of the operation field in the attribute information of the monitoring data acquisition devices to be migrated within the registry's running group.

[0146] Accordingly, the monitoring and data acquisition actuator synchronizes the attribute information of the monitoring and data acquisition device to be migrated to the registration center for pre-grouping, and automatically triggers the operation process for adding a new monitoring and data acquisition device. The specific implementation method can be as described in the aforementioned embodiment.

[0147] Then, the monitoring and data acquisition scheduler determines whether the monitoring data acquisition device has completed the clearing process. If so, it unbinds the monitoring data acquisition device to be migrated in the monitoring server to complete the legacy data cleanup operation, and then transfers the processing steps to step 501 for reprocessing. If the clearing operation is not completed normally, an error message is returned, notifying manual intervention.

[0148] In one optional embodiment, after completing the scheduling of the monitoring collectors, a corresponding topology diagram can be generated to facilitate subsequent maintenance by technical personnel. Specifically, the monitoring collector scheduler displays the associated topology diagram of the running host, monitoring server, and monitoring collector on the console interface of the monitoring scheduling module based on the attribute information of all monitoring collectors and monitoring servers in the registration center, as well as the association information of each monitoring server and the running host that carries the monitoring server.

[0149] In this optional embodiment, the registration center contains information on all monitoring servers and connected monitoring collectors, as well as the association information between the monitoring servers and the hosts running the monitoring servers. Based on this data, the operating status of the hosts, monitoring servers, and monitoring collectors, as well as the association topology diagram of the hosts, monitoring servers, and monitoring collectors, can be displayed in real time on the console interface. The topology diagram file can also be exported as needed so that technicians can clearly see the current monitoring and collection status of the data center.

[0150] This embodiment specifically illustrates the handling method when the monitoring server's data collection volume is overloaded or close to full load, as well as the topology map generation method provided to ensure efficient subsequent maintenance of the data center, which improves the stability of the data center.

[0151] Optionally, this application also provides a data center adaptive monitoring and acquisition system, which includes a monitoring data acquisition module, a monitoring scheduling module, and a monitoring acquisition module, wherein the monitoring scheduling module includes a monitoring acquisition scheduler, a monitoring acquisition executor, and a registration center.

[0152] The monitoring and acquisition scheduler is used to acquire the change data information synchronized to the registration center by the data synchronization tool in the monitoring data acquisition module. The change data includes: instruction operation information and the identifier of the monitoring acquisition device to be processed.

[0153] The monitoring and acquisition scheduler is also used to determine whether the instruction operation information in the changed data information is a newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the corresponding monitoring acquisition device to be processed, and updates the attribute information of the monitoring acquisition device to be processed or the attribute information of the target monitoring server in the pre-grouping of the registration center.

[0154] The monitoring and data acquisition scheduler is also used to trigger the monitoring and data acquisition executor, so that the monitoring and data acquisition executor generates configuration information that can be recognized by the target monitoring server for the monitoring and data acquisition device to be processed according to the attribute information of the monitoring and data acquisition device to be processed or the target monitoring server, and the configuration parameters of the configuration module center of the monitoring and scheduling module. It obtains the key-value pair relationship between the monitoring and data acquisition device to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the running group in the registration center to complete the monitoring and data acquisition scheduling process.

[0155] The implementation principle and beneficial effects of this optional embodiment are the same as those of the foregoing embodiments, and will not be repeated here.

[0156] Figure 6 This is a schematic diagram of a monitoring and data acquisition scheduler provided in this application. For ease of explanation, this embodiment only illustrates the structural parts related to this application, such as... Figure 6 As shown, the monitoring and acquisition scheduler 60 includes an acquisition module 601 and a processing module 602.

[0157] The acquisition module 601 is used to acquire the changed data information synchronized to the pre-group of the registration center by the data synchronization tool in the monitoring data acquisition module. The changed data includes: instruction operation information and the identifier of the monitoring collector to be processed. The processing module 602 is used to determine whether the instruction operation information in the changed data information is newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the monitoring collector to be processed corresponding to the identifier of the monitoring collector to be processed, and updates the attribute information of the monitoring collector to be processed or the attribute information of the target monitoring server in the pre-group of the registration center. The processing module 602 is also used to trigger the monitoring acquisition executor so that the monitoring acquisition executor generates configuration information that can be recognized by the target monitoring server for the monitoring collector to be processed according to the attribute information of the monitoring collector to be processed or the target monitoring server, and the configuration parameters of the configuration module center of the monitoring scheduling module. It acquires the key-value pair relationship between the monitoring collector to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the running group of the registration center to complete the monitoring acquisition scheduling process.

[0158] Optionally, processing module 602 is specifically used for:

[0159] The monitoring and acquisition scheduler triggers a preprocessing probe to obtain the acquisition volume information of the monitoring acquisition device to be processed, and combines it with the attribute information of the monitoring acquisition device to obtain the central feature value of the monitoring acquisition device to be processed and generate the first feature vector.

[0160] The monitoring and data acquisition scheduler obtains the central feature value of the monitoring server from the registration center and generates a second feature vector;

[0161] The monitoring and acquisition scheduler calculates the center distance between the first feature vector and the second feature vector respectively, obtains the center distance vector, and calculates the product of the pre-configured weight vector and the center distance vector to obtain the weighted distance value;

[0162] The monitoring and data acquisition scheduler determines if the weighted distance value is greater than the pre-configured weighted distance threshold. If it is, it creates the attribute information of the target monitoring server in the pre-group of the registration center.

[0163] Optionally, the processing module 602 is also specifically used for:

[0164] If the monitoring and data collection scheduler determines that the weighted distance value is less than the pre-configured weighted distance threshold, it selects the monitoring server corresponding to the highest priority weighted distance value from the registry center as the target monitoring server.

[0165] The monitoring and data acquisition scheduler updates the attribute information of the monitoring data acquisition devices to be processed based on the attribute information of the target monitoring server.

[0166] Optionally, the processing module 602 is also specifically used for:

[0167] If the monitoring and acquisition scheduler determines that the operation instruction information in the changed data information is a deletion operation instruction information, it will trigger the monitoring and acquisition executor to determine the target attribute information corresponding to the monitoring and acquisition identifier to be processed in the registration center running group according to the identifier of the monitoring and acquisition device to be processed.

[0168] The monitoring and acquisition scheduler triggers the monitoring and acquisition executor to generate a deletion configuration information file related to the monitoring and acquisition device to be processed based on the target attribute information, and synchronizes the deletion configuration information file to the monitoring server that is bound to the identifier of the monitoring and acquisition device to be processed;

[0169] The monitoring and data acquisition scheduler triggers the monitoring server that has a binding relationship with the identifier of the monitoring data acquisition to be processed to automatically hot-load and delete the configuration information file, so as to complete the deletion operation of the monitoring data acquisition to be processed;

[0170] When the monitoring and data collection scheduler determines that a monitoring server bound to the identifier of the monitoring data collector to be processed completes the deletion operation within a preset time, it triggers the monitoring and data collection executor to perform the deletion operation on the monitoring data collector to be processed in the running group of the registration center.

[0171] Optionally, the processing module 602 is also specifically used for:

[0172] The monitoring and data collection scheduler triggers the monitoring and data collection executor to obtain the average daily resource utilization rate of each monitoring server at a specified time.

[0173] The monitoring and data collection scheduler triggers the monitoring and data collection executor to determine whether the daily average resource utilization rate meets the conditions for automatic scaling down.

[0174] If the conditions for automatic scaling down are met, the monitoring and data acquisition scheduler will trigger the monitoring and data acquisition executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the average daily resource utilization rate that meets the conditions for automatic scaling down, so as to complete the automatic scaling down process.

[0175] Optional conditions for automatic scaling down include:

[0176] The average daily resource utilization rate is less than the pre-configured low load threshold, and the number of consecutive days that the average daily resource utilization rate corresponds to the monitoring server being less than the pre-configured low load threshold is greater than the number of pre-configured low load days.

[0177] Optionally, the processing module 602 is also specifically used for:

[0178] If the conditions for automatic scaling down are not met, and the average daily resource utilization rate is greater than the pre-configured high load threshold, the monitoring and data collection scheduler will trigger the monitoring and data collection executor to perform the corresponding automatic scaling up process based on the remaining collection quota in the attribute information of the monitoring server corresponding to the average daily resource utilization rate.

[0179] Optionally, the processing module 602 is also specifically used for:

[0180] If the remaining collection quota is greater than zero, the monitoring collection scheduler will trigger the monitoring collection executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the daily average resource utilization rate in order to complete the automatic expansion process.

[0181] Optionally, the processing module 602 is also specifically used for:

[0182] If the remaining collection quota is less than zero, the monitoring collection scheduler will trigger the monitoring collection executor to adjust the field value of the operation field in the attribute information of the monitoring collector corresponding to the daily average resource utilization rate to clear, and perform the clearing process according to the field value of the operation field to obtain the monitoring collector after clearing.

[0183] The monitoring and data acquisition scheduler triggers the monitoring and data acquisition executor to synchronize the cleared attribute information of the monitoring and data acquisition device to the pre-grouping in the registration center.

[0184] Optionally, the processing module 602 is also specifically used for:

[0185] Based on the attribute information of all monitoring collectors and monitoring servers in the registration center, as well as the association information of each monitoring server and the running host that hosts the monitoring server, the monitoring collection scheduler displays the association topology diagram of the running host, monitoring server and monitoring collector on the console interface of the monitoring scheduling module.

[0186] The specific implementation principle and beneficial effects of the monitoring and data acquisition scheduler provided in this embodiment are similar to those in the previous embodiments, and will not be repeated here.

[0187] Optionally, this application also provides a monitoring and acquisition scheduler, including: a processor and a memory communicatively connected to the processor; wherein the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in any of the preceding embodiments.

[0188] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the technical solutions provided in any of the foregoing embodiments.

[0189] This application also provides a computer program product, including a computer program, which, when executed by a processor, is used to implement the technical solutions provided in any of the foregoing method embodiments.

[0190] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A processing method for a data center adaptive monitoring and acquisition system, characterized in that, The processing method is applied to a data center adaptive monitoring and acquisition system, which includes a monitoring data acquisition module, a monitoring scheduling module, and a monitoring acquisition module. The monitoring scheduling module includes a monitoring acquisition scheduler, a monitoring acquisition executor, a registration center, and a preprocessing probe. The processing method includes: The monitoring and acquisition scheduler acquires the changed data information synchronized to the pre-grouped registration center by the data synchronization tool in the monitoring data acquisition module. The changed data includes: instruction operation information and the identifier of the monitoring acquisition device to be processed. The monitoring and acquisition scheduler determines whether the instruction operation information in the changed data information is newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the monitoring collector to be processed, and updates the attribute information of the monitoring collector to be processed or the target monitoring server in the pre-grouping of the registration center. This includes: the monitoring and acquisition scheduler triggers the preprocessing probe to obtain the collection volume information of the monitoring collector to be processed, and combines it with the attribute information of the monitoring collector to obtain the central feature value of the monitoring collector to be processed and generate a first feature vector; the monitoring and acquisition scheduler obtains the central feature value of the monitoring server in the registration center and generates a second feature vector; the monitoring and acquisition scheduler calculates the central distance between the first feature vector and the second feature vector to obtain a central distance vector, and calculates the product of the pre-configured weight vector and the central distance vector to obtain a weighted distance value; the monitoring and acquisition scheduler determines that the weighted distance value is greater than the pre-configured weighted distance threshold. If it is greater, it creates the attribute information of the target monitoring server in the pre-grouping of the registration center. The monitoring and acquisition scheduler triggers the monitoring and acquisition executor, so that the monitoring and acquisition executor generates configuration information that can be recognized by the target monitoring server for the monitoring and acquisition device to be processed according to the attribute information of the monitoring and acquisition device to be processed or the target monitoring server, and the configuration parameters of the configuration module center of the monitoring and scheduling module, obtains the key-value pair relationship between the monitoring and acquisition device to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the running group in the registration center to complete the monitoring and acquisition scheduling process; The monitoring and data acquisition scheduler triggers the monitoring and data acquisition executor to acquire the average daily resource utilization rate of each monitoring server at a specified time. The monitoring and data acquisition scheduler then triggers the monitoring and data acquisition executor to determine whether the average daily resource utilization rate meets the conditions for automatic scaling down. If the conditions for automatic scaling down are met, the monitoring and data acquisition scheduler then triggers the monitoring and data acquisition executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the average daily resource utilization rate that meets the conditions for automatic scaling down, so as to complete the automatic scaling down process.

2. The method according to claim 1, characterized in that, Also includes: If the monitoring and data collection scheduler determines that the weighted distance value is less than the pre-configured weighted distance threshold, it selects the monitoring server corresponding to the highest priority weighted distance value in the registration center as the target monitoring server. The monitoring and data acquisition scheduler updates the attribute information of the monitoring data acquisition device to be processed based on the attribute information of the target monitoring server.

3. The method according to claim 1, characterized in that, Also includes: If the monitoring and acquisition scheduler determines that the instruction operation information in the changed data information is a deletion instruction operation information, it triggers the monitoring and acquisition executor to determine the target attribute information corresponding to the monitoring and acquisition identifier to be processed in the registration center running group according to the identifier of the monitoring and acquisition device to be processed; The monitoring and acquisition scheduler triggers the monitoring and acquisition executor to generate deletion configuration information related to the monitoring and acquisition device to be processed based on the target attribute information, and synchronizes the deletion configuration information to the monitoring server that is bound to the identifier of the monitoring and acquisition device to be processed; The monitoring and data acquisition scheduler triggers the monitoring server that is bound to the identifier of the monitoring data acquisition device to be processed to automatically hot-load the deletion configuration information in order to complete the deletion processing operation of the monitoring data acquisition device to be processed. When the monitoring and data collection scheduler determines that a monitoring server bound to the identifier of the monitoring data collector to be processed completes the deletion operation within a preset time, it triggers the monitoring and data collection executor to perform the deletion operation on the monitoring data collector to be processed in the registration center running group.

4. The method according to claim 1, characterized in that, The conditions for the automatic volume reduction process include: The average daily resource utilization rate is less than the pre-configured low load threshold, and the number of consecutive days that the monitoring server corresponding to the average daily resource utilization rate is less than the pre-configured low load threshold is greater than the pre-configured low load days.

5. The method according to claim 1, characterized in that, Also includes: If the conditions for automatic scaling down are not met, and the average daily resource utilization rate is greater than the pre-configured high load threshold, the monitoring and data collection scheduler triggers the monitoring and data collection executor to perform corresponding automatic scaling up based on the remaining data collection quota in the attribute information of the monitoring server corresponding to the average daily resource utilization rate.

6. The method according to claim 5, characterized in that, Also includes: If the remaining collection quota is greater than zero, the monitoring collection scheduler triggers the monitoring collection executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the average daily resource utilization rate, so as to complete the automatic expansion process.

7. The method according to claim 6, characterized in that, Also includes: If the remaining collection quota is less than zero, the monitoring collection scheduler triggers the monitoring collection executor to adjust the field value of the operation field in the attribute information of the monitoring collector corresponding to the daily average resource utilization rate to clear, and performs the clearing process according to the field value of the operation field to obtain the monitoring collector after clearing. The monitoring and acquisition scheduler triggers the monitoring and acquisition executor to synchronize the attribute information of the cleared monitoring and acquisition device to the pre-grouping of the registration center.

8. The method according to claim 1, characterized in that, Also includes: The monitoring and data acquisition scheduler displays the association topology diagram of the running host, the monitoring server, and the monitoring data acquisition device on the console interface of the monitoring scheduling module, based on the attribute information of all monitoring data acquisition devices and monitoring servers in the registration center, as well as the association information of each monitoring server and the running host that hosts the monitoring server.

9. A data center adaptive monitoring and acquisition system, characterized in that, The monitoring and acquisition system includes: a monitoring data acquisition module, a monitoring scheduling module, and a monitoring acquisition module. The monitoring scheduling module includes a monitoring acquisition scheduler, a monitoring acquisition executor, a registration center, and a preprocessing probe. The monitoring and acquisition scheduler is used to acquire change data information synchronized to the pre-grouped registration center by the data synchronization tool in the monitoring data acquisition module. The change data includes: instruction operation information and the identifier of the monitoring acquisition device to be processed. The monitoring and acquisition scheduler is further configured to determine whether the instruction operation information in the changed data information is newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the monitoring collector to be processed, and updates the attribute information of the monitoring collector to be processed or the target monitoring server in the pre-grouping of the registration center. This includes: triggering the preprocessing probe to obtain the collection volume information of the monitoring collector to be processed, and combining it with the attribute information of the monitoring collector to be processed to obtain the central feature value of the monitoring collector to be processed and generate a first feature vector; obtaining the central feature value of the monitoring server in the registration center and generating a second feature vector; calculating the central distance between the first feature vector and the second feature vector to obtain a central distance vector, and calculating the product of the pre-configured weight vector and the central distance vector to obtain a weighted distance value; the monitoring and acquisition scheduler determines that the weighted distance value is greater than the pre-configured weighted distance threshold. If it is greater, it creates the attribute information of the target monitoring server in the pre-grouping of the registration center. The monitoring and acquisition scheduler is also used to trigger the monitoring and acquisition executor, so that the monitoring and acquisition executor generates configuration information that can be recognized by the target monitoring server for the monitoring and acquisition device to be processed according to the attribute information of the monitoring and acquisition device to be processed or the target monitoring server, and the configuration parameters of the configuration module center of the monitoring and scheduling module, obtains the key-value pair relationship between the monitoring and acquisition device to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the registration center running group to complete the monitoring and acquisition scheduling process; The monitoring and data acquisition scheduler is further configured to trigger the monitoring and data acquisition executor to acquire the average daily resource utilization rate of each monitoring server at a specified time; trigger the monitoring and data acquisition executor to determine whether the average daily resource utilization rate meets the conditions for automatic scaling down for each of the average daily resource utilization rates; if the conditions for automatic scaling down are met, trigger the monitoring and data acquisition executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the average daily resource utilization rate that meets the conditions for automatic scaling down, so as to complete the automatic scaling down process.

10. A monitoring and data acquisition scheduler, characterized in that, include: The acquisition module is used to acquire change data information synchronized to the pre-grouped registration center by the data synchronization tool in the monitoring data acquisition module. The change data includes: instruction operation information and the identifier of the monitoring collector to be processed. The processing module is used to determine whether the instruction operation information in the changed data information is newly added instruction operation information. If so, it uses the KNN scheduling algorithm to assign a target monitoring server to the monitoring collector to be processed, and updates the attribute information of the monitoring collector to be processed or the target monitoring server in the pre-grouping of the registration center. This includes: triggering a preprocessing probe to obtain the collection volume information of the monitoring collector to be processed, and combining it with the attribute information of the monitoring collector to be processed to obtain the center feature value of the monitoring collector to be processed and generate a first feature vector; obtaining the center feature value of the monitoring server in the registration center and generating a second feature vector; the monitoring collection scheduler calculates the center distance between the first feature vector and the second feature vector to obtain a center distance vector, and calculates the product of the pre-configured weight vector and the center distance vector to obtain a weighted distance value; determining that the weighted distance value is greater than the pre-configured weighted distance threshold. If it is greater, it creates the attribute information of the target monitoring server in the pre-grouping of the registration center. The processing module is also used to trigger the monitoring and acquisition executor, so that the monitoring and acquisition executor generates configuration information that the target monitoring server can recognize for the monitoring and acquisition device according to the attribute information of the monitoring and acquisition device to be processed or the target monitoring server and the configuration parameters of the configuration module center of the monitoring scheduling module, obtains the key-value pair relationship between the monitoring and acquisition device to be processed and the target monitoring server, and synchronizes the key-value pair relationship to the registration center running group to complete the monitoring and acquisition scheduling processing; The processing module is further configured to trigger the monitoring and acquisition executor to acquire the average daily resource utilization rate of each monitoring server at a calibrated time; trigger the monitoring and acquisition executor to determine whether the average daily resource utilization rate meets the conditions for automatic scaling down for each of the average daily resource utilization rates; if the conditions for automatic scaling down are met, trigger the monitoring and acquisition executor to adjust the pre-allocated resource configuration of the monitoring server corresponding to the average daily resource utilization rate that meets the conditions for automatic scaling down, so as to complete the automatic scaling down process.

11. A monitoring and data acquisition scheduler, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-8.

Images