System and method for dynamic management and loading of declarative cloud platform monitoring and collection configuration

Abstract

The present invention discloses a system and method for dynamic management and loading of declarative cloud platform monitoring and collection configuration, which belongs to the field of cloud computing. The technical problem to be solved by the present invention is how to realize dynamic management and loading of cloud platform monitoring and collection configuration, and reduce the cost of manual maintenance. The technical scheme is as follows: the system includes a resource management and controller controller, a monitoring collector monitor-agent and a configuration receiving processor agent, each cluster deploys a resource management and controller controller, each node deploys a monitoring collector monitor-agent and a configuration receiving processor agent, and the monitoring collector monitor-agent and the configuration receiving processor agent run in a pod and are deployed on each node of the cluster. The method is based on event-driven, monitors the changes of addition, deletion and modification of monitoring and collection configuration on the cluster, changes of cluster node information, changes of monitoring objects and changes of monitoring collectors, triggers the corresponding update processing logic, calculates the configuration information, sends it to the monitoring collector on the corresponding node, and reloads it.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing, and in particular to a system and method for dynamic management and loading of declarative cloud platform monitoring and collection configuration. Background Art

[0002] With the development of information technology and cloud computing, business needs are constantly changing, and the scale of cloud platform clusters and the services running on the clusters are also constantly changing. The change in demand is accompanied by the addition, removal and version iteration of service components, which brings certain challenges to the traditional monitoring and collection mode.

[0003] In the traditional monitoring mode, the correspondence between the monitoring collector and the monitoring object needs to be maintained manually, and adding or deleting a monitoring object requires manual maintenance of the corresponding monitoring configuration file.

[0004] In the traditional monitoring mode, changes in the parameters related to the monitoring collection configuration need to be manually modified and maintained, such as the collection cycle, account and password, etc. The changes in the monitoring objects lead to high manual maintenance costs and lack of flexibility. These factors include the expansion and reduction of cluster nodes, different monitoring requirements for different nodes, different monitoring requirements for different system architectures, and changes in the pods running the service (pod interruption and restart, changes in the number of replicas, etc.).

[0005] In the traditional monitoring mode, the traditional sidecar injection monitoring collection program used by the monitoring service has inherent defects, including the dependency injection manager and the risk of collector injection failure and the injection of the collector interfering with the stability of the service itself.

[0006] In the traditional monitoring mode, the monitoring collector does not support hot loading of collection configurations. To modify the configuration file, you need to restart the program to take effect.

[0007] Therefore, how to realize dynamic management and loading of cloud platform monitoring and collection configuration and reduce manual maintenance costs is a technical problem that needs to be solved urgently. Summary of the invention

[0008] The technical task of the present invention is to provide a system and method for declarative cloud platform monitoring and acquisition configuration dynamic management and loading, so as to solve the problem of how to realize dynamic management and loading of cloud platform monitoring and acquisition configuration and reduce manual maintenance costs.

[0009] The technical task of the present invention is achieved in the following way: a declarative cloud platform monitoring collection configuration dynamic management and loading system, the system includes a resource management controller controller, a monitoring collector monitor-agent and a configuration receiving processor agent, each cluster deploys a resource management controller controller, each node deploys a monitoring collector monitor-agent and a configuration receiving processor agent, the monitoring collector monitor-agent and the configuration receiving processor agent run in a pod and are deployed on each node of the cluster;

[0010] The resource management and controller include event monitoring module, event processing module and configuration task sending management module; the event monitoring module is used for resource management and controller monitoring events; the event processing module is used for resource management and controller event processing and configuration generation encapsulation; the configuration task sending management module is used for resource management and controller sending queue management.

[0011] As a preference, resource management and controller monitoring events are as follows:

[0012] The event monitoring module monitors the creation, modification, and deletion events of configuration resources, cluster node expansion and reduction events, cluster node events, and cluster pod creation, update, and deletion events;

[0013] The event monitoring module encapsulates the monitored change event into a configuration change message and sends it to the notification event processing module; the configuration change information includes the change operation type and the change information; the change operation type includes adding, deleting and updating.

[0014] Preferably, resource management and controller event processing and configuration generation encapsulation are as follows:

[0015] After receiving the event, the event processing module selects to execute the corresponding processing logic and updates the cluster metadata; the cluster metadata includes node metadata, pod data of the monitoring collector monitor-agent, monitoring collection configuration data, and pod data of the service component declared to be monitored;

[0016] Configuration update of the corresponding node collection program: When declaring the collection configuration, the user declares the monitoring object feature matching information, sets the agentSelector to declare which cluster nodes the collection configuration applies to, matches the corresponding nodes through node labels, and supports node multi-label key-value pair matching; sets the podSelector to declare which service component pods the collection configuration corresponds to, matches the corresponding pods through pod labels, and supports pod multi-label key-value pair matching;

[0017] Encapsulate messages and manage them in a queue; the monitoring and collection encapsulation message includes the specific configuration of monitoring and collection, the information of the node being sent, and the corresponding action to be executed; the information of the node being sent includes the node name and the IP address of the monitor-agent pod of the monitoring collector running on the node; the corresponding actions to be executed include adding, deleting, and modifying.

[0018] Preferably, resource management and controller send queue management are as follows:

[0019] Configure the task sending management module to monitor the queue entry signal in real time, and after receiving the queue entry signal, process the tasks of the same node in the queue, merge the tasks, repackage the tasks, and queue the sending tasks;

[0020] The configuration task sending management module periodically polls and executes the configuration sending task, sends the sending task in the queue to the corresponding acquisition end, and determines whether the sending is successful:

[0021] ①. If the sending is successful, the task will be dequeued;

[0022] ② If the sending fails, record the number of retries, calculate the next sending time, and re-enter the queue;

[0023] The retransmission time calculation formula is as follows:

[0024] T=T1+(retryNum+1)*interval

[0025] Where T represents the next resend time; T1 represents the current time; retryNum represents the number of retries; and interval represents the retry cycle.

[0026] Preferably, the configuration receiving processor agent is as follows:

[0027] (1) The configuration is sent to the configuration receiving processor agent;

[0028] (2) After receiving the configuration, the configuration receiving processor agent reads the MD5 value carried in the sent configuration and the message header;

[0029] (3) The configuration receiving processor agent generates an MD5 based on the configuration content and compares it with the MD5 value sent to perform configuration integrity verification:

[0030] ①. If the verification passes, proceed to step (4);

[0031] ② If the verification fails, the verification fails and the status is returned, waiting for the next issuance;

[0032] (4) After the configuration integrity check passes, preloading is performed to check whether the preloading is successful:

[0033] ①. If successful, notify the acquisition program to reload the configuration;

[0034] ② If the preloading fails, the preloading failure status is returned and the relevant failure log information is printed.

[0035] A method for dynamic management and loading of declarative cloud platform monitoring and collection configurations. The method is event-driven and monitors changes in addition, deletion and modification of monitoring and collection configurations on a cluster, changes in cluster node information, changes in monitoring objects and changes in monitoring collectors, triggers corresponding update processing logic, calculates configuration information, sends it to the monitoring collector on the corresponding node, and reloads it to achieve dynamic update. The Kubernetes CRD mechanism is then used to customize and extend the monitoring and collection configurations into Kubernetes (k8s) resources for addition, deletion, modification and search management. The dynamic management and loading of monitoring and collection configurations are achieved through the existing mature resource orchestration and management functions of Kubernetes.

[0036] Preferably, the method is as follows:

[0037] S1. By defining the monitoring collection configuration as a Kubernetes resource, users can create, update, and delete the corresponding monitoring collection configuration resources according to their needs, and realize dynamic management and loading of monitoring collection configuration; and use the Kubernetes custom resource extension feature to realize the unified model definition of monitoring collection configuration resources; the definition of the resource model is as follows:

[0038] apiVersion:lma.inspur.com / v1alpha1

[0039] kind:MonitorCollectorConfig

[0040] metadata:

[0041] name:string

[0042] namespace:string

[0043] spec:

[0044] conf:string

[0045] podSelector:[]string

[0046] agentSelector:[]string;

[0047] Resource model name: MonitorCollectorConfig;

[0048] Resource model configuration content: The collection configuration content declared in spec.conf, spec.agentSelector node label feature matching, the type is a list, multiple feature label matching calculations are implemented using logic and, the function is to declare that the configuration will act on the node with the matching feature; spec.podSelector is pod label feature matching, the type is a list, multiple feature label matching calculations are implemented using logic and, the function is to declare that the configuration will act on the service pod with the matching feature;

[0049] Resource model metadata: metadata.name is the name of the monitoring collection configuration resource, and metadata.namespace is the namespace defined by the collection configuration resource;

[0050] S2, dynamically calculates and updates the monitoring and collection configuration by listening to the creation, modification and deletion events of the monitoring and collection configuration resources, the pod creation and deletion events of the monitoring collector monitor-agent, the creation and deletion events of the service component pods that have been declared to be monitored and running on the cluster, and the cloud platform node events.

[0051] S3. The node where the monitored object is located is used to encapsulate the configuration information. A monitoring collection and sending task encapsulated includes the specific configuration of monitoring collection, the information of the sending node, and the type of task sending. The types of task sending include adding, updating, and deleting. The information of the sending node includes the node name and the IP address of the monitoring collector monitor-agent pod running on the node.

[0052] S4. After being queued to the unified sending queue management module for management, a configuration sending task is encapsulated and generated, and the task information is sent to the configuration receiving processor agent of the corresponding node through the sending task queue. The monitoring collector monitor-agent is notified through the configuration receiving processor agent to reload the collection configuration to take effect.

[0053] Preferably, the changes to the addition, deletion, and modification of monitoring collection configuration on the listening cluster are as follows:

[0054] (1) Listening for monitoring and configuration resource events: The resource management and controller receive and process monitoring and collection configuration resource events, and determine the type of monitoring and collection configuration resource events:

[0055] ①. If it is a creation event, execute step (2);

[0056] ② If it is a deletion event, execute step (3);

[0057] ③. If it is an update event, execute step (4);

[0058] (2) Processing the creation event: Generate the task message agentConfigs through feature matching FeatureSelector, generate multiple tasks according to nodeName, send them to the queue management module, and then execute step (5);

[0059] The feature matching FeatureSelector calculation process is as follows: Generate the monitoring configuration file of the node resource through agentSelector feature matching, compare the key-value pairs between the agentSelector list in the custom monitoring configuration resource and the cluster node label, and generate the configuration file (fileName=crNamespace_crName.conf); Calculate the configuration of the corresponding service monitoring through podSelector feature matching, and generate the configuration file (fileName=crNamespace_podName.conf); Dynamic discovery and configuration generation of service IP is achieved by obtaining the pod IP of the service;

[0060] (3) Processing deletion events: Generate the task message agentConfigs through feature matching FeatureSelector, generate multiple tasks according to nodeName, send them to the queue management module, and execute the next step (5);

[0061] (4) Process update events: Generate and send task message agentConfigs through feature matching FeatureSelector to determine whether it is an agentSelector or podSelector modification:

[0062] If yes, it is necessary to generate the task information for sending and deleting the configuration file under the old matching rule, generate multiple sending tasks according to the nodeName, send them to the queue management module, and then execute step (5);

[0063] (5) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful:

[0064] ①. If the sending is successful, the task is completed;

[0065] ② If the sending fails, the status is recorded, the task is re-queued, and the next step is to execute step (6);

[0066] (6) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful:

[0067] ①. If the verification fails, the execution result is returned;

[0068] ② If the verification is successful, execute step (7);

[0069] (7) Execute preloading and determine whether the preloading is successful:

[0070] ①. If preloading fails, the execution result is returned;

[0071] ② If the preloading is successful, execute step (8);

[0072] (8) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

[0073] Preferably, the changes in monitoring cluster node information are as follows:

[0074] (1) Event reception: Determine whether the node label is updated:

[0075] If it is a node label update event, the feature matching FeatureSelector generates a task message agentConfigs, action = create to add a new monitoring configuration, action = delete to delete the current unmatched monitoring configuration, generate an update task and send it to the queue management module, merge and process the tasks sent with the same node configuration, and put the task into the queue and wait for delivery;

[0076] (2) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful:

[0077] ①. If yes, the task is over;

[0078] ② If not, record the status, re-queue the task, and execute step (3) next;

[0079] (3) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful:

[0080] ①. If the verification fails, the execution result is returned;

[0081] ② If the verification is successful, execute step (4);

[0082] (4) Execute preloading and determine whether the execution is successful:

[0083] ①. If the execution fails, the execution result is returned;

[0084] ② If the execution is successful, go to step (5);

[0085] (5) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

[0086] Preferably, the pod creation and deletion events of the monitor-agent monitoring collector are monitored as follows:

[0087] (1) Event reception: Determine whether it is a monitoring collector pod event:

[0088] If yes, determine whether it is a pod ready event (other events are ignored):

[0089] If the configuration receives the agent pod ready event, the agentConfigs task message is generated through feature matching FeatureSelector, and the update task is sent to the queue management module. The sending tasks of the same node configuration are merged and processed, and the sending tasks are queued and wait for delivery. The next step is to execute step (3).

[0090] (2) Event reception: Determine whether it is a pod of a declared monitored service:

[0091] If so, the task message agentConfigs is generated through feature matching FeatureSelector, podready event action = create, pod deletion event action = delete, and the update task is generated and sent to the queue management module. The tasks with the same node configuration are merged and processed, and the sending tasks are queued and wait for distribution. The next step is to execute step (3);

[0092] (3) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful:

[0093] ①. If yes, the task is over;

[0094] ② If not, record the status, re-queue the task, and execute step (4) next;

[0095] (4) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful:

[0096] ①. If the verification fails, the execution result is returned;

[0097] ② If the verification is successful, execute step (5);

[0098] (5) Execute preloading and determine whether the execution is successful:

[0099] ①. If the execution fails, the execution result is returned;

[0100] ② If the execution is successful, go to step (6);

[0101] (6) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

[0102] The system and method for dynamic management and loading of declarative cloud platform monitoring and collection configuration of the present invention have the following advantages:

[0103] (1) The present invention gives full play to the characteristics of KubernetesCRD, defines a unified monitoring and collection configuration resource model, and provides a unified standard interface for users to manage the monitoring and collection configuration of their own service components. It makes full use of the Kubernetes resource event monitoring mechanism to dynamically calculate and update the monitoring resource information declared on the cloud platform and the monitoring and collection configuration of the corresponding node collector in real time. It also solves the problem of user-defined monitoring and collection configuration, dynamic update and loading of monitoring configuration, and adopts non-invasive external monitoring detection methods to realize the monitoring of platform service components, avoiding the risks and maintenance costs brought by the traditional method of injecting collectors; at the same time, through the unified monitoring and collection configuration model, unified management method, and the characteristics of real-time dynamic calculation of monitoring objects and dynamic loading of collection configuration, it reduces the cost of manual maintenance;

[0104] (ii) The present invention allows users to customize monitoring and acquisition configurations according to actual needs, and write corresponding monitoring and acquisition configurations according to the actual needs of the monitored object (which may be a service, a file, a device, etc.), thereby realizing the user's self-management of the acquisition configuration;

[0105] (III) The present invention implements declarative collection configuration: users can customize collection configuration when deploying services, including collection plug-in type, collection cycle, and other parameter settings supported by the collector;

[0106] (iv) The present invention implements the addition, deletion and modification of user management configuration resources: when a new service component is added, the corresponding collection configuration resource is created; when the component version is iterated, the corresponding collection configuration resource is updated; when the service component is removed from the shelves, the corresponding configuration resource is deleted;

[0107] (V) The present invention uses a non-invasive external collection probe method to achieve monitoring and collection: deploy a collector on the node where the service is running, and send the collection configuration corresponding to the service to the corresponding node collector to achieve monitoring and collection of the service, as shown in the attached figure. Figure 5 As shown in the figure, the differences between the new and old modes of service collection configuration are as follows:

[0108] Compared with the traditional sidecar mode, in the traditional mode, because the service component and the monitoring collector run in the same pod, the service IP in the monitoring collection configuration can be set to 127.0.0.1. However, in the non-intrusive monitoring collection mode of the present invention, the service IP in the monitoring collection configuration corresponding to the service pod should be set to pod IP;

[0109] Traditional sidecar mode service monitoring collection configuration:

[0110] Conf:

[0111] Server="http: / / 127.0.0.1:80 / status"

[0112] Non-dumping mode service monitoring collection configuration:

[0113] Conf:

[0114] Server="${podIp}:80 / status"

[0115] The Controller dynamically obtains the podIp and replaces the variable ${podIp} to generate the server configuration corresponding to the pod, and sends and loads the configuration to the collector of the pod running node;

[0116] (VI) The present invention realizes dynamic matching and discovery of collection objects: when defining collection configuration, users need to declare matching features corresponding to monitoring objects, so that the management program can monitor the changes of corresponding resources of the cluster and dynamically process and update the monitoring configuration through matching features in the user-defined monitoring configuration;

[0117] (VII) The present invention implements dynamic loading of collection configurations. The addition, deletion, and modification operations of the collection configurations by the user, as well as the dynamic changes of the monitoring objects (which may be node expansion and contraction, changes in node labels, creation and deletion of pods of service components), will trigger the recalculation and update of the monitoring configurations, and will be synchronously updated to the configuration files of the corresponding collectors, and will be reloaded by the collectors to take effect (the collectors implement the reload interface to support hot loading of the monitoring configurations);

[0118] (VIII) User-defined management of monitoring and collection configuration, enabling users to customize corresponding monitoring and collection configuration according to their actual needs, and manage the addition, deletion and modification of collection configuration;

[0119] (IX) The present invention realizes dynamic loading, which can realize real-time monitoring of the monitoring objects declared by the user, real-time update of the monitoring object information, and synchronous update and loading of the corresponding monitoring collection configuration, thereby reducing the cost of manual maintenance of the collection configuration file;

[0120] (10) The present invention achieves the decoupling of the monitoring collector and the service component, solves the problem of collector injection dependency when deploying the service component in the sidecar mode adopted by the traditional monitoring service component, and avoids the risk of the collector interfering with the operation of the service component. BRIEF DESCRIPTION OF THE DRAWINGS

[0121] The present invention is further described below in conjunction with the accompanying drawings.

[0122] Attached Figure 1 Deployment architecture diagram of the system for dynamic management and loading of declarative cloud platform monitoring and collection configuration;

[0123] Attached Figure 2 A flowchart for resource management and controller monitoring events;

[0124] Attached Figure 3 Generate encapsulated flow charts for resource management and controller event handling and configuration;

[0125] Attached Figure 4 Send a flow chart of queue management for resource management and controller;

[0126] Attached Figure 5 This is a comparison diagram of the service monitoring modes of the new and old architectures. DETAILED DESCRIPTION

[0127] The following detailed description is made of the system and method for dynamic management and loading of declarative cloud platform monitoring and collection configurations of the present invention with reference to the accompanying drawings and specific embodiments of the specification.

[0128] Embodiment 1:

[0129] As attached Figure 1 As shown, the declarative cloud platform monitoring collection configuration dynamic management and loading system of the present invention includes a resource management and controller controller, a monitoring collector monitor-agent and a configuration receiving processor agent, each cluster deploys a resource management and controller controller, each node deploys a monitoring collector monitor-agent and a configuration receiving processor agent, the monitoring collector monitor-agent and the configuration receiving processor agent run in a pod and are deployed on each node of the cluster;

[0130] The resource management and controller include event monitoring module, event processing module and configuration task sending management module; the event monitoring module is used for resource management and controller monitoring events; the event processing module is used for resource management and controller event processing and configuration generation encapsulation; the configuration task sending management module is used for resource management and controller sending queue management.

[0131] As attached Figure 2 As shown, the resource management and controller monitoring events in this embodiment are as follows:

[0132] (1) The event monitoring module monitors the creation, modification, and deletion events of configuration resources, cluster node expansion and reduction events, cluster node events, and cluster pod creation, update, and deletion events;

[0133] (2) The event monitoring module encapsulates the monitored change event into a configuration change message and sends it to the notification event processing module; the configuration change information includes the change operation type and the change information; the change operation type includes adding, deleting and updating.

[0134] As attached Figure 3 As shown, the resource management and controller event processing and configuration generation encapsulation in this embodiment are specifically as follows:

[0135] (1) After receiving the event, the event processing module selects to execute the corresponding processing logic and updates the cluster metadata; the cluster metadata includes node metadata, pod data of the monitoring collector monitor-agent, monitoring collection configuration data, and pod data of the service component declared to be monitored;

[0136] (2) Calculate the configuration update of the corresponding node collection program: When declaring the collection configuration, the user declares the setting of monitoring object feature matching information, sets agentSelector to declare which cluster nodes the collection configuration applies to, matches the corresponding nodes through node labels, and supports node multi-label key-value pair matching; sets podSelector to declare which service component pods the collection configuration corresponds to, matches the corresponding pods through pod labels, and supports pod multi-label key-value pair matching;

[0137] (3) Encapsulate messages and manage them in a queue. The monitoring and collection encapsulation message includes the specific configuration of monitoring and collection, the information of the node being sent, and the corresponding action to be executed. The information of the node being sent includes the node name and the IP address of the monitor-agent pod of the monitoring collector running on the node. The corresponding actions to be executed include adding, deleting, and modifying.

[0138] As attached Figure 4 As shown, the resource management and controller sending queue management in this embodiment are as follows:

[0139] (1) Configure the task sending management module to monitor the queue entry signal in real time, receive the queue entry signal, process the tasks of the same node in the queue, merge the tasks, repackage the tasks, and queue the sending tasks;

[0140] (2) The configuration task sending management module periodically polls and executes the configuration sending task, sends the sending task in the queue to the corresponding collection end, and determines whether the sending is successful:

[0141] ①. If the sending is successful, the task will be dequeued;

[0142] ② If the sending fails, record the number of retries, calculate the next sending time, and re-enter the queue;

[0143] The retransmission time calculation formula is as follows:

[0144] T=T1+(retryNum+1)*interval

[0145] Where T represents the next resend time; T1 represents the current time; retryNum represents the number of retries; and interval represents the retry cycle.

[0146] The configuration of the receiving processor agent in this embodiment is as follows:

[0147] (1) The configuration is sent to the configuration receiving processor agent;

[0148] (2) After receiving the configuration, the configuration receiving processor agent reads the MD5 value carried in the sent configuration and the message header;

[0149] (3) The configuration receiving processor agent generates an MD5 based on the configuration content and compares it with the MD5 value sent to perform configuration integrity verification:

[0150] ①. If the verification passes, proceed to step (4);

[0151] ② If the verification fails, the verification fails and the status is returned, waiting for the next issuance;

[0152] (4) After the configuration integrity check passes, preloading is performed to check whether the preloading is successful:

[0153] ①. If successful, notify the acquisition program to reload the configuration;

[0154] ② If the preloading fails, the preloading failure status is returned and the relevant failure log information is printed.

[0155] Embodiment 2:

[0156] The method for dynamic management and loading of declarative cloud platform monitoring and collection configuration of the present invention is based on event-driven, monitors the changes of addition, deletion and modification of monitoring and collection configuration on the cluster, changes of cluster node information, changes of monitoring objects and changes of monitoring collectors, triggers corresponding update processing logic, calculates configuration information, sends it to the monitoring collector on the corresponding node, and reloads it to realize dynamic update; then uses the Kubernetes CRD mechanism to customize and expand the monitoring and collection configuration into Kubernetes (k8s) resources for addition, deletion, modification and search management, and realizes dynamic management and loading of monitoring and collection configuration through the existing mature resource orchestration and management functions of Kubernetes.

[0157] Preferably, the method is as follows:

[0158] S1. By defining the monitoring collection configuration as a Kubernetes resource, users can create, update, and delete the corresponding monitoring collection configuration resources according to their needs, and realize dynamic management and loading of monitoring collection configuration; and use the Kubernetes custom resource extension feature to realize the unified model definition of monitoring collection configuration resources; the definition of the resource model is as follows:

[0159] apiVersion:lma.inspur.com / v1alpha1

[0160] kind:MonitorCollectorConfig

[0161] metadata:

[0162] name:string

[0163] namespace:string

[0164] spec:

[0165] conf:string

[0166] podSelector:[]string

[0167] agentSelector:[]string;

[0168] Resource model name: MonitorCollectorConfig;

[0169] Resource model configuration content: The collection configuration content declared in spec.conf, spec.agentSelector node label feature matching, the type is a list, multiple feature label matching calculations are implemented using logic and, the function is to declare that the configuration will act on the node with the matching feature; spec.podSelector is pod label feature matching, the type is a list, multiple feature label matching calculations are implemented using logic and, the function is to declare that the configuration will act on the service pod with the matching feature;

[0170] Resource model metadata: metadata.name is the name of the monitoring collection configuration resource, and metadata.namespace is the namespace defined by the collection configuration resource;

[0171] Example:

[0172] ```

[0173] apiVersion:lma.inspur.com / v1alpha1

[0174] kind:MonitorCollectorConfig

[0175] metadata:

[0176] name:edac

[0177] namespace:monitoring

[0178] spec:

[0179] conf:|-

[0180] ... / / Specific configuration content

[0181] agentSelector:

[0182] -node-role.kubernetes.io / bare-metal:enabled

[0183] - kubernetes.io / arch:amd64

[0184] ```

[0185] ```

[0186] apiVersion:lma.inspur.com / v1alpha1

[0187] kind:MonitorCollectorConfig

[0188] metadata:

[0189] name:redis

[0190] namespace:common

[0191] spec:

[0192] conf:|-

[0193] ... / / Specific configuration content

[0194] podSelector:

[0195] -app=redis4-ha

[0196] -component=server

[0197] ```

[0198] Among them, filename: configuration file name; crNamespace: namespace of monitoring configuration resource; crName: name of monitoring configuration resource; podName: name of monitored service component pod matched by features; agentConfigs: sending task message; agentConfigs(nodeName,agentPodIP,{action,filename,content}), agentConfigs contains the following information: nodeName: node name; agentPodIP: collector PodIP; action: execution operation of corresponding file (create,update,delete); filename: configuration file name; content: content of filename file.

[0199] S2, dynamically calculates and updates the monitoring and collection configuration by listening to the creation, modification and deletion events of the monitoring and collection configuration resources, the pod creation and deletion events of the monitoring collector monitor-agent, the creation and deletion events of the service component pods that have been declared to be monitored and running on the cluster, and the cloud platform node events.

[0200] S3. The node where the monitored object is located is used to encapsulate the configuration information. A monitoring collection and sending task encapsulated includes the specific configuration of monitoring collection, the information of the sending node, and the type of task sending. The types of task sending include adding, updating, and deleting. The information of the sending node includes the node name and the IP address of the monitoring collector monitor-agent pod running on the node.

[0201] S4. After being queued to the unified sending queue management module for management, a configuration sending task is encapsulated and generated, and the task information is sent to the configuration receiving processor agent of the corresponding node through the sending task queue. The monitoring collector monitor-agent is notified through the configuration receiving processor agent to reload the collection configuration to take effect.

[0202] The changes in the addition, deletion and modification of the monitoring collection configuration on the monitoring cluster in this embodiment are as follows:

[0203] (1) Listening for monitoring and configuration resource events: The resource management and controller receive and process monitoring and collection configuration resource events, and determine the type of monitoring and collection configuration resource events:

[0204] ①. If it is a creation event, execute step (2);

[0205] ② If it is a deletion event, execute step (3);

[0206] ③. If it is an update event, execute step (4);

[0207] (2) Processing the creation event: Generate the task message agentConfigs through feature matching FeatureSelector, generate multiple tasks according to nodeName, send them to the queue management module, and then execute step (5);

[0208] The feature matching FeatureSelector calculation process is as follows: Generate the monitoring configuration file of the node resource through agentSelector feature matching, compare the key-value pairs between the agentSelector list in the custom monitoring configuration resource and the cluster node label, and generate the configuration file (fileName=crNamespace_crName.conf); Calculate the configuration of the corresponding service monitoring through podSelector feature matching, and generate the configuration file (fileName=crNamespace_podName.conf); Dynamic discovery and configuration generation of service IP is achieved by obtaining the pod IP of the service;

[0209] (3) Processing deletion events: Generate the task message agentConfigs through feature matching FeatureSelector, generate multiple tasks according to nodeName, send them to the queue management module, and execute the next step (5);

[0210] (4) Process update events: Generate and send task message agentConfigs through feature matching FeatureSelector to determine whether it is an agentSelector or podSelector modification:

[0211] If yes, it is necessary to generate the task information for sending and deleting the configuration file under the old matching rule, generate multiple sending tasks according to the nodeName, send them to the queue management module, and then execute step (5);

[0212] (5) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful:

[0213] ①. If the sending is successful, the task is completed;

[0214] ② If the sending fails, the status is recorded, the task is re-queued, and the next step is to execute step (6);

[0215] (6) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful:

[0216] ①. If the verification fails, the execution result is returned;

[0217] ② If the verification is successful, execute step (7);

[0218] (7) Execute preloading and determine whether the preloading is successful:

[0219] ①. If preloading fails, the execution result is returned;

[0220] ② If the preloading is successful, execute step (8);

[0221] (8) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

[0222] Preferably, the changes in monitoring cluster node information are as follows:

[0223] (1) Event reception: Determine whether the node label is updated:

[0224] If it is a node label update event, the feature matching FeatureSelector generates a task message agentConfigs, action = create to add a new monitoring configuration, action = delete to delete the current unmatched monitoring configuration, generate an update task and send it to the queue management module, merge and process the tasks sent with the same node configuration, and put the task into the queue and wait for delivery;

[0225] (2) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful:

[0226] ①. If yes, the task is over;

[0227] ② If not, record the status, re-queue the task, and execute step (3) next;

[0228] (3) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful:

[0229] ①. If the verification fails, the execution result is returned;

[0230] ② If the verification is successful, execute step (4);

[0231] (4) Execute preloading and determine whether the execution is successful:

[0232] ①. If the execution fails, the execution result is returned;

[0233] ② If the execution is successful, go to step (5);

[0234] (5) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

[0235] Preferably, the pod creation and deletion events of the monitor-agent monitoring collector are monitored as follows:

[0236] (1) Event reception: Determine whether it is a monitoring collector pod event:

[0237] If yes, determine whether it is a pod ready event (other events are ignored):

[0238] If the configuration receives the agent pod ready event, the agentConfigs task message is generated through feature matching FeatureSelector, and the update task is sent to the queue management module. The sending tasks of the same node configuration are merged and processed, and the sending tasks are queued and wait for delivery. The next step is to execute step (3).

[0239] (2) Event reception: Determine whether it is a pod of a declared monitored service:

[0240] If so, the task message agentConfigs is generated through feature matching FeatureSelector, podready event action = create, pod deletion event action = delete, and the update task is generated and sent to the queue management module. The tasks with the same node configuration are merged and processed, and the sending tasks are queued and wait for distribution. The next step is to execute step (3);

[0241] (3) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful:

[0242] ①. If yes, the task is over;

[0243] ② If not, record the status, re-queue the task, and execute step (4) next;

[0244] (4) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful:

[0245] ①. If the verification fails, the execution result is returned;

[0246] ② If the verification is successful, execute step (5);

[0247] (5) Execute preloading and determine whether the execution is successful:

[0248] ①. If the execution fails, the execution result is returned;

[0249] ② If the execution is successful, go to step (6);

[0250] (6) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

[0251] The present invention uses the Kubernetes custom resource extension feature to define a unified monitoring collection configuration resource model, realizes the management of user-defined monitoring collection configuration resources, supports declarative monitoring object feature matching, supports dynamic loading of monitoring collection configuration, realizes real-time discovery of monitoring objects and real-time synchronization of monitoring collection configuration by monitoring platform resource events, uniformly manages the monitoring resources of the entire cluster, and uniformly manages and schedules the distribution of configuration tasks, and adopts a non-invasive monitoring collection implementation method to realize the decoupling between monitoring collection and service components, and abandons the invasive monitoring collection method of the traditional sidecar injection collector. By real-time monitoring of the events of the business component pod that the monitoring collection configuration resource has declared to match, the changes of the pod are automatically sensed, and the collection configuration of the corresponding collector on the pod running node is dynamically updated.

[0252] The present invention realizes dynamic management of monitoring and acquisition configuration, and the dynamic management includes two aspects. On the one hand, it is dynamic management of monitoring and acquisition configuration resources. The controller realizes the management of addition, deletion and modification of monitoring and acquisition configuration by monitoring the events of monitoring and acquisition configuration resources. When monitoring and acquisition configuration resource events are monitored, monitoring configuration change information (creation, update and deletion of monitoring configuration files) of corresponding node collectors is generated through feature matching calculation, and then sent to the corresponding collector, agentSelector matches node labels, and podSelector matches pod labels; on the other hand, it is dynamic management of monitoring objects. The controller monitors cluster node events and cluster service component pod events in real time, generates monitoring configuration change information (creation, update and deletion of monitoring configuration files) of corresponding node collectors through feature matching calculation, and then sends it to the corresponding collector, agentSelector matches node labels, and podSelector matches pod labels.

[0253] The present invention realizes the management of user-defined monitoring and acquisition configuration resources. Users can create, update, and delete corresponding monitoring and acquisition configuration resources according to needs. At the same time, it supports dynamic loading of monitoring and acquisition configuration. The monitoring collector supports configuration reload, exposes the reload interface, and externally notifies the reload interface to trigger the reload logic, and the configuration update takes effect, thereby solving the defect that the traditional monitoring collector cannot hot-load the acquisition configuration; it also supports declarative monitoring object feature matching. When users customize the monitoring and acquisition configuration, they need to declare the monitoring object selected by the configuration; and through event monitoring of platform resources, real-time discovery of monitoring objects and real-time synchronous update of monitoring and acquisition configuration are realized, cloud platform monitoring resources (including user-declared monitoring and acquisition configuration and monitoring object resources) are uniformly processed and calculated, and configuration delivery tasks are uniformly queued and re-queued and retried if sending fails.

[0254] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A declarative cloud platform monitoring collection configuration dynamic management and loading system, characterized in that: The system includes resource management and controller controller, monitoring collector monitor-agent and configuration receiving processor agent. Each cluster deploys a resource management and controller controller, and each node deploys a monitoring collector monitor-agent and configuration receiving processor agent. The monitoring collector monitor-agent and configuration receiving processor agent run in a pod and are deployed on each node of the cluster. Resource management and controller controller includes event monitoring module, event processing module and configuration task sending management module; The event monitoring module is used for resource management and controller monitoring events; the event processing module is used for resource management and controller event processing and configuration generation encapsulation; the configuration task sending management module is used for resource management and controller sending queue management; Among them, resource management and controller monitoring events are as follows: The event monitoring module monitors the creation, modification, and deletion events of configuration resources, cluster node expansion and reduction events, cluster node events, and cluster pod creation, update, and deletion events; The event monitoring module encapsulates the monitored change event into a configuration change message and sends it to the notification event processing module; the configuration change information includes the change operation type and the change information; the change operation type includes adding, deleting and updating; Resource management and controller event processing and configuration generation encapsulation are as follows: After receiving the event, the event processing module selects to execute the corresponding processing logic and updates the cluster metadata; the cluster metadata includes node metadata, pod data of the monitoring collector monitor-agent, monitoring collection configuration data, and pod data of the service component declared to be monitored; Configuration update of the corresponding node collection program: When declaring the collection configuration, the user declares the monitoring object feature matching information, sets the agentSelector to declare which cluster nodes the collection configuration applies to, matches the corresponding nodes through node labels, and supports node multi-label key-value pair matching; sets the podSelector to declare which service component pods the collection configuration corresponds to, matches the corresponding pods through pod labels, and supports pod multi-label key-value pair matching; Encapsulate messages and manage them in queues; monitoring and collection encapsulation messages include monitoring and collection specific configurations, information about the nodes being sent, and corresponding actions to be performed; the information about the nodes being sent includes the node name and the IP address of the monitor-agent pod of the monitoring collector running on the node; the corresponding actions to be performed include adding, deleting, and modifying; Resource management and controller sending queue management are as follows: Configure the task sending management module to monitor the queue entry signal in real time, and after receiving the queue entry signal, process the tasks of the same node in the queue, merge the tasks, repackage the tasks, and queue the sending tasks; The configuration task sending management module periodically polls and executes the configuration sending task, sends the sending task in the queue to the corresponding acquisition end, and determines whether the sending is successful: ①. If the sending is successful, the task will be dequeued; ② If the sending fails, record the number of retries, calculate the next sending time, and re-enter the queue; The retransmission time calculation formula is as follows: T = T1 + (retryNum +1) * interval Where, T represents the next resend time; T1 represents the current time; retryNum represents the number of retries; interval represents the retry cycle; Configure the receiving processor agent as follows: (1) The configuration is sent to the configuration receiving processor agent; (2) After receiving the configuration, the configuration receiving processor agent reads the MD5 value carried in the sent configuration and the message header; (3) The configuration receiving processor agent generates an MD5 based on the configuration content and compares it with the MD5 value sent to perform configuration integrity verification: ①. If the verification passes, proceed to step (4); ② If the verification fails, the verification fails and the status is returned, waiting for the next issuance; (4) After the configuration integrity check passes, preloading is performed and the preloading is successful: ①. If successful, notify the acquisition program to reload the configuration; ② If the preloading fails, the preloading failure status is returned and the relevant failure log information is printed.

2. A method for dynamic management and loading of declarative cloud platform monitoring and collection configuration, characterized in that: This method is event-driven, monitoring the changes of adding, deleting and modifying the monitoring collection configuration on the cluster, the changes of cluster node information, the changes of monitoring objects and the changes of monitoring collectors, triggering the corresponding update processing logic, calculating the configuration information, sending it to the monitoring collector on the corresponding node, and reloading it to achieve dynamic update; then using the Kubernetes CRD mechanism to customize the monitoring collection configuration to expand it into Kubernetes resources for adding, deleting, modifying and searching management, and realizing the dynamic management and loading of monitoring collection configuration through the resource orchestration and management functions of Kubernetes; the details are as follows: S1. By defining the monitoring collection configuration as a Kubernetes resource, users can create, update, and delete the corresponding monitoring collection configuration resources according to their needs, and realize dynamic management and loading of monitoring collection configuration; and use the Kubernetes custom resource extension feature to realize the unified model definition of monitoring collection configuration resources; the definition of the resource model is as follows: apiVersion: lma.inspur.com / v1alpha1 kind: MonitorCollectorConfig metadata: name: string namespace: string spec: conf: string podSelector: []string agentSelector: []string; Resource model name: MonitorCollectorConfig; Resource model configuration content: The collection configuration content declared in spec.conf, spec.agentSelector node label feature matching, the type is a list, multiple feature label matching calculations are implemented using logic and, the function is to declare that the configuration will act on the node with the matching feature; spec.podSelector is pod label feature matching, the type is a list, multiple feature label matching calculations are implemented using logic and, the function is to declare that the configuration will act on the service pod with the matching feature; Resource model metadata: metadata.name is the name of the monitoring collection configuration resource, and metadata.namespace is the namespace defined by the collection configuration resource; S2, dynamically calculates and updates the monitoring and collection configuration by listening to the creation, modification and deletion events of the monitoring and collection configuration resources, the pod creation and deletion events of the monitoring collector monitor-agent, the creation and deletion events of the service component pods that have been declared to be monitored and running on the cluster, and the cloud platform node events. S3. The node where the monitored object is located is used to encapsulate the configuration information. A monitoring collection and sending task encapsulated includes the specific configuration of monitoring collection, the information of the sending node, and the type of task sending. The types of task sending include adding, updating, and deleting. The information of the sending node includes the node name and the IP address of the monitoring collector monitor-agent pod running on the node. S4, after being queued to the unified sending queue management module for management, a configuration sending task is encapsulated and generated, and the task information is sent to the configuration receiving processor agent of the corresponding node through the sending task queue, and the monitoring collector monitor-agent is notified through the configuration receiving processor agent to reload the collection configuration to take effect; The changes in the monitoring collection configuration addition, deletion, and modification on the listening cluster are as follows: (1) Listening for monitoring configuration resource events: The resource management and controller receive and process monitoring collection configuration resource events and determine the type of monitoring collection configuration resource events: ①. If the event is created, execute step (2); ② If it is a deletion event, execute step (3); ③. If it is an update event, execute step (4); (2) Processing creation events: Generate the task message agentConfigs through feature matching FeatureSelector, generate multiple tasks according to nodeName, send them to the queue management module, and then execute step (5); The calculation process of feature matching FeatureSelector is as follows: Generate monitoring configuration files for node resources through agentSelector feature matching, compare key-value pairs between agentSelector list in custom monitoring configuration resources and cluster node labels, and generate configuration files (fileName=crNamespace_crName.conf); Calculate the configuration of corresponding service monitoring through podSelector feature matching, and generate configuration files (fileName=crNamespace_podName.conf); Dynamic discovery and configuration generation of service IP are achieved by obtaining pod IP of service; (3) Processing deletion events: Generate the task message agentConfigs through feature matching FeatureSelector, generate multiple tasks according to nodeName, send them to the queue management module, and execute the next step (5); (4) Process update events: Generate and send task message agentConfigs through feature matching FeatureSelector, and determine whether it is agentSelector or podSelector modification: If yes, it is necessary to generate the task information for sending and deleting the configuration file under the old matching rule, generate multiple sending tasks according to the nodeName, send them to the queue management module, and then execute step (5); (5) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful: ①. If the sending is successful, the task is completed; ② If the sending fails, the status is recorded, the task is re-queued, and the next step is to execute step (6); (6) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful: ①. If the verification fails, the execution result is returned; ② If the verification is successful, proceed to step (7); (7) Execute preloading and determine whether the preloading is successful: ①. If preloading fails, the execution result is returned; ② If the preloading is successful, execute step (8); (8) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect; The changes in monitoring cluster node information are as follows: (1) Event reception: Determine whether the node label is updated: If it is a node label update event, the feature matching FeatureSelector generates a task message agentConfigs, action=create adds a new monitoring configuration, action=delete deletes the current unmatched monitoring configuration, generates an update task and sends it to the queue management module, merges and processes the tasks sent with the same node configuration, and puts the task into the queue and waits for delivery; (2) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful: ①. If yes, the task is over; ② If not, record the status, re-queue the task, and execute step (3) next. (3) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful: ①. If the verification fails, the execution result is returned; ② If the verification is successful, proceed to step (4); (4) Execute preloading and determine whether the execution is successful: ①. If the execution fails, the execution result is returned; ② If the execution is successful, go to step (5); (5) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect; The pod creation and deletion events of the monitor-agent monitoring collector are as follows: (1) Event reception: Determine whether it is a monitoring collector pod event: If yes, determine whether it is a pod ready event: If the configuration receives the agent pod ready event, the agentConfigs task message is generated through feature matching FeatureSelector, and the update task is sent to the queue management module. The tasks configured to send the same node are merged and processed, and the tasks are queued and wait for delivery. The next step is to execute step (3). (2) Event reception: Determine whether it is a pod of a declared monitored service: If so, the task message agentConfigs is generated through feature matching FeatureSelector, pod ready event action = create, pod deletion event action = delete, and the update task is generated and sent to the queue management module. The tasks with the same node configuration are merged and processed, and the sending tasks are queued and wait for delivery. The next step is to execute step (3); (3) The queue management module polls the processing queue, sends configuration updates, records the sending results, and determines whether the sending results are successful: ①. If yes, the task is over; ② If not, record the status, re-queue the task, and execute step (4) next; (4) lma-agent receives the message, processes the configuration update task, verifies the configuration, and determines whether the verification is successful: ①. If the verification fails, the execution result is returned; ② If the verification is successful, proceed to step (5); (5) Execute preloading and determine whether the execution is successful: ①. If the execution fails, the execution result is returned; ② If the execution is successful, go to step (6); (6) lma-agent updates the node collection configuration file, calls the reload interface of the monitoring collector monitor-agent to notify the monitoring collector monitor-agent to execute reload, and the monitoring collector monitor-agent hot-loads the collection configuration, and the configuration loading takes effect.

Images