Adaptive control method and device for cloud-native cluster intelligent operation and maintenance

Abstract

The application discloses a self-adaptive control method and device for cloud native cluster intelligent operation and maintenance, and relates to the technical field of cloud native application.The operation and maintenance demand of a K8s cluster is described through a YAML format file, and the operation and maintenance demand is acquired by analyzing the YAML format file; the planning control logic and the index analyzer are generated according to a rule trigger condition and an operation and maintenance operation supported by the cluster; the index collector type is determined according to an index type and an index parameter, and the corresponding index collector is generated according to a template; the index monitor to be created is determined according to the analyzed operation and maintenance demand, the index collector and the index analyzer are combined to obtain the index monitor, the index is automatically collected through the index collector, and the operation and maintenance planning logic corresponding to the operation and maintenance demand is determined through the index analyzer; the index monitor and the planning control logic are integrated, the code of the K8s controller is generated and deployed into the K8s cluster, and the self-adaptive control for the cloud native cluster intelligent operation and maintenance is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud-native application, and in particular to a self-adaptive control method and device for cloud-native cluster intelligent operation and maintenance. BACKGROUND

[0002] At present, Kubernetes (K8s, an open-source container orchestration application) has become the de facto standard for cloud-native cluster management. Applications are deployed on the K8s platform after being encapsulated by containers, and collaborate with each other to form a micro-service application cluster with the help of the capabilities provided by the K8s cluster. However, containerized applications only have basic failure recovery capabilities. In order to achieve more complex operation and maintenance requirements, K8s provides a controller mode for developers, which allows developers to write cluster controllers to compare and update the state of resources in the cluster, and to achieve more rich operation and maintenance control requirements.

[0003] In K8s, the controller mode specifies the expected state of the monitored resource object in the back-and-forth cycle, and the controller is responsible for ensuring that the current state of the resource is consistent with the expected state. A series of native controllers are built into the K8s cluster, which can manage the state of cluster resources by communicating with the API server. For example, the Job controller will interact with the K8s API server to create a Pod to execute a task when a new Job task is created by the user. When the Pod completes execution, the Job controller determines that the expected state has been achieved, and updates the state of the Job object.

[0004] The controller mode relies on human-made operation and maintenance rules, and developers need to have a full understanding of the cluster and the business domain. Even so, as the size of the cluster expands and a large number of different controllers are deployed into the cluster, it is difficult for developers to reach a consensus in advance, which may lead to conflicts between the logic of controller behavior, reducing the efficiency and correctness of operation and maintenance.

[0005] In addition, the rapid iteration development scenario faced by cloud-native applications requires timely updating of system functions through user feedback requirements, which also poses higher challenges for developers to write new operation and maintenance logic. SUMMARY

[0006] In order to solve the problem of high difficulty in developing operation and maintenance controllers and complex iteration in increasingly complex and variable business scenarios, the present application proposes a self-adaptive control method and corresponding device based on K8s, which describes the operation and maintenance requirements of K8s by creating a specific language and format text, and then monitors the indicators and operation and maintenance planning by analyzing the operation and maintenance requirements, to realize the self-adaptive control of the K8s cluster.

[0007] The technical solutions adopted by the present application are as follows:

[0008] The application discloses a self-adaptive control method for cloud-native cluster intelligent operation and maintenance.

[0009] The operation and maintenance requirement description step comprises the following steps:

[0010] The operation and maintenance requirement analysis step comprises the following steps:

[0011] The operation and maintenance planning generation step comprises the following steps:

[0012] The index collector generation step comprises the following steps:

[0013] The index monitoring and analysis step comprises the following steps:

[0014] The K8s controller code deployment step comprises the following steps:

[0015] Preferably, in the operation and maintenance requirement description step, the language used in the YAML format file is defined as follows:

[0016] The state S of the cluster at a certain time is defined, and each element in S represents the state s of each resource in the cluster at the time.

[0017] The operation and maintenance requirement R of the cluster is defined as the change amount of the resources in the cluster from the initial state S0 to the desired state St.

[0018] The set A of operation and maintenance operations supported by the cluster is defined, and each element in A represents an operation a.

[0019] Preferably, the state s comprises a condition L for screening resources, storage information V related to the resources and attributes P that need to be focused on in the resources.

[0020] Preferably, L comprises one or more of an API group, a type, a name, a namespace, a label list and an annotation; V comprises mounting Volume and local storage system mapping information; and P comprises key-value information of multiple types.

[0021] Preferably, the defining operation a comprises a resource type rt to be operated, a selector rs of the resource, an operation type op to be executed, and a parameter list p when the operation is executed.

[0022] Preferably, in the operation and maintenance planning generation step, the step of generating planning control logic comprises:

[0023] When obtaining the operation and maintenance requirements, the resource types related to the operation and maintenance requirements and the API interfaces for coping with the operation and maintenance requirements are screened, the API interfaces are used for adding, deleting, modifying and inquiring the resources in the cluster, and the API server provides the API interfaces;

[0024] Random parameters are generated, API interfaces affecting the resources are screened from the API interfaces, and the random parameters are adjusted according to the screened API interfaces;

[0025] If the state of the cluster resources reaches the expected state after the adjustment of the parameters, the planning is stopped and the planning control logic is generated, otherwise the API interfaces of the resources are combined and the state of the cluster resources is observed;

[0026] If the state of the cluster resources reaches the expected state, the planning is completed and the planning control logic is generated, otherwise an error report is generated.

[0027] Preferably, in the index monitoring and analysis step, the index monitor executes the coping operation according to the monitored state, comprising:

[0028] When the disk overload is monitored, the PersistentVolumeClaim is expanded according to the defined rule;

[0029] When the creation of the model file is monitored, the inference service corresponding to the model file name is newly created;

[0030] When the deletion of the model file is monitored, the inference service corresponding to the model file name is unloaded;

[0031] When the update of the model file is monitored, the original inference service is replaced to keep the latest version.

[0032] An adaptive control device for intelligent operation and maintenance of a cloud-native cluster, comprising:

[0033] An operation and maintenance requirement analysis module is configured to analyze operation and maintenance requirements from a YAML format file, the YAML format file describing operation and maintenance requirements of a K8s cluster and containing resource objects and monitoring indexes for operation and maintenance monitoring;

[0034] A planning generator is configured to generate planning control logic and an index analyzer according to a rule trigger condition in the analyzed operation and maintenance requirements and operation and maintenance operations supported by the cluster.

[0035] The index collector generation module is used for determining an index collector type according to the index type and the index parameter in the parsed operation and maintenance requirement, and generating a corresponding index collector according to an index collector template.

[0036] The index monitor generation module is used for determining an index monitor to be created according to the parsed operation and maintenance requirement, and then combining the index collector and the index analyzer generated in the above steps to obtain the index monitor, automatically collecting indexes through the index collector, and determining an operation and maintenance planning logic corresponding to the operation and maintenance requirement through the index analyzer.

[0037] The adaptive controller is used for integrating the generated index monitor and the planning control logic, generating a code of a K8s controller, and deploying the code into a K8s cluster, so as to realize adaptive control of intelligent operation and maintenance of a cloud-native cluster.

[0038] The technical scheme of the present application has the following advantages:

[0039] 1. The system external observation index should meet the description of the specification, which can break the barrier from the user to the developer, and more clearly convey the operation and maintenance requirement.

[0040] 2. The operation and maintenance logic is automatically generated by using the operation and maintenance operation supported by the cluster, which can adapt to different deployment deployments and improve the compatibility of the operation and maintenance code.

[0041] 3. The more general operation and maintenance requirement can be described, the common operation and maintenance scene can be met, and the work burden of the developer can be reduced. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 The flowchart of the adaptive control method for the cloud-native cluster intelligent operation and maintenance of the present application.

[0043] Figure 2 The workflow diagram of the main module of the present application.

[0044] Figure 3 The flowchart of creating the index monitor according to the planning control logic of the present application. DETAILED DESCRIPTION

[0045] The present application will be further described in detail below in combination with the drawings, and the examples are only used to explain the present application, and are not used to limit the scope of the present application.

[0046] The embodiment discloses a self-adaptive control method for cloud-native cluster intelligent operation and maintenance, and a premise of executing the method is to create a domain-specific language for describing K8s cluster operation and maintenance requirements, which is used to describe operation and maintenance requirements, and subsequent processes are all processing requirements described by the language. The method is realized by a self-adaptive control system for K8s intelligent operation and maintenance, and the system includes basic modules such as an operation and maintenance requirement analysis module, a planning generator, an index collector generation module, an index monitor generation module and a self-adaptive controller. The specific description is as follows.

[0047] 1. Operation and maintenance requirement description: create a domain-specific language for describing K8s cluster operation and maintenance requirements

[0048] The language includes a formal definition for describing requirements and a requirement expression based on the YAML format, and the specific description is as follows.

[0049] (1) Formal definition method for user to describe requirements

[0050] In this embodiment, the state of the cluster at time t is defined as S t =<s t 1,s t 2,…,s t n >,wherein each element such as s t n represents the state of a certain resource in the cluster at time t.

[0051] For the state of the resource in the cluster, this embodiment further defines it as s=<L,V,P>, wherein L=<l1,l2,…l m > represents the condition for screening the resource, V=<v1,v2,…,v k represents the storage information related to the resource, and P=<p1,p2,…p x > represents the attribute that needs to be concerned in the resource. For L, the cluster resource can be specified by API group, type, name, belonging namespace, label list, annotation and the like. For V, the cluster resource can be mapped with the local storage system through mounting a Volume. It should be noted that K8s itself does not directly provide state monitoring of the internal file system of the resource, and in this embodiment, a controller and a corresponding Pod are used to monitor the storage resource concerned. For P, the specification of the cluster resource has various attributes in the form of key-value pairs; in addition, the running cluster resource also has a state, and this embodiment regards both as an attribute and monitors them uniformly.

[0052] Since there are a large number of resources in the K8s cluster, and the operation and maintenance behavior is only concerned about a part of them, the embodiment defines the difference between the cluster states before and after the operation and maintenance as a specific operation and maintenance requirement, that is, R = (S t- S0), representing the changes in resources in the cluster between the initial state and the desired state after the operation and maintenance is completed.

[0053] The embodiment defines a set of operation and maintenance operations supported by the cluster A = (a1, a2, …, a s The elements in the set are not commutative. K8s allows to add, delete, modify and query the resources in the cluster through the Restful API interface provided by the API server. In addition, for the local storage system, the controller of the embodiment also provides a corresponding interface for compatible modification of the resources therein.

[0054] For each operation a in the set A, it is defined as a tuple a = <rt, rs, op, p>, where rt represents the resource type to be operated, rs represents the resource selector, op represents the operation type to be executed, and p represents the parameter list when the operation is executed, which is empty by default.

[0055] According to the above formal definition, the operation and maintenance process of the K8s cluster is expressed as F(A) = R, which means that the specified operation and maintenance requirement is achieved through a combination of a series of operation and maintenance operations, and the combination is the operation and maintenance logic that needs to be generated (the related generation algorithm is described in detail below).

[0056] (2) Requirement expression based on YAML format

[0057] For the formal definition proposed above, in order to facilitate the use of developers and users, the embodiment is expressed through the YAML format, and the following is an example of a requirement description:

[0058]

[0059] This YAML code describes the following operation and maintenance requirements:

[0060] a. The resource object of the operation and maintenance monitoring, whose namespace is kubeflow-user-example-com, type is PersistentVolumeClaim (PVC), which is a persistent volume claim, and name is model-pvc;

[0061] b. The monitoring index of operation and maintenance is the disk space usage rate, which is an attribute of the resource and the final state expected by operation and maintenance. The index parameter (rule) is that the storage space usage rate declared in the PersistentVolumeClaim is always less than 60% of the total amount. The file name stored under the PersistentVolumeClaim is the same as the storageUri of the InferenceService resource attribute.

[0062] The workflow shown in FIG. 6 is performed on the language created above, and the specific description is as follows: Figures 1-2

[0063] 2. Operation and maintenance requirement analysis

[0064] The above YAML format operation and maintenance requirement description file needs to be input into the requirement description analysis module for analysis, from which the monitoring target and operation and maintenance monitoring index are identified, so as to facilitate the planning and generation of the subsequent index monitor and operation and maintenance logic.

[0065] 3. Operation and maintenance planning generation

[0066] The planning generator searches for a feasible operation and maintenance plan according to the specified operation and maintenance requirement and the operation and maintenance operation supported by the cluster. That is, according to the control action specified in the operation and maintenance requirement, the index trigger condition required is combined to generate the planning control logic, and the planning control logic of the adaptive controller is expressed in the form of a state machine in this embodiment. In this planning generator, the initial state and the expected state of the cluster are obtained according to the analysis of the requirement description, and the state jump required from the initial state to the expected state is analyzed, and then the adaptive controller is constructed. In addition, this embodiment supports manual specification of the operation and maintenance control action that should be performed to cope with more complex operation and maintenance scenarios and provide more flexible configuration capabilities.

[0067] The planning generator extracts the trigger condition contained in the specified operation and maintenance requirement, and the trigger condition describes the threshold that the monitoring index should reach when the rule is triggered, so as to generate the index analyzer. The index analyzer continuously checks the specific index and judges whether it has exceeded the threshold required by the trigger rule, and decides when the operation and maintenance plan and operation and maintenance operation should be performed.

[0068] The operation of the planning generator includes the following steps:

[0069] (1) Obtain the supported resource types and operation types in the K8s cluster, and expand them to support the modification of the storage state inside the resource;

[0070] (2) Obtain the expected operation and maintenance requirement, and filter the resource types related to the operation and maintenance requirement and the corresponding API (i.e. Restful API); ​

[0071] (3) generate random parameters, traverse the related API, observe the impact on the expected resource state, further filter out the API that can actually affect the target resource according to the impact, and adjust the generated parameters according to the feedback;

[0072] (4) If the parameter is adjusted in step (3) to make the cluster resource state reach the expectation (i.e., reach the expected state), stop planning and export the result; otherwise, combine the APIs of the resources and observe the impact on the target resource state.

[0073] (5) If the combination in step (4) can make the cluster resource state reach the expectation, complete the planning and export the result; otherwise, export an error report.

[0074] 4. Index collector generation

[0075] For the monitored index, the index type and index parameter obtained by the index collector generation module are parsed to determine the type of the index collector. This embodiment proposes a technology of generating an index collector based on an index collector template. For example, for an index related to disk capacity, a container that continuously checks the remaining disk space in the container can be created to collect such indexes. The index collector is generated by the index collector template according to the index type and index parameter, which is to cope with complex and changeable indexes in the actual production environment. The index collector template can be expanded in combination with the domain knowledge of the system to support more different types of indexes and improve the accuracy of the collector in collecting indexes.

[0076] 5. Index monitoring and analysis

[0077] The index monitoring and analysis are performed by an index monitor, which consists of two parts: one is an index collector, which can automatically collect and aggregate different types of indexes; the other is an index analyzer, which determines whether the expected operation and maintenance planning logic needs to be taken according to the specified operation and maintenance requirements, and updates the state of the cluster.

[0078] In specific execution, the corresponding index monitor is created by the index monitor generation module according to the operation and maintenance requirements, as shown in FIG. 8. Figure 3 When the number of index monitors is 0, the adaptive controller is in an uninitialized state, at which time the index monitor generation module creates an index monitor and judges whether the resource state to be monitored has been updated. When there is a corresponding change, jump to the corresponding state to perform the operation, as follows:

[0079] (1) When the disk overload is monitored, the PVC is expanded to meet the defined index;

[0080] (2) When monitoring the creation of a model file, a new inference service InferenceService corresponding to the model file name is created;

[0081] (3) When monitoring the deletion of a model file, the inference service corresponding to the model file name is unloaded;

[0082] (4) When monitoring the update of a model file, the original inference service is replaced by combining the unloading and new creation methods to keep the version up to date.

[0083] 6. K8s controller code deployment

[0084] The above generated index monitor and planning control logic are integrated by the adaptive controller to generate the code of the K8s controller and deploy it in the cluster. By monitoring specific indicators in the cluster, when the trigger condition is met, the planning control logic is executed to update the state of the cluster, achieving the purpose of adaptive control.

[0085] Experimental test:

[0086] In order to verify the technical solutions of the present application (hereinafter referred to as the method of the present application), the following tests are carried out in a K8s cluster built by 3 servers.

[0087] The configuration of the 3 servers is as follows: Ubuntu18.04 operating system, equipped with 8-core CPU, 32G memory, system disk size 200G, data disk 800G, the built K8s cluster is version 19.3; one server as master node, the other two servers as worker nodes.

[0088] Kubeflow1.14 is installed on the cluster, which is an open source platform for machine learning workflow management. It is based on Kubernetes and provides a set of tools and APIs to help data scientists, machine learning engineers and operation teams more efficiently manage machine learning workflow. Among them, Kubeflow allows users to deploy the trained model as an API service, for which a trained model file is needed. When the user trains a new model, a corresponding service needs to be created and published. Through the method of the present application, the corresponding service can be automatically published according to the trained model, only by specifying the following rules to be met:

[0089]

[0090] To this end, the requirement is described by YAML format, the adaptive controller identifies two indicators that need to be monitored, namely the internal file list of the PVC resource storing the trained model file and the published API service resource. At the same time, the regulation requires that for each trained model file, there is a corresponding API service resource with the same name. Therefore, by analyzing the supported operations of the cluster, the method constructs the required state transition automaton, which automatically creates, unloads and replaces the API service resource when the trained model is updated, ensuring that the regulation is met.

[0091] Finally, the method encapsulates the above-mentioned indicator monitoring and control logic as a K8s controller and deploys it to the cluster to make it effective.

[0092] Although the tool implementation example of the present application is disclosed for the purpose of illustration, which is intended to help understand the content of the present application and to implement it, those skilled in the art can understand that various alternatives, changes and modifications are possible without departing from the spirit and scope of the present application and the appended claims. Therefore, the present application should not be limited to the content disclosed in the best implementation example, and the scope of protection claimed by the present application is defined by the scope of the claims.

Claims

1. An adaptive control method for cloud-native cluster intelligent operation and maintenance, characterized in that, The method comprises the following steps: An operation and maintenance requirement description step: describing the operation and maintenance requirements of a K8s cluster through a YAML format file, including resource objects and monitoring indicators for operation and maintenance monitoring; An operation and maintenance requirement analysis step: analyzing the operation and maintenance requirements from the YAML format file; An operation and maintenance planning generation step: generating planning control logic and indicator analyzers according to the rule trigger conditions in the analyzed operation and maintenance requirements and the operation and maintenance operations supported by the cluster; The step of generating planning control logic comprises: when the operation and maintenance requirements are obtained, screening the resource types related to the operation and maintenance requirements and the API interfaces for adding, deleting, modifying and inquiring the resources in the cluster, the API interfaces being provided by an API server; generating random parameters, screening the API interfaces affecting the resources from the API interfaces, and adjusting the parameters according to the screened API interfaces; if the state of the cluster resources reaches the expected state after the parameters are adjusted, stopping the planning and generating the planning control logic, otherwise combining the API interfaces of the resources and observing the state of the cluster resources; if it is observed that the state of the cluster resources reaches the expected state, completing the planning and generating the planning control logic, otherwise generating an error report; An indicator collector generation step: determining the type of the indicator collector according to the indicator type and the indicator parameter in the analyzed operation and maintenance requirements, and generating the corresponding indicator collector according to the indicator collector template; An indicator monitoring and analyzing step: determining the indicator monitor to be created according to the analyzed operation and maintenance requirements, then combining the indicator collector and the indicator analyzer generated in the above step to obtain the indicator monitor, automatically collecting the indicators through the indicator collector, and determining the operation and maintenance planning logic corresponding to the operation and maintenance requirements through the indicator analyzer; A K8s controller code deployment step: integrating the indicator monitor and the planning control logic generated above, generating the code of the K8s controller, and deploying the code into the K8s cluster, so as to realize the adaptive control of the intelligent operation and maintenance of the cloud-native cluster.

2. The method of claim 1, wherein, In the operation and maintenance requirement description step, the language used in the YAML format file is defined as follows: A state S of the cluster at a certain time is defined, each element in S representing the state s of each resource in the cluster at the time; The operation and maintenance requirement R of the cluster is defined as the change amount of the resources in the cluster from the initial state S0 to the expected state St; A set A of operation and maintenance operations supported by the cluster is defined, each element in A representing an operation a.

3. The method of claim 2, wherein, The state s is defined to include a condition L for screening resources, storage information V related to the resources, and attributes P that need to be focused on in the resources.

4. The method of claim 3, wherein, L includes one or more of an API group, a type, a name, a namespace, a label list, and an annotation; V includes mounting Volume and local storage system mapping information; and P includes multiple types of key-value information.

5. The method of claim 2, wherein, The operation a is defined to include a resource type rt to be operated, a selector rs of the resource, an operation type op to be executed, and a parameter list p when the operation is executed.

6. The method of claim 1, wherein, In the indicator monitoring and analyzing step, the indicator monitor executes the corresponding operation according to the monitored state, including: When the disk overload is monitored, the PersistentVolumeClaim is expanded according to the defined rule; When the creation of the model file is monitored, a new inference service corresponding to the model file name is created; When the deletion of the model file is monitored, the inference service corresponding to the model file name is unloaded; When the update of the model file is monitored, the original inference service is replaced, and the version is kept up-to-date.

7. An adaptive control device for cloud-native cluster intelligent operation and maintenance, characterized in that, Comprise: An operation and maintenance requirement analysis module for analyzing operation and maintenance requirements from a YAML format file, which describes the operation and maintenance requirements of a K8s cluster, including resource objects and monitoring indicators for operation and maintenance monitoring; A planning generator for generating planning control logic and indicator analyzers according to rule trigger conditions in the parsed operation and maintenance requirements and operation and maintenance operations supported by the cluster; The steps of generating planning control logic include: when obtaining operation and maintenance requirements, filtering resource types related to operation and maintenance requirements and API interfaces for adding, deleting, modifying and querying resources in the cluster, which are provided by the API server; generating random parameters, selecting API affecting resources from the API, and adjusting the parameters according to the selected API; if the adjusted parameters make the state of the cluster resources reach the desired state, stop planning and generate planning control logic, otherwise combine the API of the resources and observe the state of the cluster resources; if the state of the cluster resources reaches the desired state, complete the planning and generate the planning control logic, otherwise generate an error report; An indicator collector generation module for determining the type of indicator collector according to the type and parameters of the indicators in the parsed operation and maintenance requirements, and generating the corresponding indicator collector according to the indicator collector template; An indicator monitor generation module for determining the indicator monitor to be created according to the parsed operation and maintenance requirements, then combining the indicator collector and the indicator analyzer generated in the above steps to obtain the indicator monitor, automatically collecting indicators through the indicator collector, and determining the operation and maintenance planning logic corresponding to the operation and maintenance requirements through the indicator analyzer; An adaptive controller for integrating the generated indicator monitor and planning control logic, generating the code of the K8s controller and deploying it to the K8s cluster, and realizing adaptive control for cloud-native cluster intelligent operation and maintenance.

Images