A method for realizing container resource customized scheduling and deletion protection based on Webhook technology

Abstract

The application belongs to the technical field of cloud computing, and provides a method for realizing container resource customized scheduling and deletion protection based on Webhook technology. Through the method, the following can be realized: automatically scheduling container resources to specified nodes for creation, ensuring that the resources on the tenant side do not preempt the resources on the management side when being created; adding protection annotations to the resources when creating resource objects; when deleting key resources of a cluster, deciding whether the resources can be deleted according to whether the protection features are enabled, so as to realize resource deletion protection. The application has the beneficial effects that: the container resource customized scheduling is realized, and the resources on the management side can be protected from being occupied by the resources on the tenant side without human intervention; the deletion protection is performed on the key resources in the cluster, so as to prevent the cluster from being unavailable due to the misdeletion of the key resources of the cluster; the nodes on which the tenants install the containers are specified, so as to meet the user-defined scheduling.

Description

TECHNICAL FIELD

[0001] The patent relates to the technical field of cloud computing, and particularly relates to a method for realizing container resource customized scheduling and deletion protection based on Webhook technology. BACKGROUND

[0002] In Kubernetes, custom resource types and custom controllers are provided to extend functions, and dynamic admission control is also provided, that is, admission control is realized through Webhook, which is divided into two types: Validating Admission Webhook and Mutating Admission Webhook. The calling principle of Webhook in Kubernetes is as follows: first, an Admission Webhook (Validating / Mutating) is registered in the Kubernetes cluster, when the request of a user reaches the Kubernetes Apiserver, the Apiserver configures the MutatingWebhookConfiguration and ValidatingWebhookConfiguration according to the configuration. The MutatingWebhookConfiguration is first called to modify the configuration file of the user request, and finally the ValidatingWebhookConfiguration is called to verify whether the modified configuration file is legal. When a resource is created or deleted, the configured Webhook is called to modify or verify, and finally it is persisted to ETCD. Admission Webhook is an HTTP callback that receives an Admission request and does some processing. Validating Webhook can reject the request, but cannot modify the object obtained in the admission request, Mutating Webhook can modify the object by creating a patch before returning the admission response, if the Webhook rejects a request, an error will be returned to the user.

[0003] With the wide application of custom resource types, the resource types in the cluster become various, for the operation personnel who are not familiar with the cluster resources, there is a risk that the key resources of the cluster are easily deleted, which causes the cluster to hang, therefore, a resource deletion protection mechanism is needed to ensure the security and stability of the cluster; in the cluster deployment process, the containers created by the tenant side are often automatically scheduled to the nodes of the management side, thereby occupying the resources of the management side and affecting the operation of the container cloud, in order to protect the nodes of the management side, the scheduling of the containers of the tenant side needs to be limited, so as to improve the stability and reliability of the system. SUMMARY

[0004] The application aims to provide a method for realizing container resource customized scheduling and deletion protection based on Webhook technology to solve the problems in the background art.

[0005] To achieve the above-mentioned purpose, the application provides the following technical solutions.

[0006] A method for realizing container resource customized scheduling and deletion protection based on Webhook technology, the method for realizing container resource customized scheduling and deletion protection based on Webhook technology designs a protection mechanism for deleting custom cloud resources, supports scheduling to a specified node when a container under a tenant-side namespace is created, and the like.

[0007] Parameters are set by a parameter configurator for use by a Webhook server.

[0008] A protected annotation (mutate method) is automatically added when a protected resource is created in a cluster, and a Mutating controller automatically adds a protection annotation to a resource to be created when a request for creating the resource is received.

[0009] When a request for creating a container under a tenant-side namespace is received, a node selector is automatically added to schedule the container resource to a specified node for creation.

[0010] After the protection annotation is added, deletion of the protected resource is refused (validate method), and a Validating controller controls whether to refuse a request for deleting a cluster key resource according to whether a protection feature is enabled and whether the protection annotation is present when processing the request.

[0011] The method for realizing container resource customized scheduling and deletion protection comprises the following steps.

[0012] Step 1: Configure a Kubernetes cluster to support an Admission Webhook plug-in function.

[0013] Step 2: Deploy a Webhook server and enable the function of the Webhook.

[0014] Step 3: Use the deletion protection and customized scheduling function of the Webhook.

[0015] Preferably, the custom cloud resources include custom virtual machine resources, custom cluster resources, custom node resources, custom components, custom component upgrade resources, custom component groups, and custom component upgrade groups.

[0016] Preferably, the management side namespace includes the namespace of important clusters such as k8s cluster, network cluster, storage cluster, monitoring cluster, openstack cluster and the like which are necessary for building platform, and the non-management side namespace is the tenant side namespace, and the management side namespace supports configuration to ManagedNameSpace.

[0017] Preferably, when the DaemonSet, Deployment, ReplicaSet, Pod, StatefulSet, Job, CronJob resources under the tenant side namespace are created, the value of the TenantNodeSelectors configuration is specified to the nodeselector field of the corresponding resource.

[0018] Preferably, the process of step one includes:

[0019] S1: Ensure that the Kubernetes APIServer starts MutatingAdmissionWebhook and ValidatingAdmissionWebhook, check the starting parameters of the Apiserver, and the following configuration exists:

[0020] --enable-admission-plugins=MutatingAdmissionWebhook, ValidatingAdmissionWebhook;

[0021] S2: Check whether the admission registration API is enabled in the cluster:

[0022] $kubectl api-versions|grep admission

[0023] admissionregistration.k8s.io / ${version};

[0024] S3: Create RBAC, since the Webhook will modify the resource, a ServiceAccount needs to be created separately, which can be directly created in the Kubernetes cluster;

[0025] S4: Certificate authentication, the Kubernetes cluster is HTTPS communication by default, and the process of Apiserver calling Webhook is also HTTPS, so certificate authentication is needed. Certificate authentication is equivalent to authenticating the domain name of the Service. The domain name of the Service is placed in the authentication request server.csr file, and then a K8S certificate signing request resource CertificateSigningRequest is created. After the APIServer signs the certificate, server-cert.pem is generated. Then the initially created private key server-key.pem and the signed certificate server-cert.pem are placed in the Secret for Deployment to call;

[0026] S5: Create service, the certificate above is issued to the domain name of the Service;

[0027] S6: Deploy Admission Deployment, ServiceAccount and Secret are created in the above two steps, mount the certificate stored in Secret, and the image is the Admission image compiled by us.

[0028] Preferably, the process of deploying the Webhook Server and enabling the function of the Webhook in step two comprises:

[0029] D1: Ensure that the Webhook service is running and can receive requests from the Apiserver;

[0030] D2: Deploy Validating Webhook, configure the yaml file of ValidatingWebhookConfiguration, pay attention to the path in clientConfig, which is / validate. Configure some RBAC rules according to the needs, create a Validating Webhook configuration object in the cluster, which will dynamically add the Webhook to the Webhook chain. Once the resource is deleted, the request will be intercepted and then the Webhook service will be called;

[0031] D3: Deploy Mutating Webhook, need to configure MutatingWebhookConfiguration's yaml file, need to pay attention to the path path in clientConfig is / mutate, configure some RBAC rules according to the needs, create Mutating Webhook configuration object in the cluster, dynamically add Webhook to the Webhook chain, once the resource is created, it will intercept the request and then call the Webhook service.

[0032] Preferably, the process of using the deletion protection and custom scheduling function of the Webhook in step three includes:

[0033] N1: Use the deletion protection function;

[0034] Deploy a custom virtual machine resource that can be protected in the cluster, check the manifest file for the existence of protected annotations after creation, and the EnabledValidator parameter of Admission has the resource type, and the deletion of this custom virtual machine resource will be rejected;

[0035] N2: Use the container resource custom scheduling function;

[0036] Set MutateNodeSelectorGate of Admission to true, ManagedNameSpace configures the management side's namespace, and TenantNodeSelectors saves a node label, which is the node that needs to schedule resources to create, create a pod in the cluster, whose namespace does not belong to the management side's namespace, will be automatically scheduled to the node with the specified label.

[0037] Preferably, the Webhook server is an HTTP service with TLS authentication, deployed in the cluster using Deployment;

[0038] Validating Webhook and Mutating Webhook verification are performed by listening to two different HTTP paths (validate and mutate), after receiving an HTTP request, deserializing the AdmissionReview object from the request, performing some basic content verification, calling the corresponding Mutating and Validating methods according to the URL path, and serializing the AdmissionReview object.

[0039] Preferably, the Validating controller checks whether the resource object needs deletion protection;

[0040] The resource type needing to enable the deletion protection feature is set through the EnabledValidator parameter, when the resource type of the request for deletion is a key resource in the cluster, such as a custom virtual machine resource, a custom cluster resource, a custom node resource, a custom component, a custom component upgrade resource, a custom component group, a custom component upgrade group, etc.; if the protected annotation is also carried, the deletion is refused, and the failure reason is written in the response, and the end user will receive the information of deletion failure.

[0041] Preferably, the Mutating controller processes the creation resource request;

[0042] When the MutateNodeSelectorGate is set to true, it indicates that the container resource customization scheduling feature is enabled; if the resource type is one of a Pod, a DaemonSet, a Deployment, a ReplicaSet, a StatefulSet, a Job and a CronJob, it is checked whether the namespace of the resource belongs to the management side namespace stored in the ManagedNameSpace; if it is the management side namespace, no control is performed; if it is the tenant side namespace, the node selector needs to be automatically set to schedule the container to the specified node; if the resource type is a custom virtual machine resource, a custom cluster resource, a custom node resource, a custom component, a custom component upgrade resource, a custom component group, a custom component upgrade group, etc., the custom resource is added with the protection annotation.

[0043] Compared with the prior art, the present application has the beneficial effects that:

[0044] The present application realizes the automatic scheduling of the tenant side container to the specified node by adopting the container resource customization scheduling and deletion protection, and protects the management side resource from being occupied; the deletion protection is performed on the resource types of a custom virtual machine resource, a custom cluster resource, a custom node resource, a custom component, a custom component upgrade resource, a custom component group, a custom component upgrade group, etc., to prevent the misdeletion of the key resources of the cluster from causing the cluster to be unavailable; the node resource of the specified installation container under the tenant side namespace is supported, and the user-defined scheduling is met. BRIEF DESCRIPTION OF DRAWINGS

[0045] Fig. 1 The system flowchart of the present application is shown in the figure;

[0046] Fig. 2 The parameter configurator diagram of the present application is shown in the figure. DETAILED DESCRIPTION

[0047] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0048] Please refer to Figs. 1-2 The present application provides a technical solution:

[0049] Some special configurations are automatically added by using the Mutating mechanism of the Admission Webhook, without the user's care. In the Validating, the code is written to set the verification rule, and the request is checked to be legal. In order to realize the automatic scheduling of the container resource to the specified node creation on the tenant side and realize the deletion protection of the key resources of the cluster, the present application designs a method for realizing the customized scheduling and deletion protection of the container resource based on the Webhook technology.

[0050] The overall design scheme is:

[0051] The parameter configurator is used to set the parameters for the Webhook server;

[0052] A protected annotation is automatically added (mutate method) when the protected resource is created in the cluster. The Mutating controller automatically adds the protected annotation to the to-be-created resource when receiving the request for creating the resource;

[0053] When receiving the request for creating the container under the namespace on the tenant side, the node selector is automatically added to schedule the container resource to the specified node for creation;

[0054] After the protected annotation is added, the protected resource is refused to be deleted (validate method). The Validating controller controls whether to refuse the deletion request according to whether the protection feature is enabled and whether the protected annotation exists when processing the request for deleting the key resource of the cluster.

[0055] The main parameter meanings are:

[0056] EnabledValidator: This parameter sets the key resources of the cluster that need to be protected from deletion, such as custom virtual machine resources and custom cluster resources;

[0057] MutateNodeSelectorGate: This parameter sets whether to enable the customized scheduling function of the container resource;

[0058] ManagedNameSpace: This parameter only makes sense when MutateNodeSelectorGate is enabled, which stores the namespace on the management side of the cluster;

[0059] TenantNodeSelectors: This parameter only makes sense when MutateNodeSelectorGate is enabled, which specifies the labels of the nodes that the cluster tenant side expects to be scheduled to.

[0060] The Webhook server design scheme is as follows:

[0061] The Webhook server is an HTTP service with TLS authentication, deployed in the cluster by Deployment, and performs Validating Webhook and Mutating Webhook verification by listening to two different HTTP paths (validate and mutate). After receiving an HTTP request, it deserializes the AdmissionReview object from the request, performs some basic content verification, calls the corresponding Mutating and Validating methods according to the URL path, and serializes the AdmissionReview object.

[0062] The Validating controller design scheme is as follows:

[0063] The Validating controller checks whether the resource object needs deletion protection;

[0064] The EnabledValidator parameter is used to set the resource types that need to enable deletion protection features. When the resource type being deleted is a critical resource in the cluster, such as a custom virtual machine resource, a custom cluster resource, a custom node resource, a custom component, a custom component upgrade resource, a custom component group, a custom component upgrade group, etc., if it also has a protected annotation, it will refuse to delete and write the failure reason in the response. The end user will receive a deletion failure message.

[0065] The Mutating controller design scheme is as follows:

[0066] The Mutating controller handles creation resource requests;

[0067] When MutateNodeSelectorGate is set to true, it means that the container resource customization scheduling feature is enabled;

[0068] If the resource type is one of Pod, DaemonSet, Deployment, ReplicaSet, StatefulSet, Job, CronJob, check whether the namespace of the resource belongs to the management side namespace stored in ManagedNameSpace; if it is a management side namespace, no control is performed; if it is a tenant side namespace, the node selector needs to be automatically set to schedule the container to the specified node; if the resource type is a custom virtual machine resource, a custom cluster resource, a custom node resource, a custom component, a custom component upgrade resource, a custom component group, a custom component upgrade group, etc., a protection annotation is added to the custom resource.

[0069] The method of container resource customized scheduling and deletion protection designs a protection mechanism for deleting custom cloud resources. For containers under the tenant side namespace, it supports scheduling to the specified node when created, realizes automatic scheduling of tenant side containers to the specified node, and protects the management side resources from being occupied; for custom virtual machine resources, custom cluster resources, custom node resources, custom components, custom component upgrade resources, custom component groups, custom component upgrade groups, and other resource types, deletion protection is performed to prevent misdeletion of critical cluster resources from causing the cluster to be unavailable; it supports specifying the node resources of the installed container under the tenant side namespace, meeting the user's custom scheduling.

[0070] The method of container resource customized scheduling and deletion protection includes the following steps:

[0071] Step one, configure the Kubernetes cluster to support the Admission Webhook plug-in function:

[0072] S1: Ensure that the Kubernetes APIServer has enabled MutatingAdmissionWebhook and ValidatingAdmissionWebhook. Check the startup parameters of the Apiserver. If the following configuration exists, it can be used:

[0073] --enable-admission-plugins=MutatingAdmissionWebhook, ValidatingAdmissionWebhook;

[0074] S2: Check whether the admission registration API is enabled in the cluster:

[0075] $ kubectl api-versions | grep admission

[0076] admissionregistration.k8s.io / ${version};

[0077] S3: Create RBAC, since the Webhook will modify the resource, a ServiceAccount needs to be created separately, which can be created directly in the Kubernetes cluster;

[0078] S4: Certificate authentication, Kubernetes cluster is HTTPS communication by default, and the process of Apiserver calling Webhook is also HTTPS, which requires certificate authentication. Certificate authentication is equivalent to authenticating the domain name of the Service. Put the Service domain name into the authentication request server.csr file, then create a K8S certificate signing request resource CertificateSigningRequest. After the APIServer signs the certificate, it generates server-cert.pem. Then put the initially created private key server-key.pem and the signed certificate server-cert.pem into Secret for Deployment to call;

[0079] S5: Create service, the certificate above is issued to the domain name of the Service;

[0080] S6: Deploy Admission Deployment, ServiceAccount and Secret are created in the above two steps, mount the certificate stored in Secret, and the image is the Admission image compiled by us.

[0081] Step two, deploy Webhook server and enable Webhook function:

[0082] D1: Ensure that the Webhook service is running and can receive requests from the Apiserver;

[0083] D2: Deploy Validating Webhook, configure the yaml file of ValidatingWebhookConfiguration, pay attention to the path in clientConfig, which is / validate. Configure some RBAC rules according to needs, create Validating Webhook configuration object in the cluster, which will dynamically add Webhook to the Webhook chain. Once the resource is deleted, it will intercept the request and then call the Webhook service;

[0084] D3: Deploy Mutating Webhook, need to configure MutatingWebhookConfiguration yaml file, need to pay attention to the path path in clientConfig is / mutate, configure some RBAC rules according to the needs, create Mutating Webhook configuration object in the cluster, which will dynamically add Webhook to the Webhook chain, once the resource is created, it will intercept the request and then call the Webhook service.

[0085] Step three, use the deletion protection and custom scheduling functions of Webhook:

[0086] N1: Use the deletion protection function;

[0087] Deploy a custom virtual machine resource that can be protected in the cluster, check the manifest file for the existence of protected annotations after creation, and the EnabledValidator parameter of Admission has the resource type, and the deletion of this custom virtual machine resource will be rejected;

[0088] N2: Use the container resource custom scheduling function;

[0089] Set MutateNodeSelectorGate of Admission to true, ManagedNameSpace to configure the management side namespace, and TenantNodeSelectors to save a node label, which is the node that needs to be scheduled for resource creation, create a pod in the cluster whose namespace does not belong to the management side namespace, which will automatically be scheduled to the node with the specified label

[0090] Custom cloud resources include: custom virtual machine resources, custom cluster resources, custom node resources, custom components, custom component upgrade resources, custom component groups, and custom component upgrade groups.

[0091] Management side namespace includes: k8s cluster, network cluster, storage cluster, monitoring cluster, openstack cluster and other important cluster namespaces that must be built on the platform, non-management side namespace is the tenant side namespace, and supports configuring the management side namespace to ManagedNameSpace.

[0092] When creating DaemonSet, Deployment, ReplicaSet, Pod, StatefulSet, Job, CronJob resources under the tenant side namespace, the value of TenantNodeSelectors configuration is specified to the nodeselector field of the corresponding resource.

[0093] While embodiments of the present application have been shown and described with reference to particular embodiments thereof, it will be understood by those skilled in the art that various changes in form and details can be made therein without departing from the spirit and scope of the application as defined by the appended claims and their equivalents.

Claims

1. A method for customized scheduling and deletion protection of container resources based on Webhook technology, characterized in that: The method for customized scheduling and deletion protection of container resources is designed with a protection mechanism for deleting customized cloud resources. For containers under the tenant-side namespace, it supports scheduling to a specified node when they are created. Parameters are set via the parameter configurator for use by the Webhook server; The mutate method automatically adds a protected annotation when creating a protected resource in the cluster. When the mutating controller receives a request to create a resource, it automatically adds the protected annotation to the resource to be created. When a request to create a container under a tenant's namespace is received, a node selector is automatically added to schedule container resources to the specified node for creation. The validate method: When a protection annotation is present, the deletion of protected resources is rejected. When processing requests to delete critical cluster resources, the Validating controller controls whether to reject the deletion request based on whether the protection feature is enabled and whether a protection annotation is present. The method for custom scheduling and deletion protection of container resources includes the following steps: Step 1: Configure the Kubernetes cluster to support the Admission Webhook plugin: Step 2: Deploy the Webhook server and enable Webhook functionality: Step 3: Use the Webhook's deletion protection and customized scheduling features; The custom cloud resources include: custom virtual machine resources, custom cluster resources, custom node resources, custom components, custom component upgrade resources, custom component groups, and custom component upgrade groups; The management-side namespace includes the namespaces for the Kubernetes cluster, network cluster, storage cluster, monitoring cluster, and OpenStack cluster, all necessary for building the platform. The non-management-side namespace is the tenant-side namespace, which supports configuring the management-side namespace to ManagedNameSpace; When creating DaemonSet, Deployment, ReplicaSet, Pod, StatefulSet, Job, and CronJob resources under the tenant-side namespace, specify the value of the TenantNodeSelectors configuration to the nodeselector field of the corresponding resource; Step one describes the process of configuring the Kubernetes cluster to support the Admission Webhook plugin, which includes: S1: Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook are enabled on the Kubernetes APIServer. Check the APIServer startup parameters; the following configuration should be present: --enable-admission-plugins= MutatingAdmissionWebhook, ValidatingAdmissionWebhook; S2: Check if the admission registration API is enabled in the cluster: $ kubectl api-versions |grep admission admissionregistration.k8s.io / ${version}; S3: Create an RBAC. Create a separate ServiceAccount, which can be done directly in the Kubernetes cluster. S4: Certificate Authentication: Authenticate the domain name of the Service. Put the Service domain name in the authentication request server.csr file, then create a K8S certificate signing request resource CertificateSigningRequest. After the APIServer signs the certificate, it generates server-cert.pem. Then put the initially created private key server-key.pem and the signed certificate server-cert.pem into the Secret for Deployment to call. S5: Create a service; the certificate above is issued to the domain name of this service. S6: Deploy Admission Deployment. ServiceAccount and Secret are created in the previous two steps, respectively. Mount the certificate stored in Secret. The image is the compiled Admission image. The process of deploying the Webhook Server and enabling Webhook functionality as described in step two includes: D1: Ensure the Webhook service is running and receiving requests from the Apiserver; D2: To deploy Validating Webhook, you need to configure the ValidatingWebhookConfiguration yaml file. Note that the path in clientConfig is / validate. Configure RBAC rules as needed. Create a Validating Webhook configuration object in the cluster. The Webhook will be dynamically added to the Webhook chain. Once the resource is deleted, the request will be intercepted and the Webhook service will be called. D3: Deploying Mutating Webhook requires configuring the MutatingWebhookConfiguration yaml file. Note that the path in clientConfig is / mutate. Configure RBAC rules as needed. Create a Mutating Webhook configuration object in the cluster. This will dynamically add the Webhook to the Webhook chain. Once the resource is created, it will intercept requests and then call the Webhook service. Step three describes the process of using Webhook's deletion protection and customized scheduling features, which includes: N1: Use deletion protection function; Deploy a custom virtual machine resource that can be protected in the cluster. After successful creation, check if there is a protected annotation in the manifest file. If the EnabledValidator parameter of Admission contains this resource type, then deleting this custom virtual machine resource will be rejected. N2: Customize scheduling functionality using container resources; Set Admission's MutateNodeSelectorGate to true, ManagedNameSpace to configure the management-side namespace, and TenantNodeSelectors to store a node label. This node needs to be created by scheduling resources. When a pod is created in the cluster, if its namespace does not belong to the management-side namespace, it will be automatically scheduled to a node with a specified label. The Webhook server is an HTTP service with TLS authentication, deployed in the cluster using the Deployment method; Validating Webhook and Mutating Webhook are performed by listening to two different HTTP paths. Upon receiving an HTTP request, an AdmissionReview object is deserialized from the request, some basic content validation is performed, the corresponding Mutating and Validating methods are called according to the URL path, and the AdmissionReview object is serialized. The Validating controller checks whether the resource object needs to be protected against deletion; The EnabledValidator parameter sets the types of resources for which deletion protection needs to be enabled. When the requested deletion resource type is a critical resource in the cluster, it includes custom virtual machine resources, custom cluster resources, custom node resources, custom components, custom component upgrade resources, custom component groups, and custom component upgrade groups. Meanwhile, annotations that are protected will refuse deletion and will write the reason for failure in the response, so that the end user will receive a message that the deletion failed; When the Mutating controller processes a resource creation request; When MutateNodeSelectorGate is set to true, it indicates that the container resource customization scheduling feature is enabled. If the resource type is one of Pod, DaemonSet, Deployment, ReplicaSet, StatefulSet, Job, or CronJob, check whether the namespace of the resource belongs to the management-side namespace stored in ManagedNameSpace. If it is the management-side namespace, no control is performed. If it is the tenant-side namespace, the node selector needs to be automatically set to schedule the container to the specified node. If the resource type is a custom virtual machine resource, custom cluster resource, custom node resource, custom component, custom component upgrade resource, custom component group, or custom component upgrade group, add a protection annotation to the custom resource.

Images