A method for implementing multi-MGR service deployment based on rook

Abstract

The present invention particularly relates to a method for implementing multi-MGR service deployment based on rook. The method for implementing multi-MGR service deployment based on rook modifies the definition of the API model to allow the user to define the number of MGRs; the controller maintains the MGR service of the current Ceph cluster according to the number specified by the user; each POD running the MGR service periodically accesses the Ceph service, obtains its own status, and updates the definition of the Service according to the situation. The method for implementing multi-MGR service deployment based on rook supports the deployment of multiple MGR services by expanding the deployment model of Rook, which can minimize the offline time of the MGR service and maximize the high availability of the MGR service.

Description

Technical Field

[0001] The present invention relates to the field of cloud computing technology, and in particular to a method for implementing multi-MGR service deployment based on rook. Background Art

[0002] Kubernetes is an open source container orchestration engine for automated deployment, scaling, and management of containerized applications.

[0003] Service is a term in Kubernetes, which is used to expose services running in PODs to other applications running in the same Kubernetes cluster. Service actually implements a dynamic load balancer, which selects services that meet the definition by matching the labels of PODs running in the cluster, and randomly selects an available backend through simple load balancing.

[0004] Ceph is a distributed storage system that natively provides centralized and commonly used storage services including block storage, file storage, and object storage. Its design goal is to use commercial hardware devices to build a large-scale, highly available, highly scalable, and high-performance distributed storage system. By adopting decentralized metadata storage and multi-copy storage strategies, Ceph can support PB-level storage capacity. When some components or nodes of the system are abnormal, Ceph can still provide services through data redundancy to maintain system availability. MGR is the abbreviation of manager. MGR is a module of Ceph, responsible for maintaining runtime monitoring data and cluster status information. MGR's high availability adopts a master-slave mode, and only one service can handle client requests at the same time.

[0005] Rook is an open source distributed storage orchestration system in a cloud-native environment; it can transform distributed storage software into a self-managing, self-scaling, and self-healing storage service. Currently, Rook has been increasingly used as a solution for containerized deployment of Ceph. However, a known problem is that Rook currently cannot support the deployment of multiple MGR services. There are several main problems:

[0006] 1) When the node where the MGR service is running is abnormally shut down or the node is unavailable, the service often cannot be quickly migrated to an available node.

[0007] 2) In the case of single MGR service, normal service management, such as configuration updates, will increase the length of service unavailability time.

[0008] 3) In the case of a single MGR service, if there is a bug in the MGR service code itself, if the program is deadlocked, the MGR service may become known to be unavailable and require human intervention.

[0009] A current compromise is to add a running check to the MGR container, that is, to configure a livenessprobe strategy for the POD, to detect the availability of the service by periodically calling the service interface, and to restart the POD when a service anomaly is detected. This method can alleviate the problem to a certain extent, but there are also problems, that is, the recovery time of the MGR service depends on the restart speed of the POD. When a problem occurs on the node where the POD is running, the POD often cannot recover quickly, resulting in a long period of unavailability of the MGR service.

[0010] In view of the above situation, the present invention proposes a method for implementing multi-MGR service deployment based on rook. Summary of the invention

[0011] In order to make up for the defects of the prior art, the present invention provides a simple and efficient method for implementing multi-MGR service deployment based on rook.

[0012] The present invention is achieved through the following technical solutions:

[0013] A method for implementing multi-MGR service deployment based on rook, characterized in that it includes the following steps:

[0014] The first step is to modify the definition of the API model to allow users to define the number of MGRs;

[0015] In the second step, the controller maintains the MGR service of the current Ceph cluster according to the number specified by the user;

[0016] In the third step, each POD running the MGR service periodically accesses the Ceph service to obtain its own status and updates the Service definition as needed.

[0017] In the first step, the API model is modified by modifying the definition of CustomResourceDefinition (CRD, custom resource definition) and naming it as mgr attribute, adding a count parameter, and the count parameter is used to specify the number of MGR services of the currently defined Ceph cluster.

[0018] In the first step, add the count parameter configuration in the CustomResource (CR, custom resource) attribute of the Ceph cluster to specify the number of desired MGR services.

[0019] In the second step, the controller maintains the Ceph cluster MGR service status. The specific process is as follows:

[0020] 2) The controller monitors the status of the CustomResource attributes of the Ceph cluster through the watch listening method provided by the Kubernetes client, and filters the changes of the CustomResource attributes through the custom predicate function;

[0021] 2) When the count parameter is monitored to change, first determine the number of MGR services specified by the count parameter:

[0022] When the number of MGR services specified by the count parameter is 1, there is no need to change the existing logic. Just point the Service defined in kubernetes to the POD with the label app=rook-ceph-mgr. At the same time, the controller deletes the redundant MGR services and only keeps the PODs of the MGR services with lower ordinal numbers, or deletes the container used for control in the POD and restarts the POD.

[0023] When the number of MGR services specified by the count parameter is greater than 1, the controller performs the following processing:

[0024] S1. Create a deployment definition for a specified number of MGRs and define the deployment names in sequence;

[0025] S2. Add the id option to the startup command of the MGR container and specify the serial number of the current deployment to associate the MGR service with the deployment. Ensure that the label of the POD defined in the deployment contains this serial number to ensure that all the labels of the POD can be combined to specify a unique POD.

[0026] S3. Add a new container based on the deployment definition to synchronize the MGR service status in the Ceph cluster, and modify the Service definition as needed.

[0027] In step S3, the implementation method of adding a new container is as follows:

[0028] S31. First, add permissions to the ServiceAccount (service account) specified by the container, so that it can perform CRUD (Create, Retrieve, Update, and Delete) and patch operations on the Service under the current namespace;

[0029] S32. The configuration file and keyring file of the Ceph cluster are pre-mounted in the container, or the configuration file is automatically generated by obtaining the configuration file, so as to call the interface of the Ceph cluster in the container and obtain the status of the Ceph cluster;

[0030] Automatically obtain and generate the configuration file. The specific process is as follows:

[0031] S321. Add the permission for the ServiceAccount user of the newly added container to access the Secret encrypted content and ConfigMap (an API object used to save non-encrypted data in a key-value pair) under the same namespace, allowing it to obtain the content therein;

[0032] S322. Read the contents of rook-ceph-admin-keyring and rook-ceph-mon-endpoints to obtain the keyring used by the admin and the access address of ceph mon;

[0033] S323. Referring to the standard form of the configuration file of the Ceph cluster, assemble the acquired data according to the requirements of the Ceph cluster and write it to the specified location of the container;

[0034] S324. When calling the ceph command, specify the configuration file to the created configuration file to call the API of the Ceph cluster in the newly added sidecar container.

[0035] In step S3, the MGR service status in the Ceph cluster is synchronized in the container, and the implementation method is as follows:

[0036] S31. After generating the configuration file, you can call the ceph command in the container to obtain the MGR service status. By executing ceph-s-format json-pretty|jq'.mgrmap.activename', you can obtain the name of the currently active (running and able to process requests) MGR service.

[0037] S32. Compare the name of the active MGR service in the container with the sequence number defined in the deployment:

[0038] If the current serial number is the same as the serial number of the active MGR service, the POD where the current MGR service is located is the POD that the Service needs to point to;

[0039] If the sequence number of the currently active MGR service is not equal to the sequence number of the current container, the loop is skipped once;

[0040] If the ordinal number of the currently active MGR service is equal to the ordinal number of the current container, it means that the backend pointed to by the Service needs to be updated, that is, the selector of the Service is defined as all labels of the current POD. By accessing the combination of the ClusterIP (cluster IP) of the MGR service and the defined port, the http or tcp service provided by the currently active MGR service can be accessed;

[0041] S33. The container of the inspection service needs to run in a loop, and check and obtain the status of the MGR in Ceph once every custom cycle.

[0042] In step S3, when an application outside the Kubernetes cluster accesses the MGR service, the type of Service needs to be configured. The implementation method is as follows:

[0043] S31. Modify the definition of dashboard in CustomResourceDefinition of Ceph cluster, add three attributes: accessPort, serviceType and accessIP, where the three attributes are of integer type, string type and string type respectively;

[0044] S32. When defining a Ceph cluster, the user chooses to configure a combination of accessPort, serviceType, and accessIP;

[0045] S33. After the controller captures the modified information of CustomResource through the interface provided by the Kubernetes client, it starts to synchronize the cluster status. The work content is as follows:

[0046] S331. Determine the number of MGR services in the cluster:

[0047] When the number of MGR services specified by the count parameter is 1, the type of service is modified directly in the controller;

[0048] When the number of MGR services specified by the count parameter is 2, modify the definition of the deployment according to the configuration content, generate the serviceType, accessPort and accessIP parameters into the definition of the deployment through environment variables, and update the current deployment;

[0049] S332. After the container of the MGR service is started, the above configuration items are obtained through environment variables, and then the definition of the Service is tried to be obtained through the interface provided by the kubenretes client:

[0050] If the current Service has not been successfully created, generate an MGR service of the specified type according to the configuration items provided in the environment variables and fill in the relevant configuration;

[0051] If a service with the same name already exists, it means that the service already exists. Update its content and change the type of the service to the required type.

[0052] S333. Steps S331 and S332 are repeated to ensure that the switching of the MGR service will eventually be synchronized with the definition of the Service.

[0053] The beneficial effects of the present invention are: the method for implementing multi-MGR service deployment based on Rook supports the deployment of multiple MGR services by expanding the deployment model of Rook, which can minimize the offline time of the MGR service and ensure the high availability of the MGR service to the greatest extent possible. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0055] Attached Figure 1 A schematic diagram of a method for maintaining the MGR service status of a Ceph cluster by a controller of the present invention.

[0056] Attached Figure 2 A schematic diagram of a method for synchronizing the MGR service status in a Ceph cluster in a container according to the present invention. DETAILED DESCRIPTION

[0057] In order to enable those skilled in the art to better understand the technical solutions in the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0058] The method for implementing multi-MGR service deployment based on rook includes the following steps:

[0059] The first step is to modify the definition of the API model to allow users to define the number of MGRs;

[0060] In the second step, the controller maintains the MGR service of the current Ceph cluster according to the number specified by the user;

[0061] In the third step, each POD running the MGR service periodically accesses the Ceph service to obtain its own status and updates the Service definition as needed.

[0062] In the first step, the API model is modified by modifying the definition of CustomResourceDefinition (CRD, custom resource definition) and naming it as mgr attribute, adding a count parameter, and the count parameter is used to specify the number of MGR services of the currently defined Ceph cluster.

[0063] In the first step, add the count parameter configuration in the CustomResource (CR, custom resource) attribute of the Ceph cluster to specify the number of desired MGR services.

[0064] In the second step, the controller maintains the Ceph cluster MGR service status. The specific process is as follows:

[0065] 3) The controller monitors the status of the CustomResource attribute of the Ceph cluster through the watch listening method provided by the Kubernetes client, and filters the changes of the CustomResource attribute through the custom predicate function;

[0066] 2) When the count parameter is monitored to change, first determine the number of MGR services specified by the count parameter:

[0067] When the number of MGR services specified by the count parameter is 1, there is no need to change the existing logic. Just point the Service defined in kubernetes to the POD with the label app=rook-ceph-mgr. At the same time, the controller deletes the redundant MGR services and only keeps the PODs of the MGR services with lower ordinal numbers, or deletes the container used for control in the POD and restarts the POD.

[0068] When the number of MGR services specified by the count parameter is greater than 1, the controller performs the following processing:

[0069] S1. Create a deployment definition for a specified number of MGRs and define the names of the deployments in sequence, i.e. the first one is named a, and so on;

[0070] S2. Add the id option to the startup command of the MGR container and specify the serial number of the current deployment to associate the MGR service with the deployment. Ensure that the label of the POD defined in the deployment contains this serial number to ensure that all the labels of the POD can be combined to specify a unique POD.

[0071] S3. Add a new container based on the deployment definition to synchronize the MGR service status in the Ceph cluster, and modify the Service definition as needed.

[0072] In step S3, the implementation method of adding a new container is as follows:

[0073] S31. First, add permissions to the ServiceAccount (service account) specified by the container, so that it can perform CRUD (Create, Retrieve, Update, and Delete) and patch operations on the Service under the current namespace;

[0074] S32. The configuration file and keyring file of the Ceph cluster are pre-mounted in the container, or the configuration file is automatically generated by obtaining the configuration file, so as to call the interface of the Ceph cluster in the container and obtain the status of the Ceph cluster;

[0075] Automatically obtain and generate the configuration file. The specific process is as follows:

[0076] S321. Add the permission for the ServiceAccount user of the newly added container to access the Secret encrypted content and ConfigMap (an API object used to save non-encrypted data in a key-value pair) under the same namespace, allowing it to obtain the content therein;

[0077] S322. Read the contents of rook-ceph-admin-keyring and rook-ceph-mon-endpoints to obtain the keyring used by the admin and the access address of ceph mon;

[0078] S323. Referring to the standard form of the configuration file of the Ceph cluster, assemble the acquired data according to the requirements of the Ceph cluster and write it to the specified location of the container;

[0079] S324. When calling the ceph command, specify the configuration file to the created configuration file to call the API (API: Application Program Interface) of the Ceph cluster in the newly added sidecar container.

[0080] In step S3, the MGR service status in the Ceph cluster is synchronized in the container, and the implementation method is as follows:

[0081] S31. After generating the configuration file, you can call the ceph command in the container to obtain the MGR service status. By executing ceph-s-format json-pretty|jq'.mgrmap.activename', you can obtain the name of the currently active (running and able to process requests) MGR service.

[0082] S32. Compare the name of the active MGR service in the container with the sequence number defined in the deployment:

[0083] If the current serial number is the same as the serial number of the active MGR service, the POD where the current MGR service is located is the POD that the Service needs to point to;

[0084] If the sequence number of the currently active MGR service is not equal to the sequence number of the current container, the loop is skipped once;

[0085] If the ordinal number of the currently active MGR service is equal to the ordinal number of the current container, it means that the backend pointed to by the Service needs to be updated, that is, the selector of the Service is defined as all the labels of the current POD. Note that as mentioned above, the label can ensure the uniqueness of the POD. In this way, by accessing the combination of the ClusterIP (cluster IP) of the MGR service and the defined port, you can access the http or tcp service provided by the currently active MGR service;

[0086] S33. The container of the inspection service needs to run in a loop, and check and obtain the status of the MGR in Ceph once every custom cycle.

[0087] In step S3, when an application outside the Kubernetes cluster accesses the MGR service, the type of Service needs to be configured. The implementation method is as follows:

[0088] S31. Modify the definition of dashboard in CustomResourceDefinition of Ceph cluster, add three attributes: accessPort, serviceType and accessIP, where the three attributes are of integer type, string type and string type respectively;

[0089] S32. When defining a Ceph cluster, the user chooses to configure a combination of accessPort, serviceType, and accessIP, such as serviceType is NodePort and accessPort is 30000;

[0090] S33. After the controller captures the modified information of CustomResource through the interface provided by the Kubernetes client, it starts to synchronize the cluster status. The work content is as follows:

[0091] S331. Determine the number of MGR services in the cluster:

[0092] When the number of MGR services specified by the count parameter is 1, the type of service is modified directly in the controller;

[0093] When the number of MGR services specified by the count parameter is 2, modify the definition of the deployment according to the configuration content, generate the serviceType, accessPort and accessIP parameters into the definition of the deployment through environment variables, and update the current deployment;

[0094] S332. After the container of the MGR service is started, the above configuration items are obtained through environment variables, and then the definition of the Service is tried to be obtained through the interface provided by the kubenretes client:

[0095] If the current Service has not been successfully created, a service of the specified type is generated according to the configuration items provided in the environment variables, and related configurations are filled in, such as nodePort, etc.;

[0096] If a service with the same name already exists, it means that the service already exists. Update its content and change the type of the service to the required type.

[0097] S333. Steps S331 and S332 are repeated to ensure that the switching of the MGR service will eventually be synchronized with the definition of the Service.

[0098] The embodiment described above is only one specific implementation of the present invention. Common changes and substitutions made by those skilled in the art within the scope of the technical solution of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for implementing multi-MGR service deployment based on rook, characterized in that: The following steps are involved: The first step is to modify the definition of the API model to allow users to define the number of MGRs; In the second step, the controller maintains the MGR service of the current Ceph cluster according to the number specified by the user; The controller maintains the Ceph cluster MGR service status. The specific process is as follows: 1) The controller monitors the status of the CustomResource attribute of the Ceph cluster through the watch listening method provided by the Kubernetes client, and filters the changes of the CustomResource attribute through the custom predicate function; 2) When the count parameter is monitored to change, first determine the number of MGR services specified by the count parameter: When the number of MGR services specified by the count parameter is 1, there is no need to change the existing logic. Just point the Service defined in Kubernetes to the POD with the label app=rook-ceph-mgr. At the same time, the controller deletes the redundant MGR services and only keeps the PODs of the MGR services with lower ordinal numbers, or deletes the containers used for control in the POD and restarts the POD. When the number of MGR services specified by the count parameter is greater than 1, the controller performs the following processing: S1. Create a deployment definition for a specified number of MGRs and define the deployment names in sequence; S2. Add the id option to the startup command of the MGR container and specify the serial number of the current deployment to associate the MGR service with the deployment. Ensure that the label of the POD defined in the deployment contains this serial number to ensure that all the labels of the POD can be combined to specify a unique POD. S3. Add a new container based on the deployment definition to synchronize the MGR service status in the Ceph cluster, and modify the Service definition as needed; In the third step, each POD running the MGR service periodically accesses the Ceph service to obtain its own status and updates the Service definition as needed.

2. The method for implementing multi-MGR service deployment based on rook according to claim 1 is characterized in that: In the first step, the API model is modified by modifying the definition of CustomResourceDefinition and naming it mgr attribute, and adding count parameter configuration in the CustomResource attribute of the Ceph cluster to specify the number of MGR services of the currently defined Ceph cluster.

3. The method for implementing multi-MGR service deployment based on rook according to claim 1 is characterized in that: In step S3, the implementation method of adding a new container is as follows: S31. First, add permissions to the ServiceAccount specified by the container, so that it can perform Create, Retrieve, Update, Delete, and Patch operations on the Service in the current namespace; S32. The configuration file and keyring file of the Ceph cluster are pre-mounted in the container, or the configuration file is automatically generated by obtaining, so as to call the interface of the Ceph cluster in the container and obtain the status of the Ceph cluster.

4. The method for implementing multi-MGR service deployment based on rook according to claim 3 is characterized in that: In step S32, the configuration file is automatically obtained and generated, and the specific process is as follows: S321. Add the permission for the ServiceAccount user of the newly added container to access the Secret encrypted content and ConfigMap in the same namespace, allowing it to obtain the content; S322. Read the contents of rook-ceph-admin-keyring and rook-ceph-mon-endpoints to obtain the keyring for admin and the access address of ceph mon; S323. Referring to the standard form of the configuration file of the Ceph cluster, assemble the acquired data according to the requirements of the Ceph cluster and write it to the specified location of the container; S324. When calling the ceph command, specify the configuration file to the created configuration file to call the Ceph cluster API in the newly added sidecar container.

5. The method for implementing multi-MGR service deployment based on rook according to claim 1 is characterized in that: In step S3, the MGR service status in the Ceph cluster is synchronized in the container, and the implementation method is as follows: S31. After generating the configuration file, you can call the ceph command in the container to obtain the MGR service status. You can get the name of the currently active MGR service by executing ceph –s –format json-pretty | jq '.mgrmap.activename'. S32. Compare the name of the active MGR service in the container with the sequence number defined in the deployment: If the current serial number is the same as the serial number of the active MGR service, the POD where the current MGR service is located is the POD that the Service needs to point to; If the sequence number of the currently active MGR service is not equal to the sequence number of the current container, the loop is skipped once; If the ordinal number of the currently active MGR service is equal to the ordinal number of the current container, it means that the backend pointed to by the Service needs to be updated, that is, the selector of the Service is defined as all labels of the current POD. By accessing the combination of the ClusterIP of the MGR service and the defined port, the http or tcp service provided by the currently active MGR service can be accessed; S33. The container of the inspection service needs to run in a loop, and check and obtain the status of the MGR in Ceph once every custom cycle.

6. The method for implementing multi-MGR service deployment based on rook according to claim 1 is characterized in that: In step S3, when an application outside the Kubernetes cluster accesses the MGR service, the type of Service needs to be configured. The implementation method is as follows: S31. Modify the definition of dashboard in CustomResourceDefinition of Ceph cluster and add three attributes: accessPort, serviceType and accessIP. The three attributes are of integer type, string type and string type respectively. S32. When defining a Ceph cluster, the user chooses to configure a combination of accessPort, serviceType, and accessIP; S33. After the controller captures the modified information of CustomResource through the interface provided by the Kubernetes client, it starts to synchronize the cluster status.

7. The method for implementing multi-MGR service deployment based on rook according to claim 6 is characterized in that: In step S33, the synchronization of cluster status is performed as follows: S331. Determine the number of MGR services in the cluster: When the number of MGR services specified by the count parameter is 1, the type of service is modified directly in the controller; When the number of MGR services specified by the count parameter is 2, modify the definition of the deployment according to the configuration content, generate the serviceType, accessPort and accessIP parameters into the definition of the deployment through environment variables, and update the current deployment; S332. After the container of the MGR service is started, the above configuration items are obtained through environment variables, and then the definition of the Service is tried to be obtained through the interface provided by the kubenretes client: If the current Service has not been successfully created, generate an MGR service of the specified type according to the configuration items provided in the environment variables and fill in the relevant configuration; If a service with the same name already exists, it means that the service already exists. Update its content and change the type of the service to the required type. S333. Steps S331 and S332 are repeated to ensure that the switching of the MGR service will eventually be synchronized with the definition of the Service.

Images