Collaborative backup and cross-cluster quick recovery method based on container cloud platform underlying component

Abstract

The invention belongs to the technical field of cloud computing and virtualized containers, and relates to a collaborative backup and cross-cluster quick recovery method based on a container cloud platform underlying component. Comprising the steps that in the production cluster timing backup stage, Ceph Storage is used as a default storage class of a container cluster, platform cluster resources, configuration resources and storage volume information are backed up in a timing mode through Velero, and backup resource list data are exported to high-availability storage in Minio; and in a cross-cluster recovery stateful service stage, uploading backup data in a new cluster Mini, recovering a Ceph CSI plug-in through Velero, recovering cluster configuration, and recovering stateful services and corresponding storage volumes in a container cluster. According to the method, the stateful service application can be quickly recovered in a cross-cluster manner through the backup resource list data when the container cloud platform has a disaster, and the real-time performance of online services is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of cloud computing and virtualized containers, and specifically is a method for collaborative backup and cross-cluster rapid recovery of underlying components based on a container cloud platform. Background Art

[0002] In the era of rapid development of the industrial Internet, various industrial enterprises have increasing requirements for data reliability and platform disaster recovery of container cloud platforms. With the widespread application of microservice technology under container platforms, the number of project containers deployed on container cloud platforms is increasing. Especially in some industrial fields, there are more and more types of stateful service containers, and the business needs to run 24 hours a day. Therefore, the data storage reliability, platform disaster recovery, and business real-time performance of the platform are crucial. Summary of the invention

[0003] The technical method of the present invention is to provide a method for collaborative backup and rapid cross-cluster recovery based on the underlying components of a container cloud platform. In the Openshift container cluster, Velero, Ceph Storage, Minio and other components are used to collaboratively back up the Openshift container cluster resources and the stateful service application resource list. When a cluster disaster occurs and is unrecoverable, the Openshift container cluster resources and stateful service applications are quickly restored by restoring the resource list, which provides strong protection for online business and has theoretical and practical significance for cluster disaster recovery.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0005] Based on the collaborative backup and cross-cluster rapid recovery method of the underlying components of the container cloud platform, this method uses CephStorage, Ceph CSI, Velero, and Minio components to coordinate and regularly back up the container cloud resource list on the production cluster side, and restore it on the pre-production cluster side to achieve cross-cluster recovery of stateful services and ensure the real-time performance of online services; the steps are as follows:

[0006] Production cluster scheduled backup stage: formulate scheduled backup tasks, regularly back up production container cluster resources and persistent storage volumes, and save the resource list data of the backup data to high-availability storage;

[0007] Cross-cluster stateful service recovery phase: When a disaster occurs in the production cluster and data is lost, the backup resource list data is uploaded from the high-availability storage to the pre-production cluster. The pre-production cluster uses Velero to restore the container cluster configuration resources and restore the stateful services and corresponding storage volumes in the namespace.

[0008] The scheduled backup phase of the production cluster includes the following steps:

[0009] Set up Red Hat Ceph Storage as a distributed object storage to store upper-layer application data;

[0010] Set up the container cluster to access the block storage Ceph Storage through the Ceph CSI plug-in to store cloud data;

[0011] Set the NFS file system as the storage of Minio to store the collected resource list data;

[0012] Set up Velero to collect resource list data from the user side of the production cluster;

[0013] Formulate scheduled backup tasks: including backup time and data to be backed up, and a list of scheduled backup resources;

[0014] Regularly export the backup resource list data in Minio to a high-availability storage device.

[0015] The container cloud cluster accesses the block storage Ceph Storage through the Ceph CSI plug-in, including: setting the plug-in CephCSI access type to RBD, installing the block storage plug-in rbdplugin and the block storage provider rbdplugin-provisioner in the production cluster Openshift, and using Ceph RBD as the cluster default storage class default StorageClass.

[0016] The NFS file system is used as the storage of Minio, which sets the storage type mounted by Minio in the Openshift container cloud production cluster to NFS and exposes the interface to Velero.

[0017] The Velero includes a client and a server. The Velero client is installed on the oc management command line client side, and the Velero server is installed on the Openshift container production cluster side.

[0018] The backup task comprises the following steps:

[0019] Back up Ceph CSI namespace resources Project Resource, role access control resources ClusterRoleBinding, and service account ServiceAccount cluster configuration resources;

[0020] Back up the namespace resources Project Resource of the stateful service, including the storage volume declaration, and add the persistent storage Persistentvolumes parameter of --ordered-resources to the scheduled backup command.

[0021] The cross-cluster stateful service recovery phase includes the following steps:

[0022] Import the restored resource list data into Minio of the pre-production cluster;

[0023] Use Velero to restore the basic configuration resources of the OpenShift production cluster container;

[0024] Use Velero to recover stateful service applications and persistent storage declarations for Persistentvolumes.

[0025] Before importing the recovered resource list data into Minio of the pre-production cluster: set the version number, DNS domain name system, Harbor image repository, Ceph CSI plug-in, Velero component, and Minio component of the pre-production cluster to be consistent with the production cluster settings where the disaster occurred.

[0026] The use of Velero to restore the basic configuration resources of the production Openshift cluster container includes restoring the Harbor image repository Registry configuration, Ceph CSI block storage RBD, ClusterRoleBinding role access control, service account ServiceAccount, and storage class StorageClass Openshift container cluster resources.

[0027] This method has the following advantages:

[0028] 1. First of all, stateful services do not need to migrate data during the backup and recovery process. They only need to back up the resource list, which reduces the time spent on transmitting a large amount of data and ensures the timeliness of business recovery.

[0029] 2. By backing up the basic configuration of the cluster, including the service account ServiceAccount, the role access control resource ClusterRolebinding, and the storage class StoargeClass, the plug-in in the container Openshift cluster can be quickly restored to ensure the consistency of the cluster and avoid the differences caused by cross-cluster recovery.

[0030] 3. The present invention improves the disaster recovery capability and data security of the cluster platform through the collaborative work of tools such as Ceph Storage, Minio, and Velero. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 Software architecture diagram for cross-cluster backup and recovery

[0032] Figure 2 Flowchart for cross-cluster backup and recovery DETAILED DESCRIPTION

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0034] The data that can be processed or backed up by the present invention mainly involves the container cluster resource list, while the data of the actual stateful service application involves the industrial manufacturing industry, including production execution data, plan management data, equipment management data, master data management data, etc.

[0035] A method for coordinated backup and cross-cluster rapid recovery based on the underlying components of a container cloud platform. The method is to coordinate and regularly back up container cloud resources through components such as Ceph Storage, Ceph CSI, Velero, and Minio to achieve cross-cluster recovery of stateful services, including the following steps:

[0036] Step 1: Cluster scheduled backup phase

[0037] Develop a scheduled backup plan to regularly back up production OpenShift cluster resources or persistent storage volumes, and save the resource list file of the backup data to high-availability storage;

[0038] Step 2: Recover stateful services across clusters

[0039] When a disaster occurs in the cluster, upload the backup resource list data to the new cluster, and Velero restores resources such as OpenShift CSI and storage volumes, and restores stateful services in the Project.

[0040] The cluster scheduled backup phase includes the following steps:

[0041] Step 1-1: Red Hat Ceph Storage as distributed object storage;

[0042] Step 1-2: Openshift cluster connects to Ceph Storage through Ceph CSI;

[0043] Step 1-3: NFS as Minio storage;

[0044] Step 1-3: Use Minio as Velero storage;

[0045] Step 1-4: Velero includes client and server;

[0046] Step 1-5: Set up backup tasks and backup resource lists regularly;

[0047] Step 1-6: Regularly export the backup resource list data in Minio to a high-availability storage device.

[0048] The Openshift cluster is connected to Ceph Storage through Ceph CSI. First, the type of Ceph CSI access is RBD, rbdplugin and rbdplugin-provisioner are installed in Openshift, and secondly, Ceph rdb is used as the default StorageClass.

[0049] In the NFS as Minio storage, the storage type mounted by Minio in the Openshift cluster is NFS, including the two directories / storage and / config, and it is necessary to expose ports 9001 and 9000 and service resources named minio-console and minio.

[0050] The Velero includes a client and a server. The Velero client is installed in the oc command line client, and the Velero server is installed in the Openshift cluster.

[0051] The backup task includes the following steps:

[0052] Step 1-5-1: Back up Ceph CSI Project Resource, ClusterRoleBinding, ServiceAccount and other resources;

[0053] Step 1-5-2: Back up the Project Resource of the stateful service, including the storage volume, and add the Persistentvolumes parameter of --ordered-resources to the scheduled backup command.

[0054] The command is as follows: velero backup create backup-test--ordered-resources'persistentvolumes=xxx'--include-namespaces=test--schedule="0 0 1**".

[0055] The cross-cluster stateful service recovery phase includes the following steps:

[0056] Step 2-1: Import the restored resource list data into Minio of the new cluster;

[0057] Step 2-2: Use Velero to restore OpenShift cluster resources;

[0058] Step 2-3: Use Velero to restore stateful service applications and Persistent volumes.

[0059] In the backup data that needs to be restored imported into Minio of the new cluster, the version number, DNS service, Harbor service, Ceph CSI plug-in, Velero component, Minio component of the new OpenShift cluster is consistent with the cluster where the disaster occurred.

[0060] The use of Velero to restore Openshift cluster resources includes restoring Harbor Registry configuration, Ceph CSI's rbd, ClusterRoleBinding, ServiceAccount, StorageClass and other Openshift cluster resources.

[0061] The method of the present invention has been verified in practice and can solve the problem of rapid cross-cluster recovery of stateful service businesses when a disaster occurs on a container cloud platform.

[0062] Example 1

[0063] In the experimental process of the present invention, the underlying components are coordinated and quickly restored across clusters in two Openshift clusters. First, the two clusters use the same common components, including DNS domain name system, Ceph distributed storage, file system service, Harbor image repository, load balancing service, etc.

[0064] The container environment is as follows:

[0065] Openshift production cluster: Openshift 4.9.36, three master nodes, three worker nodes. Ceph CSI 3.4.0 and Velero 1.8.1 installed. All nodes are configured with 8C32G.

[0066] Openshift pre-production cluster: Openshift 4.9.36 version, three master nodes, three worker nodes. Ceph CSI 3.4.0 version and Velero 1.8.1 version are installed. All nodes are configured with 8C32G.

[0067] Red Hat Ceph Storage cluster: Red Hat ceph version 14.2.11-95.el8cp, four nodes, all nodes are configured with 32C256G.

[0068] DNS service: two nodes in active / standby mode.

[0069] Harbor service: Harbor version 2.0.3, one node, configured as 2c8g.

[0070] File system service NFS: Inspur high-availability storage device.

[0071] First, configure Ceph plug-in and NFS plug-in in the production cluster to complete the configuration of Ceph RBD and NFS. Use Ceph RBD as the default storage class StorageClass of the Openshift container cluster, create a namespace Project for stateful services, create application Deployment, PVC, Route and Service that expose stateful services in the namespace Project, and save data in the application to Ceph block storage or file system type PV. Use the Velero client to create a scheduled task with the following command create backup. Simulate a disaster and use the console Web-console in Minio to export the backup resource data list to a high-availability storage device.

[0072] Import the backup resource list file into Minio of the pre-production cluster. First, use the velero command to restore cluster resources such as Ceph CSI, NFS CSI, storage class, role access control ClusterRoleBingding, service account ServiceAccount from the backup resource file. Then restore the namespace Project of the stateful service, including the resource list of Deployment, PVC, Route, Service, and persistent storage volume declaration Persistentvolumes, and restore the stateful service. At the same time, the business is also restored.

[0073] The present invention proposes a method for coordinated scheduled backup of underlying components and cross-cluster rapid recovery when a cluster disaster occurs and cannot be recovered. The present invention, on the one hand, realizes scheduled backup of cluster resource lists and stateful service resource lists; on the other hand, in the process of realizing cross-cluster rapid recovery, there is no need to migrate storage data, only the cluster and stateful service resource lists need to be restored, thus realizing low-latency and efficient recovery.

[0074] The above-mentioned implementation examples are typical examples of the examples disclosed in this disclosure. All equivalent changes and modifications made by equivalent replacement or according to the ideas and schemes of this example are within the scope of protection required by this patent.

Claims

1. A method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform, characterized in that: The Ceph Storage, Ceph CSI, Velero, and Minio components are used to coordinate and regularly back up the container cloud resource list on the production cluster side, and restore it on the pre-production cluster side, so as to achieve cross-cluster recovery of stateful services and ensure the real-time performance of online services. The steps include: Production cluster scheduled backup stage: formulate scheduled backup tasks, regularly back up production container cluster resources and persistent storage volumes, and save the resource list data of the backup data to high-availability storage; Cross-cluster stateful service recovery phase: When a disaster occurs in the production cluster and data is lost, the backup resource list data is uploaded from the high-availability storage to the pre-production cluster. The pre-production cluster uses Velero to restore the container cluster configuration resources and restore the stateful services and corresponding storage volumes in the namespace.

2. According to claim 1, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: The scheduled backup phase of the production cluster includes the following steps: Set up Red Hat Ceph Storage as a distributed object storage to store upper-layer application data; Set up the container cluster to access the block storage Ceph Storage through the Ceph CSI plug-in to store cloud data; Set the NFS file system as the storage of Minio to store the collected resource list data; Set up Velero to collect resource list data from the user side of the production cluster; Formulate scheduled backup tasks: including backup time and data to be backed up, and a list of scheduled backup resources; Regularly export the backup resource list data in Minio to a high-availability storage device.

3. According to claim 2, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: The container cloud cluster accesses the block storage Ceph Storage through the Ceph CSI plug-in, including: setting the plug-in Ceph CSI access type to RBD, installing the block storage plug-in rbdplugin and the block storage provider rbdplugin-provisioner in the production cluster Openshift, and using Ceph RBD as the cluster default storage class defaultStorageClass.

4. According to claim 2, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: The NFS file system is used as the storage of Minio, which sets the storage type mounted by Minio in the Openshift container cloud production cluster to NFS and exposes the interface to Velero.

5. According to claim 2, the method for collaborative backup and cross-cluster rapid recovery of underlying components based on the container cloud platform is characterized in that: The Velero includes a client and a server. The Velero client is installed on the oc management command line client side, and the Velero server is installed on the Openshift container production cluster side.

6. According to claim 2, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: The backup task comprises the following steps: Back up Ceph CSI namespace resources Project Resource, role access control resources ClusterRoleBinding, and service account ServiceAccount cluster configuration resources; Back up the namespace resources Project Resource of the stateful service, including the storage volume declaration, and add the persistent storage Persistentvolumes parameter of --ordered-resources to the scheduled backup command.

7. According to claim 1, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: The cross-cluster stateful service recovery phase includes the following steps: Import the restored resource list data into Minio of the pre-production cluster; Use Velero to restore the basic configuration resources of the OpenShift production cluster container; Use Velero to recover stateful service applications and persistent storage declarations for Persistentvolumes.

8. According to claim 7, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: Before importing the recovered resource list data into Minio of the pre-production cluster: set the version number, DNS domain name system, Harbor image repository, Ceph CSI plug-in, Velero component, and Minio component of the pre-production cluster to be consistent with the production cluster settings where the disaster occurred.

9. According to claim 7, the method for collaborative backup and cross-cluster rapid recovery of underlying components of a container cloud platform is characterized in that: The use of Velero to restore the basic configuration resources of the production Openshift cluster container includes restoring the Harbor image repository Registry configuration, Ceph CSI block storage RBD, ClusterRoleBinding role access control, service account ServiceAccount, and storage class StorageClass Openshift container cluster resources.

Images