Object storage gateway multi-active deployment method, electronic device and storage medium

Abstract

The application relates to a method for multi-active deployment of an object storage gateway, an electronic device and a storage medium. The method comprises the following steps: adopting a networking mode of Vxlan over IPSec to realize cross-machine-room networking; deploying a distributed file system module, and configuring Ceph cluster nodes; installing MON units and MGR units in the Ceph cluster through binary installation packages; deploying OSD units in units of single server hard disks; configuring at least three copies for a Ceph cluster copy strategy; installing a lightweight container orchestration module K3S cluster on servers where regional center Ceph nodes are located, and deploying at least three ServerNode nodes; selecting two regional centers to build a high-availability database unit connected with K3SServerNode; mirroring an official RGW unit image in the K3S cluster; and binding a vip address on a load balancing module in each regional center for accessing multiple RGW unit services deployed in the regional center. The method for multi-active deployment of the object storage gateway can realize multi-active functions of a multi-regional center architecture.

Description

Technical Field

[0001] This invention relates to the field of IT and software development, and more specifically, to a method, electronic device, and storage medium for multi-active deployment of an object storage gateway based on K3S. Background Technology

[0002] Ceph is a distributed file system that supports three storage interfaces: object storage, block devices, and file systems. Among Ceph distributed file systems, object storage is the most mature and widely used.

[0003] Ceph Object Gateway is an interface built on top of Ceph object storage, providing RESTful gateway services for applications. Object storage supports two API interfaces: OpenStack's Swift and AWS's S3. Object storage uses the Ceph Object Gateway daemon, radosgw, to provide HTTP services for interacting with the Ceph storage cluster.

[0004] K3S is a lightweight, highly available, CNCF-certified Kubernetes distribution. Because it requires minimal resources to run, it is well-suited for resource-constrained or small-scale containerized orchestration scenarios.

[0005] The existing CephRGW gateway uses a primary / backup multi-regional cluster architecture, with the primary regional cluster acting as a read-write cluster and the backup regional cluster as a read-only cluster, and data is synchronized in real time across the multiple regional clusters. This architecture is not a true multi-active architecture, cannot achieve multi-regional cluster active-active functionality, and requires deploying a separate Ceph cluster in each regional cluster, resulting in redundant cluster construction, low resource utilization, and high operational complexity. Furthermore, in the event of a failure in the primary regional cluster, the auxiliary region configuration of the backup regional cluster needs to be manually changed back to the primary region to continue providing object storage gateway services. This leads to long recovery times, low efficiency, and may also result in the loss of small amounts of data during synchronization.

[0006] Therefore, there is an urgent need to develop a multi-active architecture deployment method for object storage gateways that can solve the above problems. Summary of the Invention

[0007] The technical problem this invention aims to solve is that existing multi-regional center architectures cannot achieve multi-active functionality across multiple regional centers; and after a failure of the primary regional center, it is necessary to manually change the auxiliary region configuration of the backup regional center to the primary region to continue providing object storage gateway services, which results in long recovery times and low efficiency.

[0008] To address the aforementioned technical problems, according to one aspect of the present invention, a method for multi-active deployment of an object storage gateway is provided. This method is applied to cross-data center and cross-regional center scenarios, and includes the following steps: S1. Cross-data center networking is performed using Vxlanover IPSec. In cross-regional center scenarios, dual-line access to the VXLAN network is achieved via CE. S2. Multiple servers are prepared in each of the multiple regional centers, and a distributed file system module is deployed. Ceph cluster nodes are configured, with the Ceph cluster having MON units, MGR units, and OSD units. S3. The MON and MGR units in the Ceph cluster are installed, configured, and started using binary installation packages. S4. The OSD units in the Ceph cluster are deployed on a per-server hard drive basis. S5. The Ceph cluster replication policy is configured to have at least three replicas. S6. A lightweight container orchestration module K3S cluster is deployed on the servers where the Ceph nodes are located in each regional center, with at least three ServerNode nodes deployed. These three ServerNode nodes are distributed across the regional centers. S7. Two regional centers are selected to build the K3S... The ServerNode connects to a highly available database unit. The high-availability database node is deployed using MySQL + Keepalived, and a VIP is configured on the node via Keepalived to provide access to the K3SServerNode node; S8. Ceph configuration files and library files are stored in a unified path on all Ceph nodes. In the K3S cluster, the official RGW unit image is used, and the configuration file path and library file path mapping are set in the deployment YAML file; S9. VIP addresses are bound to the load balancing module of each regional center for access to multiple RGW unit services deployed in the regional center.

[0009] According to an embodiment of the present invention, in step S3, each Ceph cluster unit may deploy at least 2 MON units and at least 2 MGR units, and the MON units and MGR units are distributed in the center of each region.

[0010] According to an embodiment of the present invention, in step S5, the Ceph cluster replica policy number can be configured to 3 replicas.

[0011] According to an embodiment of the present invention, in step S5, Ceph cluster replicas can be configured with fault domains at regional centers to distribute the replicas.

[0012] According to an embodiment of the present invention, in step S6, after deploying the ServerNode node, three AgentNode nodes can be deployed. The three AgentNode nodes are distributed in various regional centers and perform cluster registration.

[0013] According to an embodiment of the present invention, in step S7, the database files can be stored in the back-end storage of the node using a shared storage volume provided by a Ceph distributed storage cluster.

[0014] According to an embodiment of the present invention, in step S8, the service can be published in the manner of NodePort, the number of Replicasets is configured to be greater than or equal to 2, and multiple RGW services are deployed by specifying multiple node ports.

[0015] According to an embodiment of the present invention, in step S9, if there are AgentNode nodes that have not been assigned RGW services, they can also be added to the load balancing pool.

[0016] According to a second aspect of the present invention, an electronic device is provided, comprising: a memory, a processor, and an object storage gateway multi-active deployment program stored in the memory and executable on the processor, wherein the object storage gateway multi-active deployment program, when executed by the processor, implements the steps of the object storage gateway multi-active deployment method described above.

[0017] According to a third aspect of the present invention, a computer storage medium is provided, wherein the computer storage medium stores an object storage gateway multi-active deployment program, which, when executed by a processor, implements the steps of the object storage gateway multi-active deployment method described above.

[0018] Compared with the prior art, the technical solution provided by the embodiments of the present invention can achieve at least the following beneficial effects:

[0019] 1) This invention provides a K3S-based method for multi-active deployment of object storage gateways. By connecting multiple regional center networks, a distributed storage cluster is deployed across regional centers. Simultaneously, a lightweight container orchestration cluster spanning multiple data centers is deployed on the distributed storage cluster. The object gateway service is deployed in a containerized manner and provides service capabilities to the outside world. This enables a single cluster to achieve multi-node active-active access across multiple regional centers. In the event of a failure in one regional center, the object storage gateway can be automatically migrated to the remaining normally operating regional center nodes, ensuring continuous service availability.

[0020] 2) The technical solution of this invention implements the multi-active function of CephRGW units, truly achieving multi-point read / write capability. Each module is redundant; within a single regional center, regardless of which module fails, a redundant unit replaces the failed unit to continue providing service. In the event of a severe failure of the entire regional center, the underlying units of the Ceph cluster retain redundant nodes in each regional center, and replicas of stored objects are distributed across regional centers according to the fault domain, ensuring the entire Ceph distributed storage cluster remains available. If the high-availability database connected to the K3S ServerNode is located in this failed data center, the keepalived script determines the database status. If a backup database needs to be activated, it will automatically start backup database nodes in other regions. Since the database files are stored in the shared Cephfs file system, the backup database automatically mounts the corresponding directory, reads the original database configuration and files, and restores database functionality. At this point, the K3S cluster is in normal operation. This invention guarantees a multi-regional-center, multi-active architecture for the cluster, fully leveraging the advantages of Ceph.

[0021] 3) In the technical solution of this invention, if the AgentNode where the corresponding RGW unit pod or service resides is in the relevant faulty data center, the corresponding pod or service will be restarted on the AgentNode of the K3S in normal condition, thus realizing the rapid fault self-healing function of the RGW unit. The RGW unit of this invention is deployed in a containerized mode, and in the event of a fault, it can achieve rapid self-healing.

[0022] 4) When setting up the entire environment, this invention improves the utilization rate of server resources, and the network interconnection also has the advantages of low cost and high security.

[0023] 5) Since data synchronization is performed using the underlying object storage mechanism, this invention leverages Ceph's strong consistency characteristics to ensure data integrity.

[0024] 6) This invention deploys existing Ceph clusters via installation packages, which offers good stability but lacks flexibility in maintenance. While the official method offers containerized deployment, which greatly improves maintenance efficiency, it can lead to unit instability. To address these issues, a layered deployment approach is used. First, the stability of the Ceph cluster is ensured. Key underlying Ceph units are deployed via installation packages, while RGW units are containerized and published via NodePort. This improves the high availability and flexibility of the RGW units, enabling rapid fault self-healing.

[0025] 7) The lightweight container orchestration module of this invention utilizes the functional characteristics of Ceph distributed storage during high-availability deployment, and the RGW unit running on K3S also ensures the high availability of the Ceph object storage gateway. By fully leveraging the advantages of each unit, a synergistic effect greater than the sum of its parts is achieved. Attached Figure Description

[0026] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of the present invention and are not intended to limit the present invention.

[0027] Figure 1 This is a schematic diagram illustrating the method architecture for multi-active deployment of an object storage gateway according to an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the described embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Unless otherwise defined, the technical or scientific terms used herein shall have the ordinary meaning as understood by one of ordinary skill in the art to which this invention pertains. The terms “first,” “second,” and similar terms used in the specification and claims of this patent application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Similarly, the terms “an” or “a” and similar terms do not indicate a limitation of quantity, but rather indicate the presence of at least one.

[0030] Figure 1 This is a schematic diagram illustrating the method architecture for multi-active deployment of an object storage gateway according to an embodiment of the present invention.

[0031] The method of multi-active deployment of object storage gateways is applied in cross-data center networking and cross-regional center scenarios. For example... Figure 1 As shown, the method for multi-active deployment of an object storage gateway includes the following steps:

[0032] S1. Use Vxlan over IPSec networking method for cross-data center networking. In cross-regional center scenarios, access the VXLAN network via CE dual-line connection.

[0033] S2. Prepare multiple servers in multiple regional centers, deploy distributed file system modules, and configure Ceph cluster nodes. The Ceph cluster has MON units, MGR units, and OSD units.

[0034] In S3 and Ceph clusters, MON and MGR units are installed, configured, and started using binary installation packages. Each Ceph cluster must deploy at least two MON units and at least two MGR units, with the MON and MGR units distributed across regional centers.

[0035] In S4 and Ceph clusters, OSD units are deployed on a per-server hard drive basis.

[0036] The S5 and Ceph cluster replica policy is configured to have at least 3 replicas. Ceph cluster replicas are configured with fault domains based on region centers to distribute replicas across the cluster.

[0037] S6. On the servers where the Ceph nodes reside in each regional center, deploy a lightweight container orchestration module K3S cluster with at least 3 ServerNode nodes. These three ServerNode nodes should be distributed across the regional centers. After deploying the ServerNode nodes, deploy three AgentNode nodes, also distributed across the regional centers, and register them with the cluster.

[0038] S7. Select two regional centers to build highly available database units connected to K3S ServerNodes. The high-availability database nodes are deployed using MySQL and Keepalived, with a VIP configured on the nodes via Keepalived to provide access to the K3S ServerNodes. The database files are stored on shared storage volumes provided by a Ceph distributed storage cluster in the backend storage of the nodes.

[0039] S8. Store Ceph configuration files and library files in a unified path across all Ceph nodes. In the K3S cluster, use the official RGW unit image and configure the configuration file and library file path mappings in the deployment YAML file. Publish services using NodePort, configure the number of Replicasets to be greater than or equal to 2, and specify multiple node ports to deploy multiple RGW services.

[0040] The technical solution of this invention implements the multi-active function of CephRGW units, truly achieving multi-point read / write capability. Each module is redundant; within a single regional center, regardless of which module fails, a redundant unit replaces the failed unit to continue providing service. In the event of a severe failure of the entire regional center, the underlying units of the Ceph cluster retain redundant nodes in each regional center, and replicas of stored objects are distributed across regional centers according to the fault domain, ensuring the entire Ceph distributed storage cluster remains available. If the high-availability database connected to the K3S ServerNode is located in this failed data center, the keepalived script determines the database status. If a backup database needs to be activated, backup nodes in other regions will be automatically started. Since the database files are stored in the shared Cephfs file system, the backup database automatically mounts the corresponding directory, reads the original database configuration and files, and restores database functionality. At this point, the K3S cluster is in a normal state.

[0041] The lightweight container orchestration module of this invention utilizes the functional characteristics of Ceph distributed storage during high-availability deployment, and the RGW unit running on K3S also ensures the high availability of the Ceph object storage gateway. By fully leveraging the advantages of each unit, a synergistic effect greater than the sum of its parts is achieved.

[0042] S9. Bind VIP addresses to the load balancing modules in each regional center for accessing multiple RGW unit services deployed in the regional center. Add any AgentNode nodes without assigned RGW services to the load balancing pool.

[0043] This invention provides a K3S-based method for multi-active deployment of object storage gateways. By connecting multiple regional center networks, a distributed storage cluster is deployed across regional centers. Simultaneously, a lightweight container orchestration cluster spanning multiple data centers is deployed on the distributed storage cluster. The object gateway service is deployed in a containerized manner and provides service capabilities externally. This enables a single cluster to achieve multi-node, multi-active access across multiple regional centers. In the event of a failure in one regional center, the object storage gateway can be automatically migrated to the remaining normally functioning regional center nodes, ensuring continuous service availability. This invention guarantees a multi-regional center, multi-active architecture for the cluster and fully leverages the advantages of Ceph.

[0044] In the technical solution of this invention, if the AgentNode containing the corresponding RGW unit pod or service is located in the relevant faulty data center, the corresponding pod or service will be restarted on the AgentNode of the K3S in normal condition, thus enabling the rapid fault self-healing function of the RGW unit. The RGW unit of this invention is deployed in a containerized mode, enabling rapid self-healing in the event of a fault.

[0045] This invention improves server resource utilization and offers advantages such as low cost and high security for network interconnection when setting up the entire environment. Because data synchronization is handled by the underlying object storage mechanism, this invention leverages Ceph's strong consistency characteristics to ensure data integrity.

[0046] This invention deploys existing Ceph clusters via installation packages, offering good stability but lacking flexibility in maintenance. While the official method offers containerized deployment, which greatly improves maintenance efficiency, it can lead to unit instability. To address these issues, a layered deployment approach is used. First, the stability of the Ceph cluster is ensured by deploying the underlying critical units via installation packages, while the RGW units are containerized and published via NodePort. This improves the high availability and flexibility of the RGW units, enabling rapid fault self-healing.

[0047] In view of the above solutions, the embodiments specifically describe one of the technical methods for implementing the module, and other technical personnel can choose other technical solutions to replace them.

[0048] Firstly, by establishing interconnected underlying networks in the regional centers, a VxLAN over IPSec networking approach is adopted for cross-datacenter networking. In cross-regional center scenarios, the CE (Customer Equipment) uses dual-line access to the VxLAN network, enhancing VxLAN access reliability. This allows for faster convergence in case of faults. IPSec encapsulation enables encrypted transmission, ensuring transmission security and ensuring that servers requiring the deployment of distributed file system module clusters are on the same network segment.

[0049] This embodiment preferably uses a three-regional-center deployment, with multiple servers deployed in each regional center. The following configurations are performed on all servers: Install a Linux operating system, upgrading the kernel to version 4.19 or higher. Name all servers and write the names to the hosts file. Install and configure a time service for time synchronization. Create a new deployment account and set up trusted communication between all servers. Create and configure the relevant Ceph source and install the ceph-deploy deployment module. Select one server for Ceph deployment. Generate a ceph.conf configuration file in the / etc / ceph / directory. Select two servers in each regional center for MON unit deployment. Create OSD units on each server, using the hard drives to be used as Ceph data storage as individual blocks. Select one server in each regional center for MGR unit deployment, with one node as active and the other two as standby.

[0050] Configure the Ceph cluster accordingly. It is recommended to have at least 3 replicas. Divide the cluster into fault domains, using the region center as the unit. This will prevent replicas of stored objects from being allocated in the same region center, thus ensuring replica redundancy.

[0051] Configure the K3S high-availability cluster ServerNode to connect to an external high-availability database server. Prepare two servers in two data centers to install the MySQL service, one as the primary node and the other as the standby node. Maintain consistency in version number, MySQL user UID, installation directory, and data file directory. Allocate an RBD data disk for the high-availability database cluster on the Ceph cluster and mount it to the primary database node. Install and configure the database on the primary node. Install and configure keepalived on both the primary and standby nodes, configuring status detection, MySQL status change scripts, and notification scripts in the relevant configuration files. Keepalived defaults to active and standby states. The conditions for determining whether the database state has changed are: the database service is running, the VIP is alive, and the data disk is mounted. When the MySQL service stops on the primary node, the standby node switches states, and the notification script executes the task in the active state to start mounting the RBD data disk and starting the MySQL service, thus providing service on the standby node. The K3S ServerNode connects to the high-availability database VIP to access the database.

[0052] Deploy a lightweight K3S container orchestration cluster on a Ceph distributed storage cluster server, which requires the deployment of ServerNode nodes and AgentNode nodes. At least one ServerNode node and one AgentNode node must exist in each regional center.

[0053] First, deploy ServerNode nodes using the official script. After deployment, obtain the URL and token for cluster access. Deploy ServerNode nodes in the other data centers using the same method. Configure the VIP address and port in the load balancer to distribute load to the deployed ServerNode nodes. All API operations will be accessed through the load balancer's address. Finally, deploy AgentNode nodes in each data center using the official deployment script, URL, and token to form a highly available K3S cluster.

[0054] Write the YAML file to deploy the RGW unit, containerize the RGW unit, and map the configuration file path and library path of the Ceph cluster in the deployment file. Deploy at least two pods and services in each regional center, publish the services using Nodeport, and configure the load balancing module to distribute object storage traffic to the corresponding RGW unit to build high availability.

[0055] Because cross-datacenter traffic transmission is inherently less efficient and stable than local area network transmission, Ceph, to maintain strong data consistency, waits until all replicas have been written before completing the I / O. To avoid read / write performance instability caused by network latency, this example configures a cache pool for Ceph's object storage pool. The cache pool uses high-read / write performance devices such as SSDs or NVMe SSDs. The cache pool is configured to operate in WRITEBACK mode to ensure the overall read / write performance of the cluster.

[0056] External client access to the regional center can utilize intelligent domain name resolution to load balancing external object storage access traffic, achieving a multi-active, high-availability strategy.

[0057] According to another aspect of the present invention, an apparatus for multi-active deployment of an object storage gateway is provided, comprising: a memory, a processor, and an object storage gateway multi-active deployment program stored in the memory and executable on the processor, wherein the object storage gateway multi-active deployment program, when executed by the processor, implements the steps of the above-described object storage gateway multi-active deployment method.

[0058] The present invention also provides a computer storage medium.

[0059] The computer storage medium stores an object storage gateway multi-active deployment program, which, when executed by the processor, implements the steps of the object storage gateway multi-active deployment method described above.

[0060] The method implemented when the object storage gateway multi-active deployment program running on the processor is executed can be referred to in various embodiments of the object storage gateway multi-active deployment method of the present invention, and will not be repeated here.

[0061] The present invention also provides a computer program product.

[0062] The computer program product of the present invention includes an object storage gateway multi-active deployment program, which, when executed by a processor, implements the steps of the object storage gateway multi-active deployment method as described above.

[0063] The method implemented when the object storage gateway multi-active deployment program running on the processor is executed can be referred to in various embodiments of the object storage gateway multi-active deployment method of the present invention, and will not be repeated here.

[0064] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0065] The above description is merely an exemplary embodiment of the present invention and is not intended to limit the scope of protection of the present invention, which is determined by the appended claims.

Claims

1. A method for multi-active deployment of an object storage gateway, the method being applied in a cross-machine room networking, cross-region center scenario, the method comprising the following steps: S1. Cross-machine room networking is performed by using a Vxlan over IPSec networking mode, in the cross-region center scenario, a VXLAN network is accessed by a CE double-line; S2. A plurality of servers are prepared in each of a plurality of regional centers, a distributed file system module is deployed, and a Ceph cluster node is configured, the Ceph cluster having a MON unit, a MGR unit and an OSD unit; S3. The MON unit and the MGR unit in the Ceph cluster are installed, configured and started by a binary installation package deployment mode; S4. The OSD unit in the Ceph cluster is deployed in units of a single server hard disk; S5. The number of Ceph cluster replication strategies is configured to be at least three replicas; S6. A lightweight container orchestration module K3S cluster is deployed on a server where a Ceph node in each regional center is located, and at least three ServerNode nodes are deployed, the three ServerNode nodes being dispersed in each regional center; S7. Two regional centers are selected to build a high-availability database unit connected to a K3S ServerNode, the high-availability database node being deployed by using mysql+keepalived, and a vip is configured on the node by using keepalived and provided for access of the K3S ServerNode; S8. Ceph configuration files and library files are stored in a unified path on all Ceph nodes, and an official RGW unit image is used in the K3S cluster, and a configuration file path and a library file path mapping are set in a deployment yaml file; S9. A vip address is bound to a load balancing module in each regional center for accessing a plurality of RGW unit services deployed in the regional center.

2. The method of claim 1, in step 3, at least two MON units and at least two MGR units are deployed in each Ceph cluster, and the MON units and the MGR units are dispersed in each regional center.

3. The method of claim 1, in step S5, the number of Ceph cluster replication strategies is configured to be three replicas.

4. The method of claim 3, in step S5, the Ceph cluster replicas are configured with a fault domain in units of a regional center to disperse the distribution of the replicas.

5. The method of claim 1, in step S6, three AgentNode nodes are deployed after the ServerNode nodes are deployed, the three AgentNode nodes being dispersed in each regional center and registered in a cluster.

6. The method of claim 1, in step S7, a shared storage volume provided by a Ceph distributed storage cluster is used to store database files in a node backend storage.

7. The method of claim 1, in step S8, services are published in a NodePort mode, the number of Replicaset is configured to be greater than or equal to 2, and a plurality of nodes are specified to deploy a plurality of RGW services.

8. The method of claim 1, in step S9, if there is an AgentNode node without assigned RGW service, the AgentNode node also joins the load balancing pool.

9. An electronic device comprising: A memory, a processor, and an object storage gateway multi-live deployment program stored on the memory and executable on the processor, the object storage gateway multi-live deployment program, when executed by the processor, implements the steps of the object storage gateway multi-live deployment method of any one of claims 1 to 7.

10. A computer storage medium, wherein, The computer storage medium has stored thereon an object storage gateway multi-live deployment program, the object storage gateway multi-live deployment program, when executed by a processor, implements the steps of the object storage gateway multi-live deployment method of any one of claims 1 to 7.

Images