A backup and recovery method, apparatus, device, and computer storage medium.

By generating backup containers in the data center to monitor and back up the data cluster, the problem of etcd cluster backup affecting business operations is solved, enabling efficient backup and recovery of data clusters in multiple scenarios and reducing operation and maintenance costs.

CN116795588BActive Publication Date: 2026-11-17CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
CN202211114032.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-09-14
Publication Date
2026-11-17
Estimated Expiration
2042-09-14

AI Technical Summary

Technical Problem

In existing technologies, the backup process of etcd clusters can affect cluster services, and the backup process is complex, inefficient, and costly to maintain. In particular, it cannot effectively guarantee data integrity and efficient recovery in many scenarios.

Method used

By generating backup containers in the data center to monitor the data cluster, generating backup data, and restoring business data in the data cluster when business data is lost, a combination of connection containers, policy containers, and synchronization containers is used to determine the backup strategy based on changes in status data, achieving incremental or full backup and reducing the difficulty of operation and maintenance.

Benefits of technology

Without affecting the stability of the data cluster, it enables efficient backup and recovery of data from multiple clusters in various scenarios, improving the efficiency of business data backup and reducing the difficulty of operation and maintenance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116795588B_ABST
    Figure CN116795588B_ABST
Patent Text Reader

Abstract

This application discloses a backup and recovery method, apparatus, device, and computer storage medium. The method includes: monitoring a data center through a backup module; generating a set of backup containers when data cluster registration occurs in the data center; backing up business data in the data cluster using the backup containers to obtain backup data; wherein the backup containers are associated with the data cluster; and restoring the business data in the data cluster based on the backup data in the event of business data loss. In this way, the backup module can achieve backup and recovery of multiple cluster data in different scenarios without affecting the stability of the data cluster, improving the efficiency of business data backup and reducing the difficulty of operation and maintenance.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of information technology application technology, and in particular to a backup and recovery method, apparatus, device, and computer storage medium. Background Technology

[0002] Etcd is a highly consistent distributed key-value store cluster that provides a reliable way to store data that needs to be accessed by a distributed system or cluster of machines. The Etcd data cluster itself provides backup methods, and can adaptively migrate applications to other nodes when application or network anomalies occur. This is thanks to the message queues, listening, and distributed leader election features provided by the etcd cluster itself.

[0003] In related technologies, etcd clusters provide backup methods that involve reading and writing large amounts of data. This can cause the backup process to affect cluster services, and the backup process is complex, inefficient, and costly to maintain. Summary of the Invention

[0004] This application proposes a backup and recovery method, apparatus, device, and computer storage medium, which can realize the function of backup and recovery of multiple cluster data in different scenarios, improve the efficiency of business data backup, and reduce the difficulty of operation and maintenance.

[0005] To achieve the above objectives, the technical solution of this application is implemented as follows:

[0006] In a first aspect, embodiments of this application provide a backup and recovery method, the method comprising:

[0007] Monitor the data center using the backup module;

[0008] In the event of a data cluster registration in the data center, a set of backup containers is generated;

[0009] The backup container is used to back up the business data in the data cluster to obtain backup data; wherein, the backup container is associated with the data cluster.

[0010] In the event of loss of the business data, the business data in the data cluster is restored based on the backup data.

[0011] In some embodiments, the backup container includes a connection container, a policy container, and a synchronization container;

[0012] In some embodiments, the step of backing up the business data in the data cluster through the backup container to obtain backup data includes:

[0013] The connection container sends the business data in the data cluster to the synchronization container;

[0014] The backup policy corresponding to the business data is sent to the synchronization container through the policy container;

[0015] The synchronization container performs backup processing on the business data according to the business data and the backup strategy to obtain the backup data.

[0016] In some embodiments, the backup module includes a monitoring module, a policy module, and a synchronization module, and the method further includes:

[0017] The monitoring module monitors the status data of several data clusters in the data center.

[0018] When the status data changes, the corresponding backup strategy for the strategy module is determined;

[0019] The synchronization module performs backup processing according to the backup strategy and stores the backup data in the target file.

[0020] In some embodiments, the status data includes period data and index data, and determining the backup policy corresponding to the policy module when the status data changes includes:

[0021] When the time limit data in the status data increases, the backup time of the incremental data of the data cluster is determined by the strategy module.

[0022] If the time limit data remains unchanged in the status data, but the index data changes, the backup time of the incremental data of the data cluster is determined by the strategy module.

[0023] In some embodiments, determining the backup time of incremental data for the data cluster through the policy module includes:

[0024] The first specification threshold is determined based on the maximum memory and the maximum value of the central processing unit in the storage type;

[0025] The second specification threshold is determined based on the current memory and current CPU values;

[0026] The backup time corresponding to the backup strategy is determined based on the first specification threshold and the second specification threshold.

[0027] In some embodiments, the method further includes:

[0028] If the scenario corresponding to the data cluster is a dedicated cluster, a full backup of the data cluster shall be performed.

[0029] When the scenario corresponding to the data cluster is a managed cluster, incremental backups of the data cluster are performed according to a preset strategy.

[0030] Secondly, embodiments of this application provide a backup and recovery device, characterized in that the backup and recovery device includes a monitoring unit, a generation unit, a backup unit, and a recovery unit, wherein...

[0031] The monitoring unit is configured to monitor the data center through the backup module;

[0032] The generation unit is configured to generate a set of backup containers when a data cluster registration occurs in the data center;

[0033] The backup unit is configured to back up the business data in the data cluster through the backup container to obtain backup data; wherein the backup container is associated with the data cluster.

[0034] The recovery unit is configured to recover the business data in the data cluster based on the backup data in the event of loss of the business data.

[0035] Thirdly, embodiments of this application provide an electronic device, which includes: a memory and a processor; wherein,

[0036] The memory is used to store computer programs that can run on the processor;

[0037] The processor is configured to execute the method as described in the first aspect when running the computer program.

[0038] Fourthly, embodiments of this application provide a computer storage medium storing a backup and recovery program that, when executed by at least one processor, implements the method described in the first aspect.

[0039] This application provides a backup and recovery method, apparatus, device, and computer storage medium. A backup module monitors the data center; in the event of data cluster registration within the data center, a set of backup containers is generated; the backup containers are used to back up business data in the data cluster to obtain backup data; wherein the backup containers are associated with the data cluster; in the event of business data loss, the business data in the data cluster is restored based on the backup data. In this way, the backup module can achieve backup and recovery of multiple cluster data in different scenarios without affecting the stability of the data cluster, improving the efficiency of business data backup and reducing the difficulty of operation and maintenance. Attached Figure Description

[0040] Figure 1A schematic flowchart illustrating a backup and recovery method provided in an embodiment of this application;

[0041] Figure 2 A detailed flowchart illustrating a backup and recovery method provided in this application embodiment;

[0042] Figure 3 A schematic diagram of the system framework for a backup and recovery method provided in this application embodiment;

[0043] Figure 4 This is a schematic diagram of the composition structure of a backup and recovery device provided in an embodiment of this application;

[0044] Figure 5 A schematic diagram of the composition structure of an electronic device provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram of the composition structure of another electronic device provided in an embodiment of this application. Detailed Implementation

[0046] In order to gain a more detailed understanding of the features and technical content of the embodiments of this application, the implementation of the embodiments of this application will be described in detail below with reference to the accompanying drawings. The accompanying drawings are for reference and illustration only and are not intended to limit the embodiments of this application.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0048] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. It is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. It should also be noted that the terms "first, second, third" used in the embodiments of this application are merely for distinguishing similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0049] As we can understand it, blockchain is a distributed, shared ledger and database with characteristics such as decentralization, immutability, full traceability, accountability, collective maintenance, and transparency. These characteristics ensure the "honesty" and "transparency" of the blockchain, laying the foundation for trust. Essentially, blockchain is a decentralized database. As the underlying technology of Bitcoin, it is a chain of data blocks linked together using cryptographic methods. Each data block contains information about a batch of Bitcoin network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. Data in the blockchain includes two types: ledger data and state data. Ledger data contains ordered, immutable records of state transitions, i.e., transaction records and blockchain configuration records. State data, also called world state, maintains the current state of the ledger.

[0050] In related technologies,

[0051] Etcd is a highly consistent distributed key-value store that provides a reliable way to store data that needs to be accessed by a distributed system or machine cluster. In the currently booming cloud-native field, such as the Kubernetes orchestration platform, using it as a storage backend can provide reliable application assurance. When application or network anomalies occur, it can adaptively migrate applications to other nodes. This is all thanks to the message queues, listening, and distributed leader election features provided by the etcd cluster itself, which enable the platform's data to be highly available and stable.

[0052] The high availability of Etcd clusters is achieved by adhering to the Raft protocol within the cluster and employing a leader election mechanism. The leader node synchronously sends requests to other nodes, and then feeds back to clients by tracking heartbeats from these nodes. Only when a majority of nodes are still active can the etcd cluster provide normal service. Many application platforms rely on the distributed nature of etcd to ensure high data availability. Backup methods for upper-layer application data depend on the distributed nature of etcd. In existing technical solutions, platform disaster recovery relies on the distributed storage of data in etcd, combined with other methods to switch services.

[0053] In addition, for enterprises that provide container cluster products, the deployment method of etcd is no longer to pre-plan nodes and deploy in a fixed three-node mode. Instead, it is implemented in the CRD (Custom Resource Definition) mode with the help of management tools of etcd-operator type, that is, using container (pod) instances. For data storage in this scenario, many enterprises usually rely on the high availability guarantee of the cluster itself, that is, allow a small number of pod failures. Once an failure occurs, the usual practice is to restart the pod instance according to the stored data.

[0054] etcd clusters provide built-in backup methods, but a significant drawback is their performance. The backup process can impact cluster operations due to the large volume of data reads and writes. Therefore, current solutions, including disaster recovery, rely on distributed storage backups, triggering backups via scripts or manual methods during periods of relative inactivity in the cluster. Current backup timing for etcd clusters typically involves coarse-grained backups triggered by monitoring alerts or scheduled backups. However, once an alert occurs and operations are already affected, performing a backup at this point means that a complete backup cannot be achieved if the existing network operations cannot be stopped. A complete backup requires pausing operations. In some scenarios, alerts may even indicate anomalies, preventing data backups altogether, or resulting in older, incomplete backups. Therefore, this solution aims to design a fine-grained etcd backup solution to ensure that user-critical services are not lost.

[0055] In related technologies, the backup service provided by etcd performs a full copy process, including all content, which has relatively low performance. This is because an etcd cluster typically contains one leader node and two member nodes. The leader node needs to send data to the member nodes and receive responses before it can provide feedback to the client. When the data volume is large, each backup requires repeating the entire backup process from beginning to end, resulting in significant backup overhead. For example, (1) in the scenario of a public cloud service provider's managed container cluster, the etcd cluster is not exposed to users. Applications are usually deployed using pod instances. In this case, users need to ensure their own business data backup and need to have the basic knowledge of etcd and Kubernetes. Business backup can only be achieved through other specific scripts. The backup process is complex and uncontrollable. In addition, in the scenario of multiple users and multiple etcd clusters, how the operator can ensure that the user's data is not lost is an important issue. (2) In the scenario of public cloud dedicated clusters or private clouds, the etcd cluster is deployed on the user's three replica hosts. The high availability of the cluster ensures the integrity of the data. However, the hosts are usually in the same rack. The entire cluster may become unavailable due to the destruction of the data center, which will affect the business. In order to achieve data integrity, the etcd cluster needs to be deployed on the user's three replica hosts. The high availability of the cluster ensures the integrity of the data. However, the hosts are usually in the same rack. The entire cluster may become unavailable due to the destruction of the data center, which will affect the business. Backup usually requires users to use monitoring and alarm tools to monitor the data. Once an anomaly is detected, the entire cluster data is backed up using scripts or manual methods. With the backup data, the entire cluster's business can be restored through other nodes based on the backup data. The premise is that there is backup data. The integrity of the backup data cannot be guaranteed. With the backup data, the entire business can be restored through other new nodes. The cost is very high and time-consuming. (3) In the government or education cloud industries, for scenarios with high data business stability and security, in order to ensure data performance, it is usually necessary to use high-performance physical machines to deploy the business, thereby achieving highly reliable and high-performance business. It is usually necessary to back up etcd data in real time. However, this real-time capability will affect the performance of the business as the data increases. The flexible strategy of data backup has become one of the problems that need to be solved. How to design a multi-scenario etcd backup and recovery scheme to reduce user self-maintenance is another problem that this patent needs to solve.

[0056] Based on this, this application provides a backup and recovery method. The basic idea of ​​this method is as follows: a backup module monitors the data center; in the event of data cluster registration in the data center, a set of backup containers is generated; the backup containers are used to back up the business data in the data cluster to obtain backup data; wherein the backup containers are associated with the data cluster; in the event of business data loss, the business data in the data cluster is restored based on the backup data. In this way, the backup module can achieve the function of backing up and restoring multiple cluster data in different scenarios without affecting the stability of the data cluster, improving the efficiency of business data backup and reducing the difficulty of operation and maintenance.

[0057] Example 1

[0058] In one embodiment of this application, see [link to embodiment]. Figure 1 This illustrates a flowchart of a backup and recovery method provided in an embodiment of this application. Figure 1 As shown, the method may include:

[0059] S101: Monitors the data center via the backup module.

[0060] It should be noted that, in this embodiment, the backup and recovery method provided can be applied to electronic devices that require backup and recovery of distributed key-value databases. Here, the electronic device can be such as a computer, smartphone, tablet computer, laptop computer, PDA, etc. This embodiment does not specifically limit its application.

[0061] It should also be noted that, in the embodiments of this application, the data center includes data clusters in multiple scenarios. There may be one or more for different types of users. For cloud service providers, there may be hundreds of data clusters, while for enterprises or users, there may be less than 10 data clusters. The types of data clusters include dedicated clusters deployed on cloud hosts, managed clusters deployed on pod instances, and clusters in private cloud scenarios.

[0062] S102: In the event of a data cluster registration in the data center, generate a set of backup containers.

[0063] It should be noted that, in the process of monitoring the data center in this embodiment of the application, when a data cluster is detected to be registered, a group of container pods (join, policy, sync) are started to take over the backup and recovery services of the cluster. If there are multiple data clusters, the corresponding number of container groups need to be started to ensure that the backup container and the data cluster maintain a one-to-one correspondence.

[0064] In some embodiments, the backup container may include a join container, a policy container, and a synchronization container. For example, the backup container may be a pod (join, policy, sync).

[0065] S103: The backup container is used to back up the business data in the data cluster to obtain backup data; wherein the backup container is associated with the data cluster.

[0066] It should be noted that in this embodiment, each cluster data is backed up in its own corresponding backup container. During the backup process, the common functions of the original data cluster are not interfered with. An additional backup module is added outside the nodes of the original data cluster to perform independent backup of the data cluster.

[0067] In some embodiments, backing up the business data in the data cluster using the backup container to obtain backup data may include:

[0068] The connection container sends the business data in the data cluster to the synchronization container;

[0069] The backup policy corresponding to the business data is sent to the synchronization container through the policy container; the synchronization container performs backup processing on the business data according to the business data and the backup policy to obtain the backup data.

[0070] It should be noted that, in the embodiments of this application, a backup container includes a connection container, a policy container, and a synchronization container. The connection container adds the backup node to the corresponding data cluster as a node specifically responsible for backup and recovery, in addition to the leader node and member nodes. The policy container can be used to decide the backup time for business data. The synchronization container is used to perform backup of business data according to the backup time.

[0071] In some embodiments, the backup module includes a monitoring module, a policy module, and a synchronization module, and the method may further include:

[0072] The monitoring module monitors the status data of several data clusters in the data center.

[0073] When the status data changes, the corresponding backup strategy for the strategy module is determined;

[0074] The synchronization module performs backup processing according to the backup strategy and stores the backup data in the target file.

[0075] It should be noted that, in this embodiment, when the master node in the data cluster sends incremental information to other nodes, the monitoring module listens to the status data of the data cluster, including Term, raftIndex, and logIndex. In addition, it is also necessary to collect the CPU and memory data of the leader node of the data cluster. This data needs to be pre-entered into the persistent storage module and stored according to the structure (name, IP, CPU, RAM).

[0076] In some embodiments, the status data includes period data and index data, and determining the backup policy corresponding to the policy module when the status data changes may include:

[0077] When the time limit data in the status data increases, the backup time of the incremental data of the data cluster is determined by the strategy module.

[0078] If the time limit data remains unchanged in the status data, but the index data changes, the backup time of the incremental data of the data cluster is determined by the strategy module.

[0079] It should be noted that in this embodiment, a change in the expiration date indicates a change in the leader node. The status of the master node in the data cluster needs to be checked to obtain the latest data cluster status data. If the new expiration date is less than the locally persisted expiration date, it is ignored. If the new expiration date is greater than the locally persisted expiration date, the data is saved to the persistent database, triggering the strategy module to execute. If the expiration date has not changed, but the index data (logIndex, raftIndex) has changed, and the strategy module has already started, there is no need to add the task again. Otherwise, the strategy module needs to be triggered to determine the specific backup time.

[0080] In some embodiments, determining the backup time of incremental data for the data cluster through the policy module may include:

[0081] The first specification threshold is determined based on the maximum memory and the maximum value of the central processing unit in the storage type;

[0082] The second specification threshold is determined based on the current memory and current CPU values;

[0083] The backup time corresponding to the backup strategy is determined based on the first specification threshold and the second specification threshold.

[0084] It should be noted that, in this embodiment of the application, when the strategy module determines the backup time, it specifically calculates the maximum specification threshold B of the platform based on the maximum type of storage cpu(max)*mem(max); it calculates cpu*mem using the currently selected specification to obtain the current specification threshold C; and the value obtained by B / C*f(p)*quota is the backup time for the strategy module to schedule the synchronization module to perform the backup.

[0085] It should also be noted that in the embodiments of this application, f(p) is the IOPS adjustment ratio of the same size performance disk under different specifications. For example, if the IOPS of 2 core 4G (assuming 2000) is the same as the IOPS of 32 core 128G (assuming 10000), then the ratio is 1 / 50. The quota needs to be set according to the calibration value of the platform IOPS. Assuming that the quota is set to 1, real-time backup is performed without considering other factors.

[0086] S104: In the event of loss of the business data, restore the business data in the data cluster based on the backup data.

[0087] It should be noted that in the embodiments of this application, in the case of a brand new data cluster or loss of business data, backup data can be used to restore the business data. Specifically, the backup data can be sent to the master node of the data cluster, and then the master node can distribute it to other nodes.

[0088] In some embodiments, the method may further include:

[0089] If the scenario corresponding to the data cluster is a dedicated cluster, a full backup of the data cluster shall be performed.

[0090] When the scenario corresponding to the data cluster is a managed cluster, incremental backups of the data cluster are performed according to a preset strategy.

[0091] It should be noted that in the embodiments of this application, when the scenario corresponding to the data cluster is a managed cluster, the specified data can be selectively backed up according to the preset strategy. Specifically, the backed-up data can be full data, application data, configuration data, node data, etc., and can be set according to the user's specific needs for certain data in the data cluster. No limitations are made here.

[0092] In summary, this embodiment provides a backup and recovery method. It involves a backup module monitoring the data center; generating a set of backup containers when data cluster registration occurs in the data center; backing up business data in the data cluster using these backup containers to obtain backup data; and establishing a relationship between the backup containers and the data cluster. In the event of business data loss, the business data in the data cluster is restored based on the backup data. This allows the backup module to perform backup and recovery of multiple cluster data in different scenarios without affecting the stability of the data cluster, improving the efficiency of business data backup and reducing operational complexity.

[0093] Example 2

[0094] Based on the same inventive concept as the foregoing embodiments, this application proposes a backup and recovery scheme applicable to multiple scenarios of etcd clusters based on the current etcd cluster. The etcd cluster is usually composed of 3 nodes (1 master + 2 member nodes). In addition, there is only a master node in the cluster, which provides the response to requests.

[0095] See Figure 2 This illustrates a detailed flowchart of a backup and recovery method provided in an embodiment of this application. Figure 2 As shown, a universal etcd backup and recovery solution is designed for etcd clusters in various scenarios. This invention can cover other scenarios such as private clusters of cloud service providers or users, including but not limited to dedicated clusters, managed clusters, private clouds, and hybrid clouds. Figure 1 The architecture diagram includes five elements: etcd clusters for multiple scenarios, etcdBackupCluster, etcd-backup, OBJ, and etcdRestoreCluster.

[0096] (1) etcd clusters in multiple scenarios: There may be one or more for different types of users. For cloud service providers, there are hundreds of container clusters, while for enterprises or users, there may be less than 10 clusters. The types include dedicated clusters deployed on cloud hosts, managed clusters deployed on pod instances, and clusters in private cloud scenarios.

[0097] (2) In order to provide a unified and automated backup management capability for etcd clusters, the etcdBackupCluster resource center was designed. Users can register various clusters and then perform backup management of the clusters.

[0098] (3) The etcd-backup module is used to monitor resources in etcdBackupCluster. Once a resource is registered, a group of pods (join, policy, sync) is started to take over the backup and recovery of the cluster. If there are multiple clusters, the corresponding number of pod groups need to be started.

[0099] (4) Backup service triggers the need to import data into OBJ service;

[0100] (5) An etcdRestoreCluster resource center was designed. Resource objects are constructed according to user business needs. Once created, the corresponding etcd cluster data is restored based on the data in the OBJ and the content of the etcdBackupCluster registry. So how does this solution solve the etcd backup performance problem, design different backup strategies for different scenarios to ensure data integrity, and provide etcd cluster backup solutions for multiple scenarios?

[0101] Prior to this invention, for different clusters, especially for service providers handling hundreds of etcd cluster backups, there was no unified etcd backup solution, or dedicated personnel to maintain cluster data and write backup scripts based on cluster characteristics to achieve regular backups. In this invention, etcdBackupCluster acts as a resource center object. Its fields are designed as (Id, clusterName, clusterType, namespace, conn, cert), representing the cluster ID, cluster name, cluster type (which determines the cluster deployment format, host or pod instance), the deployment domain of the cluster, cluster connection information, and cluster certificate, respectively. Other etcd clusters only need to establish network policies with the resource center and register relevant information there to achieve management of etcd clusters in multiple scenarios. After registering the cluster information, it is monitored by the etcd-sync module, which immediately triggers Kubernetes to start a group of pods containing three services, which are used to join the etcd cluster as nodes with the etcd-sync role. In the current technology, the etcd cluster includes roles such as leader and member. The leader provides services to users, and more than half of the nodes in the cluster need to provide services for the etcd cluster to run stably. Therefore, if the member nodes are expanded according to the current technology, more nodes will need to confirm and reply to the data. The leader node will need to spend more synchronization overhead to each member node. Therefore, as the number of member nodes increases, the performance of the etcd cluster will actually decrease, and the service performance of the etcd cluster will be affected.

[0102] In this scheme, the node with the etcd-sync role acts as the synchronization node of the etcd cluster. It only has the function of synchronizing log data and cluster information. It does not participate in the election or voting of the cluster, that is, it does not affect the service performance of the original etcd cluster.

[0103] Specifically, in the etcd cluster, the leader node is aware of these roles and sends cluster and log information to all roles. However, the leader node confirms the number of successfully sent messages without considering the etcd-sync node. Therefore, etcd-sync can obtain data from the entire etcd cluster incrementally. Currently, member nodes can also receive incremental data from the etcd cluster.

[0104] Would backing up data from member nodes solve the performance issue? Actually, the backup process eliminates a significant amount of read / write overhead. Since etcd member nodes act as the cluster's response nodes, if the leader node receives a large number of requests during the backup process, it may fail to respond synchronously, impacting the leader node's responsiveness and consequently affecting the etcd cluster's performance. If data is backed up from the leader node, the cluster leader will change, severely impacting etcd cluster performance.

[0105] This invention proposes a role that effectively avoids these problems without affecting cluster performance. It receives cluster data from etcd while minimizing the impact on cluster performance, thus ensuring backup efficiency. Furthermore, the next section will describe in detail how to specify fine-grained backup strategies, as mentioned in existing technologies, addressing the differences in pod and cloud host backups for different scenarios. After backing up data to OBJ object storage, this invention designs an etcdRestoreCluster resource object. Once the resource object is created, it immediately triggers a group of pods (restore-pod, restore-policy). Based on different recovery strategies, the pod instances complete the recovery of etcd cluster services.

[0106] Next, we will first describe the etcd-sync module and how it handles clusters in different scenarios. Through its design, it solves the problems of low backup efficiency and incomplete backup in existing technologies.

[0107] See Figure 3 It shows a schematic diagram of the system framework of a backup and recovery method provided in an embodiment of this application, such as... Figure 3As shown, based on the existing etcd cluster, in addition to the three main nodes (leader node, member node, etcd), a new etcd-sync node is designed. It differs from other nodes in that it does not participate in elections or count the number of cluster members; that is, its existence does not affect the previous cluster's leader election rules. However, it has the function of receiving ordinary member data. When the cluster receives a user request message, the master node needs to synchronize it to all other nodes, including the etcd-sync node. It can continuously receive incremental data, thus addressing one of the problems proposed in this invention: efficiently achieving data synchronization without performing a full backup every time. Furthermore, it includes three main modules: watch for real-time incremental data backup capabilities; policy for defining backup data strategies. For each etcd cluster, architectural data often exceeds the data that users care about; the policy needs to formulate rules based on the cluster's performance metrics to achieve lightweight and effective data backup; and sync for real-time backup of incremental data synchronized by nodes to object storage. When an etcd cluster service encounters problems, this invention, in order to recover data, reads the relevant business data from object storage and imports it in batches into the etcd cluster, thereby achieving complete data recovery. This architecture not only maintains the performance of the original etcd cluster but also enables user data backup and recovery services across multiple scenarios.

[0108] How does this solution address the issue of incomplete data backup in existing technologies? The design of the watch module is described in detail below.

[0109] This module monitors the cluster's Term, raftIndex, and logIndex, representing the etcd cluster's status. Additionally, it collects CPU and memory data from the cluster's Leader node. This data needs to be pre-loaded into a persistent storage module and stored according to a (name, IP, CPU, RAM) structure. The monitoring method involves extracting the cluster Term, raftIndex, and logIndex when the etcd cluster master node sends incremental information to each node. If a change in Term, or a change in logIndex or raftIndex, or if no change is detected for an extended period, the following algorithm is executed:

[0110] (1) When a Term changes, it indicates that a leader node has changed. At this time, the etcd cluster is relatively unstable, so it is necessary to detect the status of the etcd master node and obtain the latest cluster status data. If the obtained Term is smaller than the locally persisted Term, it is ignored. In other cases, the data is saved to the persistent database to trigger the execution of the policy module, which will be described later. The function of the policy is to execute the backup strategy.

[0111] (2) When logIndex and raftIndex change, if the policy has already been started, there is no need to add the task again. Otherwise, it is necessary to directly trigger the policy rule to execute the backup strategy.

[0112] (3) If no change occurs beyond the polling timestamp, an attempt is made to obtain cluster status information from the leader node. If the acquisition fails, the process is restarted to join the cluster. If the acquisition is successful and the obtained information is (TermA, raftIndexA, logIndexA), TermA is compared with Term.

[0113] (a) If TermA < Term, ignore it;

[0114] (b) If TermA > Term, trigger the policy to execute the backup strategy;

[0115] (c) If TermA = Term, compare logIndexA with logIndex, and raftIndexA with raftIndex respectively. If logIndex does not change, ignore it; otherwise, obtain the log data between (logIndex, logIndexA) from the master node through the etcd-sync node, and trigger the policy to execute the backup strategy. If raftIndex does not change, ignore it; otherwise, mark the data between (raftIndex, raftIndexA) as a committed state (the specific structure is described in the policy module).

[0116] The watch module designed in this solution needs to cooperate with the policy and sync modules to solve all the problems in the prior art. Then how is the policy designed?

[0117] In the policy module, the persistent structure of valid data is designed as the structure shown below, and its definition and meaning are introduced using Table 1:

[0118] Table 1

[0119]

[0120] Additionally, the etcd cluster node information statistics, depending on the cluster type, for dedicated clusters or private cloud clusters, automatically retrieve the following cluster information based on the etcdBackupCluster certificate and connection information: (name, IP, CPU, RAM, size, type, quota, k8sTypes), representing the name, IP address, CPU size, memory size, storage size, storage type, capacity, and backup service type of the etcd cluster node, respectively. For managed clusters, this basic information is provided by the service provider. With these basic design tables, the policy strategy design process is as follows:

[0121] Storage type represents the scale and performance of the business. Storage types are divided into three levels: Excellent, Medium, and General.

[0122] (i) When the setting is not optimal, it indicates that the business has high real-time requirements, good storage performance, the highest frequency of backup, and the best guarantee against data loss.

[0123] (ii) When set to medium, it indicates that the business has medium real-time requirements, good storage performance, medium frequency of backup, and can perform backup when the business characteristics are met.

[0124] (iii) When set to "Normal," it indicates a test environment with low backup requirements and a tolerance for data loss. After selecting the type, the execution time of the sync process is calculated based on the product of CPU and memory size using the following algorithm. The specific algorithm process is as follows:

[0125] (1) Based on the maximum type of storage of this type on the platform, cpu(max)*mem(max), the maximum specification threshold B of the platform is obtained;

[0126] (2) Calculate CPU*mem using the currently selected specification to obtain the current specification threshold C;

[0127] (3) Using B / C*f(p)*quota, the value obtained is the actual time when the policy scheduler sync performs the backup.

[0128] (4) Where f(p) is the IOPS adjustment ratio of a performance disk of the same size under different specifications. For example, the IOPS of a 2-core 4G disk (assuming 2000) is the same as the IOPS of a 32-core 128G disk (assuming 10000), so the ratio is 1 / 50. The backup time is calculated using the formula B / C*f(p)*quota. The meaning of this formula is that, with a fixed size, the performance is better as the host specification increases, but its baseline needs to be adjusted and calibrated based on the IOPS performance test results. Usually, quota needs to be set according to the calibration value of the platform's IOPS. When the selected storage type is optimal, when quota is set to 1, real-time backup is performed without considering other factors. When other storage types are selected, the quota calibration is set according to business needs. For example, for a general application system, the storage frequency data is backed up once every 20 records, so only this quota needs to be calibrated.

[0129] When the cluster type scenario is a dedicated cluster, the etcd cluster data backup is triggered in an "all" mode, i.e., a full backup, according to the backup strategy. The main design principle is that the etcdRestoreCluster resource is designed in this invention. When it finds that the etcdBackupCluster type is a dedicated cluster, it directly attempts to restore the service from the original cloud host. If the restoration fails, it creates a host node of the same type from the etcd cluster and restores the service (existing technology already meets this requirement). When the cluster scenario is a managed cluster, the backup strategy can be configured by the user, triggering the sync process.

[0130] With the above calculation method, the execution process of sync can be calculated. However, before that, all data arriving at etcd-sync needs to be persisted according to the defined business type. Specifically, the platform divides the stored data into several categories: all, application, configuration, and node.

[0131] When the user sets it to "all", the etcd-sync module does not need to perform the persistence process specified above. Instead, it uses etcd's built-in full data backup method to back up the data from the etcd-sync node. For other settings, it extracts valid data according to the specified data type. Once a match is found in k8sTypes, the valid data is persisted according to the format designed in Table 2 for use in the sync process. So how is the sync process designed?

[0132] The policy module described above has prepared the valid data and stored it according to the specified format. We only need to wait for the time set by the policy to arrive for the sync backup module to take effect. By calling the object storage business methods of various vendors, the data is packaged and backed up as objects, with the backup name using (clusterName_k8sTypes_time). If a full package is used, the default etcd recovery method will be executed during business recovery. For other valid data, recovery will be performed using a batch write method to the etcd cluster. It is crucial to clarify that the recovered data refers to valid data recovered in a completely new etcd cluster or in scenarios where business data has been lost. This will significantly improve the speed and quality of data recovery.

[0133] Based on the description of the above architecture implementation, we introduce the node role, watch module, and policy of etcd-sync to improve the handling of incomplete etcd data in Kubernetes scenarios for different situations. We design an effective data structure for policies to efficiently back up effective data according to the policies, thereby improving backup performance and reducing the impact on the etcd cluster. We propose an incremental backup mode for effective data to reduce the impact of irrelevant data backup.

[0134] This application presents an effective data solution for incremental backup of etcd in Kubernetes scenarios across multiple scenarios, improving the efficiency, performance, and consistency of etcd cluster backups. It proposes a unified platform management approach for multiple scenarios, significantly reducing the backup and recovery management costs for users of multi-cluster data. Backup strategies tailored to different scenario requirements are designed to enhance the effectiveness of etcd cluster backups, making them applicable to real-world projects. Furthermore, it introduces a novel incremental backup method for etcd clusters, separating etcd cluster business consistency from backup, achieving a dual high efficiency of business stability and backup performance.

[0135] This embodiment provides a backup and recovery method. The specific implementation of the aforementioned embodiments has been described in detail. It can be seen that, through the technical solutions of the aforementioned embodiments, this invention, compared to existing backup solutions, achieves both stable cluster services and real-time backup in various etcd scenarios. In existing technical solutions, especially for live network services, it is impossible to distinguish cluster performance, and backup strategies cannot be formulated for different performance scenarios. This invention designs a usable solution without affecting the stability of the original etcd cluster. This invention proposes the ability to back up valid user data, which can greatly reduce the difficulty of user self-maintenance and improve the efficiency of business data backup.

[0136] Example 3

[0137] Based on the same inventive concept as the foregoing embodiments, see [link to previous document]. Figure 4 This illustrates a schematic diagram of the composition of a backup and recovery device 40 provided in an embodiment of this application. Figure 4 As shown, the backup and recovery device 40 may include: a monitoring unit 401, a generation unit 402, a backup unit 403, and a recovery unit 404, wherein,

[0138] The monitoring unit 401 is configured to monitor the data center through the backup module;

[0139] The generation unit 402 is configured to generate a set of backup containers when a data cluster registration occurs in the data center;

[0140] The backup unit 403 is configured to back up the business data in the data cluster through the backup container to obtain backup data; wherein, the backup container is associated with the data cluster.

[0141] The recovery unit 404 is configured to recover the business data in the data cluster based on the backup data in the event of loss of the business data.

[0142] In some embodiments, the backup container includes a connection container, a policy container, and a synchronization container;

[0143] In some embodiments, the backup unit 403 is specifically configured to send business data in the data cluster to the synchronization container through the connection container; and send the backup policy corresponding to the business data to the synchronization container through the policy container; and the synchronization container performs backup processing on the business data according to the business data and the backup policy to obtain the backup data.

[0144] In some embodiments, the backup unit 403 is further configured to monitor the status data corresponding to several data clusters in the data center through the monitoring module; determine the backup strategy corresponding to the strategy module when the status data changes; and perform backup processing according to the backup strategy and store it to a target file.

[0145] In some embodiments, the backup unit 403 is specifically configured to determine the backup time of the incremental data of the data cluster through the strategy module when the time limit data in the status data increases; and to determine the backup time of the incremental data of the data cluster through the strategy module when the time limit data in the status data remains unchanged and the index data changes.

[0146] In some embodiments, the backup unit 403 is specifically configured to determine a first specification threshold based on the maximum memory and the maximum value of the central processing unit in the storage type; and to determine a second specification threshold based on the current memory and the current central processing unit value; and to determine the backup time corresponding to the backup strategy based on the first specification threshold and the second specification threshold.

[0147] In some embodiments, the backup unit 403 is further configured to perform a full backup of the data cluster when the scenario corresponding to the data cluster is a dedicated cluster; and to perform incremental backups of the data cluster according to a preset strategy when the scenario corresponding to the data cluster is a managed cluster.

[0148] Understandably, in this embodiment, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and can also be a module or a non-modular component. Furthermore, the components in this embodiment can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0149] If the integrated unit is implemented as a software functional module and not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method described in this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0150] Therefore, this embodiment provides a computer storage medium storing a backup and recovery program, which, when executed by at least one processor, implements the steps of the method described in any of the foregoing embodiments.

[0151] Based on the composition of the backup and recovery device 40 and the computer storage medium described above, see [link to documentation]. Figure 5 This illustrates a schematic diagram of the specific hardware structure of the electronic device provided in an embodiment of this application. For example... Figure 5As shown, electronic device 50 may include: a communication interface 501, a memory 502, and a processor 503; the various components are coupled together via a bus system 504. It is understood that the bus system 504 is used to implement communication between these components. In addition to a data bus, the bus system 504 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 All buses are labeled as bus system 504. Among them, communication interface 501 is used for receiving and sending signals during information exchange with other external network elements;

[0152] Memory 502 is used to store computer programs that can run on processor 503;

[0153] Processor 503, when running the computer program, performs the following:

[0154] Monitor the data center using the backup module;

[0155] In the event of a data cluster registration in the data center, a set of backup containers is generated;

[0156] The backup container is used to back up the business data in the data cluster to obtain backup data; wherein, the backup container is associated with the data cluster.

[0157] In the event of loss of the business data, the business data in the data cluster is restored based on the backup data.

[0158] It is understood that the memory 502 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDRSDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DRRAM). The memory 502 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0159] The processor 503 may be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed by the integrated logic circuitry in the hardware of the processor 503 or by instructions in software form. The processor 503 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 502, and the processor 503 reads the information in memory 502 and, in conjunction with its hardware, completes the steps of the above method.

[0160] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0161] For software implementation, the techniques described herein can be achieved through modules (e.g., procedures, functions, etc.) that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented within the processor or externally.

[0162] Alternatively, as another embodiment, the processor 503 is further configured to perform the steps of the method described in any of the foregoing embodiments when running the computer program.

[0163] Based on the composition of the backup and recovery device 40 and the computer storage medium described above, see [link to documentation]. Figure 6 This illustrates a schematic diagram of the structural composition of another electronic device 50 provided in an embodiment of this application. For example... Figure 6 As shown, the electronic device 50 may include the backup and recovery device 40 as described in any of the foregoing embodiments.

[0164] In this embodiment, for the electronic device 50, a backup module monitors the data center; in the event of data cluster registration in the data center, a set of backup containers is generated; the backup containers are used to back up the business data in the data cluster to obtain backup data; wherein, the backup containers are associated with the data cluster; in the event of business data loss, the business data in the data cluster is restored based on the backup data. In this way, the backup module can achieve the function of backing up and restoring multiple cluster data in different scenarios without affecting the stability of the data cluster, improving the efficiency of business data backup and reducing the difficulty of operation and maintenance.

[0165] It should be noted that, in this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0166] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0167] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0168] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0169] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0170] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A backup and recovery method, characterized in that, The method includes: Monitor the data center using the backup module; In the event of data cluster registration in the data center, a set of backup containers is generated; wherein, the backup containers include connection containers, policy containers, and synchronization containers; The connection container sends the business data in the data cluster to the synchronization container; the policy container sends the backup policy corresponding to the business data to the synchronization container; the synchronization container performs backup processing on the business data according to the business data and the backup policy to obtain backup data. In the event of loss of the business data, the business data in the data cluster is restored based on the backup data.

2. The method according to claim 1, characterized in that, The backup module includes a monitoring module, a policy module, and a synchronization module, and the method further includes: The monitoring module monitors the status data of several data clusters in the data center. When the status data changes, the corresponding backup strategy for the strategy module is determined; The synchronization module performs backup processing according to the backup strategy and stores the backup data in the target file.

3. The method according to claim 2, characterized in that, The status data includes period data and index data. When the status data changes, determining the backup strategy corresponding to the strategy module includes: When the time limit data in the status data increases, the backup time of the incremental data of the data cluster is determined by the strategy module. If the time limit data remains unchanged in the status data, but the index data changes, the backup time of the incremental data of the data cluster is determined by the strategy module.

4. The method according to claim 3, characterized in that, The step of determining the backup time of incremental data for the data cluster through the strategy module includes: The first specification threshold is determined based on the maximum memory and the maximum value of the central processing unit in the storage type; The second specification threshold is determined based on the current memory and current CPU values; The backup time corresponding to the backup strategy is determined based on the first specification threshold and the second specification threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: If the scenario corresponding to the data cluster is a dedicated cluster, a full backup of the data cluster shall be performed. When the scenario corresponding to the data cluster is a managed cluster, incremental backups of the data cluster are performed according to a preset strategy.

6. A backup and recovery device, characterized in that, The backup and recovery device includes a monitoring unit, a generation unit, a backup unit, and a recovery unit, wherein... The monitoring unit is configured to monitor the data center through the backup module; The generation unit is configured to generate a set of backup containers when a data cluster registration occurs in the data center; wherein the backup containers include a connection container, a policy container, and a synchronization container. The backup unit is configured to send business data in the data cluster to the synchronization container through the connection container; send the backup policy corresponding to the business data to the synchronization container through the policy container; and the synchronization container performs backup processing on the business data according to the business data and the backup policy to obtain backup data. The recovery unit is configured to recover the business data in the data cluster based on the backup data in the event of loss of the business data.

7. An electronic device, characterized in that, The electronic device includes: a memory and a processor; wherein... The memory is used to store computer programs that can run on the processor; The processor is configured to perform the method as described in any one of claims 1 to 5 when running the computer program.

8. A computer storage medium, characterized in that, The computer storage medium stores a computer program that, when executed by at least one processor, implements the method as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Cloud native database backup method based on hybrid cloud

    CN114579364A