Data backup method and system

Abstract

The present application discloses a data backup method and system. The data backup method includes: a monitoring step: make morr ir regularly monitors the network load status; a caching step: when the source cluster receives new data and the network load status is high, the new data is cached in the etcd database of the make morr ir; an output step: when the network load status returns to normal, the new data in the etcd database is output to the target cluster. The present invention proposes a data backup method and system, which can automatically detect the network load status at that time through make morr ir, and write the cached data to the mirror etcd cluster after the load network returns to normal. In addition, synchronous replication can ensure the integrity and real-time performance of etcd cluster data, and semi-synchronous lossless replication can solve the problem of high dependence on io during etcd cluster synchronization, and solve the problem of cross-computer room data backup.

Description

Technical Field

[0001] The present application relates to the technical field of data backup, and in particular to a data backup method and system. Background Art

[0002] As enterprises have higher and higher requirements for continuous integration and continuous deployment, the containerization of enterprise services is an inevitable trend. As the most widely used container governance solution, kubernetes has higher and higher requirements for data security. In order to improve the stability of the cluster, ensuring the data security of the kubernetes cluster has become an urgent problem to be solved. As the data warehouse of the kubernetes cluster, etcd, ensuring the data security in the etcd cluster is an important way to solve the problem. Although etcd clusters can back up each other, in order to ensure foolproof etcd off-site backup, it is necessary to solve. Off-site backup has high requirements for real-time data. ETCD is an extremely important service for the k8s cluster, storing all the data information of the cluster. Similarly, if a disaster occurs or etcd data is lost, it will affect the recovery of cluster data. The most commonly used method at present is to build an etcd cluster with multiple nodes, and the data after the nodes are synchronized with each other. ETCD uses the Raft protocol to maintain the consistency of the status of each node in the cluster. Simply put, the ETCD cluster is a distributed system, which consists of multiple nodes communicating with each other to form an overall external service. Each node stores complete data, and the Raft protocol is used to ensure that the data maintained by each node is consistent. Such as Figure 2 As shown, in the prior art, when a node receives a write request, it needs to wait until all other nodes have written the information before returning the data. In addition, the efficiency of data synchronization within the cluster depends on the IO of the node disk and the IO of the network. If the change operations within the current kubernetes cluster are frequent, if the node disk IO is delayed, it will cause a return timeout.

[0003] Therefore, in view of the above situation, the present invention proposes a data backup method and system. The present invention can automatically detect the network load at that time through makemorrir. If the network load is high at that time, it will automatically start caching and store the data at that time. At the same time, it will check the network load and write the cached data to the mirror etcd cluster after the load is reduced. In addition, synchronous replication can ensure the integrity and real-time performance of etcd cluster data. Semi-synchronous lossless replication can solve the problem of high dependence on io during etcd cluster synchronization and solve the problem of cross-computer room data backup. The completeness of the data is guaranteed, and the timeliness of the backup will not be affected even at the peak of network traffic access. Summary of the invention

[0004] The embodiments of the present application provide a data backup method and system to at least solve the problem of subjective factors in the related art.

[0005] The present invention provides a data backup method, comprising:

[0006] Monitoring steps: make morrir regularly monitors the network load status;

[0007] Caching step: when the source cluster receives new data and the network load is high, cache the new data in the etcd database of make morrir;

[0008] Output step: When the network load status returns to normal, the new data in the etcd database is output to the target cluster.

[0009] In the above-mentioned data backup method, the make morrir includes an etcd database and a detection interface for detecting the network load status.

[0010] In the above data backup method, the monitoring step includes: make morrir regularly monitors the network load status through the detection interface.

[0011] In the above-mentioned data backup method, the caching step includes: when the source cluster receives new data and the detection interface detects that the network load state is high, the make morrir uses a hash algorithm to obtain the hash value of the new data, and after marking the new data with the hash value, the new data is cached in the etcd database of the make morrir.

[0012] In the above-mentioned data backup method, the output step includes that make morrir continues to regularly monitor the network load status through the detection interface, and when the detection interface detects that the network load status has returned to normal, the new data in the etcd database is output to the target cluster.

[0013] The present invention also provides a data backup system, which is applicable to the above-mentioned data backup method, and the data backup system comprises:

[0014] Monitoring unit: make morrir regularly monitors the network load status;

[0015] Cache unit: when the source cluster receives new data and the network load is high, cache the new data into the etcd database of make morrir;

[0016] Output unit: When the network load status returns to normal, the new data in the etcd database is output to the target cluster.

[0017] The above-mentioned data backup system, the make morrir includes an etcd database and a detection interface for detecting the network load status.

[0018] In the above-mentioned data backup system, the make morrir in the monitoring unit periodically monitors the network load status through the detection interface in the monitoring unit.

[0019] In the above-mentioned data backup system, when the source cluster receives new data and the detection interface detects that the network load state is high, the make morrir uses a hash algorithm to obtain the hash value of the new data, and after marking the new data with the hash value, caches the new data into the etcd database of the make morrir through the cache unit.

[0020] In the above-mentioned data backup system, the make morrir continues to regularly monitor the network load status through the detection interface. When the detection interface detects that the network load status has returned to normal, the new data in the etcd database is output to the target cluster through the output unit.

[0021] Compared with the related art, the present invention proposes a data backup method and system. The present invention can automatically detect the network load at that time through make morrir. If the network load is high at that time, it will automatically start caching and store the data at that time. At the same time, it will check the network load and write the cached data to the mirror etcd cluster after the load is reduced. In addition, synchronous replication can ensure the integrity and real-time performance of etcd cluster data. Semi-synchronous lossless replication can solve the problem of high dependence on io during etcd cluster synchronization and solve the problem of cross-computer room data backup. The completeness of the data is guaranteed, and the timeliness of the backup will not be affected even at the peak of network traffic access.

[0022] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0024] Figure 1 is a flow chart of a data backup method according to an embodiment of the present application;

[0025] Figure 2 It is a prior art framework diagram according to an embodiment of the present application;

[0026] Figure 3 It is an architectural framework diagram according to an embodiment of the present application;

[0027] Figure 4 It is a structural schematic diagram of the data backup system of the present invention;

[0028] Figure 5 is a framework diagram of an electronic device according to an embodiment of the present application.

[0029] Wherein, the accompanying drawings are marked as follows:

[0030] Monitoring unit: 51;

[0031] Cache units: 52;

[0032] Output units: 53;

[0033] 81: processor;

[0034] 82: memory;

[0035] 83: Communication interface;

[0036] 80: Bus. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0038] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the disclosure of the present application, some changes in design, manufacturing or production based on the technical content disclosed in the present application are just conventional technical means, and should not be understood as the disclosure of the present application is insufficient.

[0039] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0040] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0041] The present invention is based on etcd cluster, which is briefly introduced below.

[0042] Etcd is a distributed, consistent key-value storage, mainly used for shared configuration and service discovery. Etcd has been widely used in many distributed systems. All distributed systems face the problem of data sharing between multiple nodes. This is the same as teamwork. Members can work separately, but they always need to share some necessary information, such as who is the leader, who are the members, and the order coordination between dependent tasks. Therefore, distributed systems either implement a reliable shared storage to synchronize information (such as Elasticsearch) or rely on a reliable shared storage service, and Etcd is such a service. Etcd provides the following functions: interfaces for storing and retrieving data. It ensures the strong consistency of data of multiple nodes in the Etcd cluster through protocols. It is used to store metadata and share configuration; monitoring mechanism, the client can monitor the changes of a key or some keys (the mechanisms of v2 and v3 are different, see the following article). It is used to monitor and push changes; key expiration and renewal mechanism, the client implements renewal through timed refresh (the implementation mechanisms of v2 and v3 are also different). Used for cluster monitoring and service registration and discovery; provides atomic CAS (Compare-and-Swap) and CAD (Compare-and-Delete) support (v2 is implemented through interface parameters, v3 is implemented through batch transactions). Used for distributed locks and leader election. etcd is a reliable distributed KV storage. Its underlying layer uses the Raft algorithm to ensure consistency and is mainly used for shared configuration and service discovery. etcd is an open source project initiated by CoreOS. At present, there are still many components that provide configuration sharing and service discovery functions. Among them, the most widely used and most familiar one should be ZooKeeper. Many open source projects rely on ZooKeeper to varying degrees, such as Dubbo and Kafka. In the Golang community, etcd is the only component that can rival ZooKeeper. In some aspects, etcd even surpasses ZooKeeper, which gives developers a bright feeling. As an excellent distributed KV storage product, etcd's underlying etcd-raft module implements the Raft protocol, which can help developers quickly implement the final consistency function. etcd has been favored by more and more developers and has become famous in the Golang community for its advantages such as high performance, easy maintenance, and Raft implementation. There are many highlights in the etcd code. In order to improve the overall performance, a large number of goroutines and channels are used in its implementation.In etcd 3, BoltDB is used as the underlying persistent storage, BTree index is used to accelerate query, reliable Watcher function is provided, and a new version of client based on GRPC is provided. Of course, etcd 3 is still compatible with the memory storage and HTTP API of etcd 2. If the nodes of the same etcd cluster are distributed in different computer rooms, the requirements for the disk io and network io of each server are very high. This method of backup is only suitable for the same computer room, not for cross-computer room backup. However, in actual production, off-site computer room backup is also a very important aspect. Therefore, during the backup process, a mirror cluster can be configured for etcd. The mirror cluster can be inconsistent with the original cluster configuration. It can be the same cluster configuration or a single node. The two etcd clusters are connected using the make morrir program. In the process of backing up the two clusters, the make morrir program is the most critical. The make morrir program can read the data of the source etcd cluster and write it to the destination etcd cluster synchronously or semi-synchronously, which is used to solve the cross-computer room backup problem of etcd clusters.

[0043] Kubernetes, or K8s for short, is an abbreviation that uses 8 to replace the 8 characters "ubernete". It is an open source containerized application management platform for multiple hosts. The goal of Kubernetes is to make the deployment of containerized applications simple and powerful. Kubernetes provides a mechanism for application deployment, planning, updating, and maintenance. The traditional way to deploy applications is to install applications through plug-ins or scripts. The disadvantage of this is that the operation, configuration, management, and all life cycles of the application will be bound to the current operating system, which is not conducive to the application upgrade / rollback operations. Of course, some functions can also be achieved by creating virtual machines, but virtual machines are very heavy and not conducive to portability. The new way is to deploy containers. Each container is isolated from each other, each container has its own file system, and the processes between containers will not affect each other, and computing resources can be distinguished. Compared with virtual machines, containers can be deployed quickly. Since containers are decoupled from the underlying facilities and machine file systems, they can be migrated between different clouds and different versions of operating systems. Containers take up less resources and are quick to deploy. Each application can be packaged into a container image. The one-to-one relationship between each application and the container also makes containers more advantageous. Using containers, you can create container images for applications at the build or release stage, because each application does not need to be combined with the rest of the application stack, nor does it rely on the production environment infrastructure, which enables a consistent environment from R&D to testing and production. Similarly, containers are lighter and more "transparent" than virtual machines, which makes them easier to monitor and manage. Kubernetes has the following features: support for public clouds, private clouds, hybrid clouds, and multi-clouds; modularity, plug-in, mountability, composability; automatic deployment, automatic restart, automatic replication, and automatic scaling / extension. Kubernetes is an open source container orchestration engine from Google that supports automated deployment, large-scale scalability, and application container management. When deploying an application in a production environment, you usually deploy multiple instances of the application to load balance application requests. In Kubernetes, we can create multiple containers, run an application instance in each container, and then use the built-in load balancing strategy to manage, discover, and access this group of application instances. These details do not require the operation and maintenance personnel to perform complex manual configuration and processing.

[0044] The present invention proposes a data backup method and system. The present invention can automatically detect the network load at that time through make morrir. If the network load is high at that time, it will automatically start caching and store the data at that time, and check the network load at the same time. After the load is reduced, the cached data is written to the mirror etcd cluster. In addition, synchronous replication can ensure the integrity and real-time performance of etcd cluster data, and semi-synchronous lossless replication can solve the problem of high dependence on io during etcd cluster synchronization, and solve the problem of cross-computer room data backup. The completeness of the data is guaranteed, and the timeliness of the backup will not be affected even at the peak of network traffic access.

[0045] The following describes the embodiments of the present application by taking data backup as an example.

[0046] Embodiment 1

[0047] This embodiment provides a data backup method. Figures 1 to 3 , Figure 1 is a flow chart of a data backup method according to an embodiment of the present application; Figure 2 It is a prior art framework diagram according to an embodiment of the present application; Figure 3 : is an architectural framework diagram according to an embodiment of the present application. As shown in the figure, the data backup method includes the following steps:

[0048] Monitoring step S1: make morrir regularly monitors the network load status;

[0049] Caching step S2: when the source cluster receives new data and the network load state is high, the new data is cached in the etcd database of the make morrir;

[0050] Output step S3: When the network load status returns to normal, the new data in the etcd database is output to the target cluster.

[0051] In the embodiment, the make morrir includes an etcd database and a detection interface for detecting network load status.

[0052] In the specific implementation, first in the etcd cluster, the make morrir program pretends to be a transparent node in the source cluster, only performs query behavior, and does not participate in etcd leader election and voting. In the target cluster, it writes data as a client.

[0053] In the embodiment, the monitoring step S1 includes: make morrir periodically monitors the network load status through the detection interface.

[0054] In an embodiment, the caching step S2 includes: when the source cluster receives new data and the detection interface detects that the network load state is high, the make morrir uses a hash algorithm to obtain the hash value of the new data, and after marking the new data with the hash value, the new data is cached in the etcd database of the make morrir.

[0055] In a specific implementation, when the source cluster receives new data and the detection interface detects that the network load status is high, make morrir reads the data changes in the etcd cluster, and uses a hash algorithm to obtain the hash value of the new data, and after marking the new data with the hash value, caches the new data in the etcd database of make morrir.

[0056] In an embodiment, the output step S3 includes that make morrir continues to regularly monitor the network load status through the detection interface, and when the detection interface detects that the network load status returns to normal, outputs the new data in the etcd database to the target cluster.

[0057] In the specific implementation, after the new data is cached in the etcd database of make morrir, make morrir continues to monitor the network load status regularly through the detection interface. When the detection interface detects that the network load status has returned to normal, the new data in the etcd database is output to the target cluster. etcd will automatically choose to enable semi-synchronous replication or synchronous replication mode based on the forward network io and disk io.

[0058] Therefore, the present invention proposes a data backup method and system. The present invention can automatically detect the network load at that time through make morrir. If the network load is high at that time, it will automatically start caching and store the data at that time. At the same time, it will check the network load and write the cached data to the mirror etcd cluster after the load is reduced. In addition, synchronous replication can ensure the integrity and real-time performance of etcd cluster data. Semi-synchronous lossless replication can solve the problem of high dependence on io during etcd cluster synchronization and solve the problem of cross-computer room data backup. The completeness of the data is guaranteed, and the timeliness of the backup will not be affected even at the peak of network traffic access.

[0059] Embodiment 2

[0060] Please refer to Figure 4 , Figure 4 FIG. 1 is a schematic diagram of the structure of the data backup system of the present invention. Figure 4 As shown, the data backup system of the invention is applicable to the above-mentioned data backup method, and the data backup system includes:

[0061] Monitoring unit 51: make morrir monitors the network load status regularly;

[0062] Cache unit 52: When the source cluster receives new data and the network load state is high, cache the new data in the etcd database of make morrir;

[0063] Output unit 53: When the network load status returns to normal, the new data in the etcd database is output to the target cluster.

[0064] In this embodiment, the make morrir includes an etcd database and a detection interface for detecting network load status.

[0065] In this embodiment, the make morrir in the monitoring unit 51 periodically monitors the network load status through the detection interface in the monitoring unit 51.

[0066] In this embodiment, when the source cluster receives new data and the detection interface detects that the network load state is high, the make morrir uses a hash algorithm to obtain the hash value of the new data, and after marking the new data with the hash value, the new data is cached in the etcd database of the make morrir through the cache unit 52.

[0067] In this embodiment, the make morrir continues to regularly monitor the network load status through the detection interface. When the detection interface detects that the network load status returns to normal, the new data in the etcd database is output to the target cluster through the output unit 53.

[0068] Embodiment 4

[0069] Combination Figure 5 As shown, this embodiment discloses a specific implementation of an electronic device. The electronic device may include a processor 81 and a memory 82 storing computer program instructions.

[0070] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0071] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, FPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EFPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0072] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0073] The processor 81 implements any data backup method in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0074] In some of the embodiments, the electronic device may further include a communication interface 83 and a bus 80. Figure 5 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0075] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication interface 83 can also implement data communication with other components such as: external devices, image / data backup devices, databases, external storage, and image / data processing workstations.

[0076] The bus 80 includes hardware, software or both, and couples the components of the electronic device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0077] Electronic devices can be connected to data backup systems to achieve integration Figures 1 to 3 Describe the method.

[0078] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0079] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A data backup method, characterized in that: include: Monitoring step: make morrir regularly monitors the network load status, wherein make morrir includes a detection interface between the etcd database and the network load status; Caching step: when the source cluster receives new data and the network load is high, the new data is cached in the etcd database of the make morrir; Output step: when the network load status returns to normal, the new data in the etcd database is output to the target cluster; The caching step includes: when the source cluster receives new data and the detection interface detects that the network load state is high, the make morrir uses a hash algorithm to obtain a hash value of the new data, and after marking the new data with the hash value, caches the new data in the etcd database of the make morrir; The output step further includes that after the new data is cached in the etcd database of make morrir, makemorrir continues to regularly monitor the network load status through the detection interface.

2. A data backup system, characterized in that: The data backup method according to claim 1 above is applicable, wherein the data backup system comprises: Monitoring unit: make morrir regularly monitors the network load status, and the make morrir includes the etcd database and the detection interface for detecting the network load status; Cache unit: when the source cluster receives new data and the network load is high, cache the new data in the etcd database of make morrir; Output unit: when the network load status returns to normal, output the new data in the etcd database to the target cluster; When the source cluster receives new data and the detection interface detects that the network load state is high, the make morrir uses a hash algorithm to obtain a hash value of the new data, and after marking the new data with the hash value, caches the new data into the etcd database of the make morrir through a cache unit; Among them, after the new data is cached in make morrir's etcd database, make morrir continues to monitor the network load status regularly through the detection interface.

Images