Consistency hot backup method and device for distributed database cluster

Abstract

The invention relates to the technical field of distributed database management, and provides a consistent hot backup method and system for a distributed database cluster. The method comprises the following steps: performing clock synchronization on each node in a distributed database cluster; performing physical backup on total data of the cluster after clock synchronization; backing up the incremental logic log in the physical backup process until data migration does not exist in the physical backup process; recording a time stamp without data migration in the physical backup process as an overall ending site; and the total data and the incremental logic log are played back to the overall end site, and consistent hot backup of the distributed database cluster is completed. According to the method, services can be not stopped, cluster fragment data can be migrated at will, and global consistency and high efficiency backup can be carried out on cluster backup data.

Description

Technical Field

[0001] The present invention relates to the technical field of distributed database management, and in particular to a consistent hot backup method and system for a distributed database cluster. Background Art

[0002] Distributed database clusters can be expanded horizontally. When the amount of business data expands rapidly, the cluster shards can be expanded horizontally to store massive data and improve throughput. Now the amount of system data of major users is very large, and the amount of business data is growing rapidly, and the demand for distributed database clusters is becoming more and more vigorous. In order to ensure data security, the system needs to be backed up frequently to facilitate subsequent data rollback and recovery.

[0003] The traditional backup method for distributed database clusters usually involves backing up each cluster shard separately. This data backup method has two obvious defects. First, the cluster cannot be restored to a global consistency point unless business data writing is stopped during the backup. Second, data migration cannot be performed within the cluster shard, which will result in data loss or data duplication. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the related art. To this end, the present invention provides a consistent hot backup method and system for a distributed database cluster.

[0005] The present invention provides a consistent hot backup method for a distributed database cluster, comprising:

[0006] S1: synchronize the clocks of all nodes in the distributed database cluster;

[0007] S2: Physically back up all data of the cluster after clock synchronization;

[0008] S3: Backing up the incremental logical logs in the physical backup process until there is no data migration in the physical backup process;

[0009] S4: Recording the timestamp of no data migration in the physical backup process as the overall end point;

[0010] S5: replay the full data and the incremental logical log to the overall end point to complete the consistent hot backup of the distributed database cluster.

[0011] According to a consistent hot backup method for a distributed database cluster provided by the present invention, in step S1, clock synchronization is performed on each node in the distributed database cluster according to the NTP protocol.

[0012] According to a consistent hot backup method for a distributed database cluster provided by the present invention, step S1 further comprises:

[0013] S11: Receive UTC Coordinated Universal Time from the clock source by configuring the server;

[0014] S12: The distributed database cluster sends a time acquisition request to the configuration server to obtain the UTC Coordinated Universal Time;

[0015] S13: Each node in the distributed database cluster obtains the UTC Coordinated Universal Time from the distributed database cluster, and calculates the node's own time offset and round-trip delay according to the UTC Coordinated Universal Time;

[0016] S14: Each node in the distributed database cluster corrects its own time according to the time offset and round-trip delay calculated by the node itself to complete clock synchronization.

[0017] According to a consistent hot backup method for a distributed database cluster provided by the present invention, the node types in the distributed database cluster include management nodes and storage nodes.

[0018] According to a consistent hot backup method for a distributed database cluster provided by the present invention, step S2 further comprises:

[0019] S21: Start physical backup of all data of each node in the cluster after clock synchronization;

[0020] S22: Determine whether the physical backup of the management node and the storage node has been completed. If not, continue to perform physical backup. If both are completed, execute step S23.

[0021] S23: Record the timestamp when the physical backup of the management node and the storage node are completed as the first point end point.

[0022] According to a consistent hot backup method for a distributed database cluster provided by the present invention, step S3 further comprises:

[0023] S31: Determine whether there is data migration during the physical backup process, if there is data migration, execute step S32, if not, execute step S33;

[0024] S32: Back up the incremental logical logs in the physical backup process to the first sub-point end point, and record the timestamp of the completion of the current incremental logical log as the second sub-point end point;

[0025] S33: Back up the incremental logical logs in the physical backup process. When there is no data migration in the physical backup process, record the timestamp of the completion of the current incremental logical log as the overall end point.

[0026] According to a consistent hot backup method for a distributed database cluster provided by the present invention, the second division end point in step S32 is the same as the first division end point.

[0027] The present invention also provides a consistent hot backup device for a distributed database cluster, comprising:

[0028] Time synchronization module: used to synchronize the clocks of each node in the distributed database cluster;

[0029] Physical backup module: used to physically back up the full data of the cluster after clock synchronization;

[0030] Incremental logic backup module: used to back up the incremental logic logs in the physical backup process until there is no data migration in the physical backup process, and record the timestamp of no data migration in the physical backup process as the overall end point;

[0031] Backup playback module: used to play back the full amount of data and the incremental logical log to the overall end point.

[0032] The present invention also provides a consistent hot backup device for a distributed database cluster, comprising:

[0033] A memory and at least one processor, wherein instructions are stored in the memory;

[0034] At least one of the processors calls the instruction in the memory to enable the document database audit device to execute a consistent hot backup method for a distributed database cluster as described in any one of the above items.

[0035] The present invention also provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, a consistent hot backup method for a distributed database cluster as described in any one of the above items is implemented.

[0036] The present invention provides a consistent hot backup method and device for a distributed database cluster. The NTP protocol is used to ensure the clock synchronization of each node in the distributed database cluster, and the cluster backup is assisted to form a global consistent site, so as to facilitate the rollback and recovery to the specified site. In addition, the physical full backup and the logical incremental log backup are combined to improve the backup efficiency. In addition, for the internal data migration of the cluster, if the sharding algorithm of the traditional distributed cluster is not designed reasonably, it will lead to the problem of data imbalance between the shards and the inability to migrate autonomously when the shards are backed up separately. The data between the shards can be migrated at will without affecting the consistency of the cluster data.

[0037] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 It is a flow chart of a consistent hot backup method of a distributed database cluster provided by an embodiment of the present invention;

[0040] Figure 2 It is a schematic diagram of the structure of a consistent hot backup device for a distributed database cluster provided by an embodiment of the present invention;

[0041] Figure 3 It is a schematic diagram of a consistent hot backup method for a distributed database cluster provided by an embodiment of the present invention.

[0042] Reference numerals:

[0043] 100, time synchronization module; 200, physical backup module; 300, incremental logical backup module; 400, backup playback module. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and advantages of the present invention clearer, the technical scheme of the present invention will be clearly and completely described below in conjunction with the drawings in the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0045] In the description of the embodiments of the present invention, it should be noted that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside" and the like indicate positions or positional relationships based on the positions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limitations on the embodiments of the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance.

[0046] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "connected" and "connection" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium. For ordinary technicians in this field, the specific meanings of the above terms in the embodiments of the present invention can be understood according to specific circumstances.

[0047] In the embodiments of the present invention, unless otherwise clearly specified and limited, the first feature being "above" or "below" the second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, the first feature being "above", "above" and "above" the second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. The first feature being "below", "below" and "below" the second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0048] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0049] In order to better understand the embodiments of the present invention, the special terms appearing in the present invention are explained below.

[0050] Distributed: A system consisting of multiple interconnected nodes that communicate and coordinate through a network to jointly provide services, aiming to achieve higher performance, reliability, scalability, and fault tolerance.

[0051] Database cluster: refers to connecting multiple database instances together. The cluster can be horizontally expanded, provide high throughput, and store massive amounts of data.

[0052] Global consistency: refers to maintaining data consistency between nodes in a distributed system.

[0053] Hot backup: back up data without stopping business.

[0054] Combine the following Figures 1 to 3 Embodiments of the present invention are described.

[0055] The present invention provides a consistent hot backup method for a distributed database cluster, comprising:

[0056] S1: synchronize the clocks of all nodes in the distributed database cluster;

[0057] Furthermore, the distributed database cluster consists of computing nodes, management nodes and storage nodes. Computing nodes do not store data, but mainly provide cluster routing functions. By horizontally expanding computing nodes, the database throughput can be provided. Management nodes mainly store cluster metadata configuration information. Storage nodes are used to store business data. When business data expands, massive data can be stored directly by expanding storage nodes.

[0058] Wherein, in step S1, clock synchronization is performed on each node in the distributed database cluster according to the NTP protocol.

[0059] Wherein, step S1 further comprises:

[0060] S11: Receive UTC Coordinated Universal Time from the clock source by configuring the server;

[0061] S12: The distributed database cluster sends a time acquisition request to the configuration server to obtain the UTC Coordinated Universal Time;

[0062] S13: Each node in the distributed database cluster obtains the UTC Coordinated Universal Time from the distributed database cluster, and calculates the node's own time offset and round-trip delay according to the UTC Coordinated Universal Time;

[0063] S14: Each node in the distributed database cluster corrects its own time according to the time offset and round-trip delay calculated by the node itself to complete clock synchronization.

[0064] Furthermore, the specific process of time synchronization in step S1 is as follows: the client sends an NTP request message to the server at time t0; the NTP request message arrives at the server at time t1, and the server sends an NTP response message after processing at time t2. The response message carries the timestamp t0 of the message leaving the NTP client, the timestamp t1 of the message arriving at the NTP server, and the timestamp t2 of the message leaving the NTP server; when the client receives the response message, it records the timestamp t3 of the message returned.

[0065] Furthermore, the above process can be used to calculate that the Client synchronizes its clock by calculating its time offset θ and round-trip delay δ, specifically the time offset θ:

[0066]

[0067] Round trip delay δ:

[0068] δ=(t3-t0)-(t2-t1).

[0069] S2: Physically back up all data of the cluster after clock synchronization;

[0070] The node types in the distributed database cluster include management nodes and storage nodes.

[0071] Wherein, step S2 further comprises:

[0072] S21: Start physical backup of all data of each node in the cluster after clock synchronization;

[0073] S22: Determine whether the physical backup of the management node and the storage node has been completed. If not, continue to perform physical backup. If both are completed, execute step S23.

[0074] S23: Record the timestamp when the physical backup of the management node and the storage node are completed as the first point end point.

[0075] S3: Backing up the incremental logical logs in the physical backup process until there is no data migration in the physical backup process;

[0076] Wherein, step S3 further comprises:

[0077] S31: Determine whether there is data migration during the physical backup process, if there is data migration, execute step S32, if not, execute step S33;

[0078] S32: Back up the incremental logical logs in the physical backup process to the first sub-point end point, and record the timestamp of the completion of the current incremental logical log as the second sub-point end point;

[0079] S33: Back up the incremental logical logs in the physical backup process. When there is no data migration in the physical backup process, record the timestamp of the completion of the current incremental logical log as the overall end point.

[0080] The second sub-end point in step S32 is the same as the first sub-end point.

[0081] like Figure 3 As shown, cluster data backup is started at time t0, node 2 completes full data backup at time t1, node 1 completes full data backup at time t2, and node 3 completes full data backup at time t3. Traditional database cluster backup will generate three backup files corresponding to the three nodes respectively, and the global consistency backup position of the cluster is not realized.

[0082] The method provided by the present invention enables the incremental logical log backup of all nodes in the cluster at time t0, backs up the incremental logical logs between time t0 and time t3, takes t3 as a backup end point, completes the global consistency backup of the cluster, and when the cluster data is restored, the distributed database cluster is restored to the t3 point by combining the full node data and the incremental logical log playback.

[0083] S4: Recording the timestamp of no data migration in the physical backup process as the overall end point;

[0084] like Figure 3 As shown in the figure, the following describes the scenario where there is internal data migration during the cluster backup process. During the backup process, the data of node three is migrated to node two. When the incremental logical log is backed up to the full end point t3, it is detected that there is data migration in this time period. Backup to this point is not allowed, and incremental log backup needs to continue. When t4 detects that there is no data migration, the cluster backup can be ended, and t4 is used as the consistency backup end point.

[0085] S5: replay the full data and the incremental logical log to the overall end point to complete the consistent hot backup of the distributed database cluster.

[0086] In some other embodiments, first confirm whether the clocks of each node in the distributed database cluster are synchronized. After the NTP clock synchronization is completed, the cluster data backup instruction is initiated, and the system starts to back up the cluster data. For the full data, physical backup is used, which is more efficient. At the same time, the logical incremental log of the cluster is backed up. When the full data backup of the management node and the storage node is completed, the end point t is recorded, and the logical incremental log is backed up to point t to end the cluster backup.

[0087] The present invention also provides a consistent hot backup device for a distributed database cluster, comprising:

[0088] Time synchronization module 100: used to synchronize the clocks of the nodes in the distributed database cluster;

[0089] Physical backup module 200: used to perform physical backup of the full data of the cluster after clock synchronization;

[0090] Incremental logic backup module 300: used to back up the incremental logic logs in the physical backup process until there is no data migration in the physical backup process, and record the timestamp of no data migration in the physical backup process as the overall end position;

[0091] The backup playback module 400 is used to play back the full amount of data and the incremental logical log to the overall end point.

[0092] The present invention also provides a consistent hot backup device for a distributed database cluster, comprising:

[0093] A memory and at least one processor, wherein instructions are stored in the memory;

[0094] At least one of the processors calls the instruction in the memory to enable the document database audit device to execute a consistent hot backup method for a distributed database cluster as described in any one of the above items.

[0095] In some embodiments, the device may have relatively large differences due to different configurations or performances, and may include one or more processors, for example, one or more processors and memories, one or more storage media storing applications or data, such as one or more mass storage devices. Among them, the memory and storage medium can be short-term storage or persistent storage. The program stored in the storage medium may include one or more modules, each module may include a series of instruction operations in the device. Furthermore, the processor may be configured to communicate with the storage medium and execute a series of instruction operations in the storage medium on the device.

[0096] The device may also include one or more power supplies, one or more wired or wireless network interfaces, one or more input and output interfaces, and / or one or more operating systems, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that the device structure shown above does not constitute a limitation on the device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange components differently.

[0097] The present invention also provides a computer-readable storage medium, on which instructions are stored, and when the instructions are executed by a processor, a consistent hot backup method for a distributed database cluster as described in any one of the above items is implemented.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] If the integrated unit is implemented in the form of a software functional unit and sold or passed as an independent product, it can be stored in a computer-readable storage medium. Based on the understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0100] The present invention provides a method, device, equipment and storage medium for consistent hot backup of a distributed database cluster, which well solves the problem of database cluster backup requirements. It is a method for realizing global consistent hot backup of a distributed database cluster by combining physical full backup and logical incremental log backup with NTP clock synchronization. It has the advantages of no business interruption, high backup efficiency, arbitrary migration of cluster shard data, and global consistency of cluster backup data.

[0101] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A consistent hot backup method for a distributed database cluster, characterized in that: include: S1: Synchronize the clocks of all nodes in the distributed database cluster; S2: Performs a physical backup of all data in the cluster after clock synchronization. S3: Backing up the incremental logical logs during the physical backup process until there is no data migration during the physical backup process; S4: Recording the timestamp of no data migration during the physical backup process as the overall end point; S5: replay the full data and the incremental logical log to the overall end point to complete the consistent hot backup of the distributed database cluster.

2. The consistent hot backup method of a distributed database cluster according to claim 1, characterized in that: In step S1, clock synchronization is performed on each node in the distributed database cluster according to the NTP protocol.

3. The consistent hot backup method of a distributed database cluster according to claim 2, characterized in that: Step S1 further comprises: S11: Receive UTC Coordinated Universal Time from the clock source by configuring the server; S12: The distributed database cluster sends a time acquisition request to the configuration server to obtain the UTC coordinated universal time; S13: Each node in the distributed database cluster obtains the UTC from the distributed database cluster, and calculates its own time offset and round-trip delay based on the UTC; S14: Each node in the distributed database cluster corrects its own time based on the time offset and round-trip delay calculated by the node itself to complete clock synchronization.

4. The consistent hot backup method of a distributed database cluster according to claim 1, characterized in that: The node types in the distributed database cluster include management nodes and storage nodes.

5. The consistent hot backup method of a distributed database cluster according to claim 4, characterized in that: Step S2 further comprises: S21: Start physical backup of all data on each node in the cluster after clock synchronization. S22: Determine whether the physical backup of the management node and the storage node has been completed. If not, continue to perform the physical backup. If both are completed, execute step S23. S23: Record the timestamp when the physical backup of the management node and the storage node is completed as the first minute end point.

6. The consistent hot backup method of a distributed database cluster according to claim 5, characterized in that: Step S3 further comprises: S31: Determine whether data migration occurs during the physical backup process. If data migration occurs, execute step S32; if not, execute step S33. S32: Back up the incremental logical logs during the physical backup process to the first partition end point, and record the timestamp of the completion of the current incremental logical log as the second partition end point; S33: Back up the incremental logical logs during the physical backup process. When there is no data migration during the physical backup process, record the timestamp of the completion of the current incremental logical log as the overall end point.

7. The consistent hot backup method of a distributed database cluster according to claim 6, characterized in that: The second sub-point end point in step S32 is the same as the first sub-point end point.

8. A consistent hot backup device for a distributed database cluster, characterized in that: include: Time synchronization module: used to synchronize the clocks of each node in the distributed database cluster; Physical backup module: used to perform physical backup of all cluster data after clock synchronization; Incremental logical backup module: used to back up the incremental logical logs during the physical backup process until there is no data migration during the physical backup process, and record the timestamp of no data migration during the physical backup process as the overall end point; Backup playback module: used to play back the full data and the incremental logical log to the overall end point.

9. A consistent hot backup device for a distributed database cluster, characterized in that: include: a memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instruction in the memory to enable the document database audit device to execute the consistent hot backup method for a distributed database cluster according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed by the processor, the consistent hot backup method for a distributed database cluster according to any one of claims 1 to 7 is implemented.

Images