A method for full backup and recovery of a distributed database with breakpoint resumption

Abstract

This invention provides a method for full backup and recovery of a distributed database with breakpoint resumption, comprising the following steps: distributed database backup environment check; distributed database backup; breakpoint resumption of distributed database backup; distributed database recovery; and breakpoint resumption of distributed database recovery. The beneficial effects of this invention are: it solves the problem that full backup or recovery is too time-consuming and prone to failure.

Description

Technical Field

[0001] This invention belongs to the field of database technology, and in particular relates to a method for full backup and recovery of a distributed database with breakpoint resumption. Background Technology

[0002] Data backup and recovery is a fundamental function of databases. As the volume of data stored in distributed databases continues to grow exponentially, the time required for a full backup and recovery is also increasing, potentially taking several hours or even days. If a node restarts or network issues occur during this process, the backup and recovery will fail. If users need to perform a full backup or recovery again, the time commitment is prohibitively long. Summary of the Invention

[0003] In view of this, the present invention aims to propose a method for full backup and recovery of distributed databases with interrupted resume, so as to solve the problem that full backup or recovery is too time-consuming and prone to failure.

[0004] To achieve the above objectives, the technical solution of the present invention is implemented as follows: A method for full backup and recovery of a distributed database with breakpoint resumption includes the following steps: S1. Distributed database backup environment check; S2, Distributed Database Backup; S3, Distributed Database Backup with Resume Functionality; S4, Distributed Database Recovery; S5, Distributed Database Recovery: Resuming interrupted downloads; In step S3, the distributed database backup resume function includes: S31. Issue a backup command, with the command parameters specifying whether it is a new backup or a re-backup of a failed backup; S32. Traverse the data fragments and write the backup machine's control file 1. The content to be written includes the ID of the data file to be backed up and the start log file sequence number start_lsn. S33. Data is transmitted in segments, and the control file 2 of the backup machine is written, including the data file transmission status. S34. Re-back up the data files that failed to be backed up; S35. Re-back up the updated data files; S36. If the data consistency point is created successfully, write the control file 2 of the backup machine, including the data consistency point log file sequence number stop_lsn; if the data consistency point is created unsuccessfully, traverse the data files and determine whether the data files have been updated, and back up the updated data files again. S37. Based on the starting log file sequence number start_lsn and the last log file sequence number end_lsn, back up the log file and write the log file backup status to the backup machine's control file 2. S38. Handle log files for backup failures.

[0005] Furthermore, in step S1, the distributed database backup environment check includes: S11. Backup network check, including: Ping all nodes that need to be backed up. If the ping fails, a message will be displayed indicating that the ping failed and you should check the network. Check the SSH functionality between all backup nodes and the backup server. If there are SSH problems and an SSH failure message appears, please check the SSH service. S12. Database check: Traverse all database nodes that need to be backed up and log in remotely from the backup machine. If the login fails, prompt "connection failed, please check the database service." S13. Backup machine disk space check: Iterate through the size of all databases that need to be backed up, sum up all the sizes, and check the backup machine disk to see if it is larger than the database data size. If the disk is insufficient, the message "Backup machine disk is insufficient, please check disk space" will be displayed. S14. Database configuration parameter check: Traverse all nodes that need to be backed up, check the database configuration parameters, and provide modification suggestions.

[0006] Furthermore, in step S2, the distributed database backup includes: S21. Start backup, record the starting log file sequence number start_lsn, and back up the data file to the backup machine; S22. Create a data consistency point and record the data consistency point log file sequence number stop_lsn; S23. Traverse the shards and transfer the log files from start_lsn to stop_lsn to the backup machine.

[0007] Furthermore, in step S34, the data files that failed to be backed up are re-backed up, including: Based on the contents of backup machine control file one and backup machine control file two, locate the data file that failed to be transferred and back it up again.

[0008] Furthermore, in step S35, the updated data file is backed up again, including: If a data page is modified, its corresponding log file sequence number (LSN) is recorded. The data pages in the database file and the backup file are compared. If the LSNs of the two are inconsistent, it means that the data pages are inconsistent. For inconsistent data pages, the data pages are copied from the database file to the data directory of the backup file, overwriting the original data pages.

[0009] Furthermore, in step S38, the log files of backup failures are processed, including: Based on the log file backup status in the backup machine's control file 2, locate the log files that failed to be backed up. For each failed log file, determine whether the log file is complete based on the last log file sequence number end_lsn. If it is incomplete, delete the log file; otherwise, re-trigger the log file transfer and back up the log files between start_lsn and end_lsn.

[0010] Furthermore, in step S4, distributed database recovery includes: S41. Copy the backup machine's data files and log files to the local machine; S42. Write the recovery.cnf configuration file. If it is a basic backup recovery, set the recovery target of the database node to the barrier created during backup in the recovery.cnf configuration file. If it is a PITR recovery, set the recovery target of the database node to the recovery time point in the recovery.cnf configuration file. S43. Restart the database and replay the log files based on the recovery target of the database node; S44. Recovery machine resource check.

[0011] Furthermore, in step S5, the distributed database recovery resuming from breakpoints includes: S51. Traverse the fragments and write the control file 1 of the recovery machine. The content to be written includes the ID of the data file to be recovered. S52. Write the control file 2 of the recovery machine, the content of which includes the data file recovery status; S53. Based on the contents of recovery machine control file one and recovery machine control file two, locate the data file that failed to recover, recover it again, and rewrite the recovery.cnf configuration file; S54. If the database has been successfully restored, the system will start normally; otherwise, it will restart.

[0012] Compared with existing technologies, the method for full backup and recovery of a distributed database with breakpoint resumption described in this invention has the following advantages: (1) The method for full backup and recovery of a distributed database described in this invention can save backup and recovery time: for backup / recovery failure scenarios, the failed data file is located by the control file, and only the failed or updated part is reprocessed, avoiding full reoperation and solving the problem of excessive time consumption of traditional full operation.

[0013] (2) The method for full backup and recovery of a distributed database described in this invention can improve the success rate of operations: Pre-check mechanism: Before backup, check the network, database status, disk space, and configuration parameters; before recovery, check resources such as disk, memory, and ports to avoid failures caused by environmental problems in advance. Exception handling closed loop: During backup, handle situations such as data consistency point failure and incomplete log files; during recovery, regenerate the configuration file and restart the exception node to form a complete exception handling process.

[0014] (3) The method for full backup and recovery of a distributed database described in this invention can ensure data consistency and integrity: during backup, the start / stop log sequence number is recorded and the integrity of data transmission is ensured by CRC check; during recovery, data consistency is ensured by log playback.

[0015] (4) The method for full backup and recovery of a distributed database described in this invention can adapt to the flexibility of distributed scenarios: it supports shard-level operations and adapts to the multi-node characteristics of distributed databases; at the same time, it supports parameter switching between new backup and failed backup resume to meet the needs of different scenarios.

[0016] (5) The method for full backup and recovery of a distributed database described in this invention can reduce the complexity of operation: by controlling the automatic recording of file status, the cost of manual intervention is reduced and the degree of automation of operation is improved. Attached Figure Description

[0017] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings: Figure 1 This is a schematic diagram illustrating the distributed database backup environment check as described in an embodiment of the present invention; Figure 2 This is a schematic diagram of distributed database backup as described in an embodiment of the present invention; Figure 3 This is a schematic diagram illustrating the breakpoint resumption of distributed database backup as described in an embodiment of the present invention; Figure 4 This is a schematic diagram of distributed database recovery as described in an embodiment of the present invention; Figure 5This is a schematic diagram illustrating the breakpoint resumption mechanism for distributed database recovery as described in an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0019] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first," "second," etc., may explicitly or implicitly include one or more of that feature. In the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art will understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0022] like Figures 1 to 5 As shown, a method for full backup and recovery of a distributed database with breakpoint resumption includes the following steps: S1. Distributed database backup environment check; S2, Distributed Database Backup; S3, Distributed Database Backup with Resume Functionality; S4, Distributed Database Recovery; S5, Distributed Database Recovery with Resume Functionality. Details are as follows: I. For example Figure 1 As shown, a distributed database backup environment check 1. Backup network check: 1.1 Ping all nodes that need to be backed up, for example, ping 10.0.1.23. If the ping fails, the message "10.0.1.23: ping failed, please check the network" will be displayed. 1.2 Check the SSH functionality between all backup nodes and the backup server. If there is a problem with SSH, the message "10.0.1.23: SSH failed, please check the SSH service" will be displayed.

[0023] 2. Database check: Iterate through all the database nodes that need to be backed up and log in remotely from the backup machine. If the login fails, display the following message (for example, for node DN1): "DN1: Connection failed, please check the database service".

[0024] 3. Backup machine disk space check: Iterate through the sizes of all databases that need to be backed up, sum up all the sizes, and check if the backup machine's disk size is larger than the database size. If the disk space is insufficient, display the message: "Backup machine disk is insufficient, please check disk space".

[0025] 4. Database configuration parameter check: Low timeout parameters can cause connection or data transmission timeouts. Iterate through all nodes requiring backup, check the parameters, and provide modification suggestions, such as: "DN1: sender_timeout equals 5s, it is recommended to change it to 300s."

[0026] II. Figure 2 As shown, distributed database backup 1. Start backup, record the start log file sequence number start_lsn for each shard, and back up the data files to the backup machine; 2. Create data consistency points and record the stop_lsn sequence number of the data consistency point log file for each shard; 3. Traverse the shards and transfer the log files of each shard from start_lsn to stop_lsn to the backup machine.

[0027] III. Figure 3 As shown, this demonstrates the ability to resume interrupted backups in a distributed database backup system. 1. Issue a backup command. The parameters of the command specify whether it is a new backup or a re-backup of a failed backup. If it is a re-backup of a failed backup, specify the ID of the failed backup. 2. Traverse the data fragments and write to the backup machine's control file 1. The content to be written includes the ID of the data file to be backed up and the start log file sequence number start_lsn; 3. Piecewise data transmission, write the control file two of the backup machine. The content written includes the data file transmission status. After the file is transmitted from the database to the backup machine, perform CRC check on the file. If the CRC check is successful, it is determined that the transmission is successful; otherwise, it is determined that the transmission fails. 4. Re-backup the data files that failed to be backed up: Based on the content of the control file one and the control file two of the backup machine, locate the data files that failed to be transmitted and re-back them up. 5. Re-backup the updated data files: If a data page is modified (such as INSERT / UPDATE / DELETE), record its corresponding log file sequence number LSN; compare the data pages of the database file and the backup file (usually 8KB). If the log file sequence numbers LSN of the two are inconsistent, it means that the data pages of the two are不一致; for the inconsistent data pages, copy the data pages from the database file to the data directory of the backup file to overwrite the original data pages. 6. Create a data consistency point, write the control file two of the backup machine, and the content written includes the data consistency point log file sequence number stop_lsn. 7. If the creation of the data consistency point fails, traverse the data files and determine whether the data files are updated, and re-back up the updated data files. 8. Backup the log files: Based on the starting log file sequence number start_lsn and the last log file sequence number end_lsn, backup the log files and write the log file backup status into the control file two of the backup machine. 9. Based on the log file backup status in the control file two of the backup machine, locate the log files that failed to be backed up. For the log files that failed to be backed up, based on the last log file sequence number end_lsn, determine whether the log files are complete (if the maximum lsn of the log file < end_lsn, the log file is incomplete). If it is incomplete, delete the log file; otherwise, re-trigger the log file transmission and backup the log files between start_lsn and end_lsn.

[0028] IV. As Figure 4 shown, distributed database recovery 1. Copy the data files and log files of the backup machine to the local, which generally takes a long time and is prone to failure. 2. Write the recovery.cnf configuration file. If it is a base backup recovery, set the recovery target of the database node in the recovery.cnf configuration file to the barrier created during backup. If it is a PITR recovery, set the recovery target of the database node in the recovery.cnf configuration file to the recovery time point. 3. Restart the database and replay the log files based on the recovery target of the database node. If it is a basic backup recovery, the log files have been stored locally because the recovery is to the barrier created during the backup, and the log replay generally will not fail. If the recovery is to PITR, the recovery needs to copy the log files from the log archive server. If there is a problem with the log archive server, it may fail. 4. Recovery machine resource check: 4.1 Disk Status: Compare whether the recovery machine's disk space is larger than the backup data. If the disk space is insufficient, a message will be displayed indicating that the corresponding node's disk space is insufficient. 4.2 Memory Status: Analyze the memory parameters in the backup data to determine if the recovery machine has sufficient memory to start the database; 4.3 Resource usage: Check whether the ports required for data recovery are occupied and whether there are enough remaining file handles.

[0029] V. For example Figure 5 As shown, this demonstrates the ability to resume interrupted data transfers in a distributed database. 1. Traverse the fragments and write the control file 1 of the recovery machine. The content written includes the ID of the data file to be recovered. 2. Perform data recovery and write the control file 2 of the recovery machine, including the data file recovery status; 3. Based on the contents of recovery machine control file one and recovery machine control file two, locate the data file that failed to recover, recover it again, and rewrite the recovery.cnf configuration file; 4. Traverse the shards. If the database has been successfully restored, start normally; otherwise, restart.

[0030] Advantages and beneficial effects of the present invention: Summary of the advantages of this invention: Save backup and restore time: For backup / restore failure scenarios, the failed data files are located through control files (such as backup machine control file one / two, recovery machine control file one / two), and only the failed or updated parts are reprocessed, avoiding full re-operation and solving the problem of excessive time consumption of traditional full operation.

[0031] Improve the success rate of operations: Pre-emptive checks: Before backup, check network, database status, disk space, and configuration parameters; before recovery, check disk, memory, ports, and other resources to proactively avoid failures caused by environmental issues.

[0032] Anomaly handling closed loop: During backup, it handles situations such as data consistency point failure and incomplete log files; during recovery, it regenerates configuration files and restarts abnormal nodes, forming a complete anomaly handling process.

[0033] Ensure data consistency and integrity: During backup, the start / stop log sequence number is recorded, and CRC check is used to ensure data transmission integrity; during recovery, log playback is used to ensure data consistency.

[0034] Flexibility in adapting to distributed scenarios: It supports shard-level operations (traversing shards to process data / logs), adapting to the multi-node characteristics of distributed databases; it also supports parameter switching for new backups and resumed from failed backups, meeting the needs of different scenarios.

[0035] Reduce operational complexity: By controlling the automatic recording of file status (data file ID, transmission status, log sequence number), the cost of manual intervention is reduced and the degree of operation automation is improved.

[0036] These advantages collectively solve the core problems of time-consuming and failure-prone full backup and recovery of distributed databases, improving the efficiency and reliability of backup and recovery in large-scale data scenarios.

[0037] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for full backup and resume breakpoint transmission of a distributed database, characterized in that: Includes the following steps: S1. Distributed database backup environment check, including: S11. Backup network check, including: Ping all nodes that need to be backed up. If the ping fails, a message will be displayed indicating that the ping failed and you should check the network. Check the SSH functionality between all backup nodes and the backup server. If there are SSH problems and an SSH failure message appears, please check the SSH service. S12. Database check: Traverse all database nodes that need to be backed up, log in remotely from the backup machine. If the login fails, prompt "connection failed, please check the database service". S13. Backup machine disk space check: Iterate through the size of all databases that need to be backed up, sum up all the sizes, and check the backup machine disk to see if it is larger than the database data size. If the disk is insufficient, the message "Backup machine disk is insufficient, please check disk space" will be displayed. S14. Database configuration parameter check: Traverse all nodes that need to be backed up, check the database configuration parameters, and provide modification suggestions; S2, distributed database backup, includes: S21. Start backup, record the starting log file sequence number start_lsn, and back up the data file to the backup machine; S22. Create a data consistency point and record the data consistency point log file sequence number stop_lsn for each shard; S23. Traverse the fragments and transfer the log files of each fragment from start_lsn to stop_lsn to the backup machine; S3, Distributed Database Backup with Resume Functionality; S4, Distributed Database Recovery; S5, Distributed Database Recovery: Resuming interrupted downloads; In step S3, the distributed database backup resume function includes: S31. Issue a backup command, with the command parameters specifying whether it is a new backup or a re-backup of a failed backup; S32. Traverse the data fragments and write the backup machine's control file 1. The content to be written includes the ID of the data file to be backed up and the start log file sequence number start_lsn. S33. Data is transmitted in segments, and the control file 2 of the backup machine is written, including the data file transmission status. S34. Re-back up the data files that failed to be backed up, including: Based on the contents of backup machine control file one and backup machine control file two, locate the data file that failed to be transferred and back it up again; S35. Re-back up the updated data files, including: If a data page is modified, its corresponding log file sequence number (LSN) is recorded. The data pages in the database file and the backup file are compared. If the LSNs of the two are inconsistent, it means that the data pages are inconsistent. For inconsistent data pages, the data pages are copied from the database file to the data directory of the backup file, overwriting the original data pages. S36. If the data consistency point is created successfully, write the control file 2 of the backup machine, including the data consistency point log file sequence number stop_lsn; if the data consistency point is created unsuccessfully, traverse the data files and determine whether the data files have been updated, and back up the updated data files again. S37. Based on the starting log file sequence number start_lsn and the last log file sequence number end_lsn, back up the log file and write the log file backup status to the backup machine's control file 2. S38. Handle log files for backup failures.

2. The method for full backup and recovery of a distributed database with breakpoint resumption according to claim 1, characterized in that: In step S38, the log files of backup failures are processed, including: Based on the log file backup status in the backup machine's control file 2, locate the log files that failed to be backed up. For each failed log file, determine whether the log file is complete based on the last log file sequence number end_lsn. If it is incomplete, delete the log file; otherwise, re-trigger the log file transfer and back up the log files between start_lsn and end_lsn.

3. The method for full backup and recovery of a distributed database with breakpoint resumption according to claim 1, characterized in that: In step S4, distributed database recovery includes: S41. Copy the backup machine's data files and log files to the local machine; S42. Write the recovery.cnf configuration file. If it is a basic backup recovery, set the recovery target of the database node to the barrier created during backup in the recovery.cnf configuration file. If it is a PITR recovery, set the recovery target of the database node to the recovery time point in the recovery.cnf configuration file. S43. Restart the database and replay the log files based on the recovery target of the database node; S44. Recovery machine resource check.

4. The method for full backup and recovery of a distributed database with breakpoint resumption according to claim 1, characterized in that: In step S5, the breakpoint resumption of distributed database recovery includes: S51. Traverse the fragments and write the control file 1 of the recovery machine. The content to be written includes the ID of the data file to be recovered. S52. Write the control file 2 of the recovery machine, the content of which includes the data file recovery status; S53. Based on the contents of recovery machine control file one and recovery machine control file two, locate the data file that failed to recover, recover it again, and rewrite the recovery.cnf configuration file; S54. If the database has been successfully restored, the system will start normally; otherwise, it will restart.

Images