A method and apparatus for online backup and recovery of databases with fault tolerance

Abstract

This invention proposes a fault-tolerant method and apparatus for online database backup and recovery, belonging to the field of database backup technology. The method includes: configuring the database to directly affect metadata and data files when data changes occur; notifying the database before backup begins, waiting for all database transactions prior to the notification time to complete; performing write-time replication for new database transactions after the notification time; the online database backup adopts a mode that tolerates backup failures of some tables while simultaneously backing up the write-time replicated content; after a successful backup, stopping write-time replication for new database transactions, recording backup information, including information on successfully backed-up tables and information on tables that failed to be backed up; and restoring data using the backup data according to the information on the successfully backed-up tables. This invention provides fault-tolerant backup and recovery capabilities in scenarios with large data volumes, reduces costs, and provides more detailed backup information, allowing for the recovery and specification of successfully backed-up tables.

Description

Technical Field

[0001] This invention belongs to the field of database backup technology, and in particular relates to a database online backup and recovery fault-tolerant method and apparatus. Background Technology

[0002] Currently, in the field of database backup technology, in order to ensure data read and write security, a multi-version mechanism is often used, which creates a new version of the data when the data changes. In terms of data backup, conventional database backup and recovery methods assume that if a table backup fails, the entire task will be terminated directly. That is, if a table backup fails in a backup task, the entire backup task will fail.

[0003] The aforementioned multi-version mechanism and the method of causing overall backup failure due to a single table error have certain drawbacks and are not suitable for the rapidly evolving data backup requirements. For example, in the current common big data scenarios, the amount of data and time consumed in each online database backup are relatively large. If the failure of a single table leads to the conclusion that the entire task has failed, the previous backup operations will be wasted, consuming computing resources without any results, wasting time, and increasing the cost of database backup. Summary of the Invention

[0004] This invention proposes a database online backup and recovery fault-tolerant method and apparatus, which provides backup and recovery fault tolerance capabilities in scenarios with large data volumes, reduces costs, and provides more detailed backup information, allowing recovery and specification of tables that have been successfully backed up.

[0005] To achieve the above objectives, the technical solution of the present invention is implemented as follows:

[0006] A database online backup and recovery fault-tolerant method includes:

[0007] S1. Configure the database to directly affect metadata and data files when data changes occur;

[0008] S2. Before the online database backup begins, the database is notified that the backup is about to start, and waits for all database operations before the notification time to be completed; new database operations after the notification time are copied on write.

[0009] S3. The online database backup adopts a mode that tolerates backup failures of some tables, and copies the content when backing up the data.

[0010] S4. After a successful backup, stop the write-time replication of the new database business and record the backup information, including the table information of the successfully backed-up tables and the table information of the failed backups.

[0011] S5. When using backup data to restore data, restore according to the table information of the successfully backed-up data;

[0012] The modes in step S3 that tolerate partial table backup failures include:

[0013] If a table cannot be backed up normally during the backup process, skip that table and continue with the backup task.

[0014] After all other tables have been backed up, the tables that cannot be backed up normally are retried for backup. The write-time replication ensures the data consistency of the tables during the retry.

[0015] If the attempt fails after multiple retries, the table is confirmed as a failed backup.

[0016] If even one table is successfully backed up, the backup task is considered successful and can be used for data recovery.

[0017] Furthermore, the copy-on-write method described in step S2 includes copying the metadata and the corresponding data tail blocks of the data files that change during the execution of new database services to a specified location for backup during online database backup.

[0018] Furthermore, this also includes:

[0019] If the backup task is terminated due to an abnormal reason or by the user at any time before the backup task is completed normally, the backup task will be considered a backup failure and no related backup information will be recorded.

[0020] Furthermore, the recorded backup information in step S4 includes:

[0021] The records are stored in an info file, which includes the backup cycle, backup point, backup time, and backup table summary. The backup table summary includes the number of tables that were successfully backed up, the number of tables that failed to be backed up, and the total number of tables backed up. It also records specific information about the tables that were successfully and failed to be backed up at each backup point.

[0022] Furthermore, the info file contains detailed information about the success and failure of backups at each backup point, accessible through a table linked to the backup points.

[0023] Furthermore, in the info file, the backup log file is queried through backup point links, and the reasons for backup failure are analyzed through the backup log content.

[0024] Furthermore, in step S5, once all the backed-up tables have been successfully restored, the data recovery is considered successful.

[0025] In another aspect, the present invention also proposes a database online backup and recovery fault-tolerant device, comprising:

[0026] Database settings module: Configures the database so that changes to the data directly affect the metadata and data files;

[0027] Backup notification module: Before the online database backup begins, the module notifies the database that the backup is about to start and waits for all database operations performed before the notification time to be completed; new database operations performed after the notification time are copied on write.

[0028] Backup module: The online database backup adopts a mode that tolerates backup failures of some tables, and copies content when backing up;

[0029] Backup record module: After a successful backup, stop the write-time replication of the new database business, and record backup information, including table information that was successfully backed up and table information that failed to be backed up;

[0030] Data recovery module: When performing data recovery using backup data, the recovery is performed according to the table information of the successfully backed-up data;

[0031] The backup module includes:

[0032] Skip a table: If a table cannot be backed up normally during the backup process, skip that table and continue with the backup task.

[0033] Retry Unit: After all other tables have been backed up, the table that could not be backed up normally will be retried for backup. The write-time copy ensures that the table has the latest data at the time of backup.

[0034] Failure confirmation unit: If the table still fails after multiple retries, it is confirmed that the table is a failed backup table;

[0035] Task success unit: As long as one table is successfully backed up, the backup task is considered successful and can be used for data recovery.

[0036] Furthermore, the write-time copying mentioned in the backup notification module includes copying the metadata and data tail blocks corresponding to the data files that change during the execution of new database services to a specified location for backup during online database backup.

[0037] Furthermore, this also includes:

[0038] Task Failure Unit: If a backup task is terminated due to an abnormal reason or by the user before it is completed normally, the backup task is considered to have failed and no related backup information is recorded.

[0039] Furthermore, the backup information recorded in the backup recording module includes:

[0040] The records are stored in an info file, which includes the backup cycle, backup point, backup time, and backup table summary. The backup table summary includes the number of tables that were successfully backed up, the number of tables that failed to be backed up, and the total number of tables backed up. It also records specific information about the tables that were successfully and failed to be backed up at each backup point.

[0041] Furthermore, the info file contains detailed information about the success and failure of backups at each backup point, accessible through a table linked to the backup points.

[0042] Furthermore, in the info file, the backup log file is queried through backup point links, and the reasons for backup failure are analyzed through the backup log content.

[0043] Furthermore, in the data recovery module, data recovery is considered successful once all backed-up tables have been successfully restored.

[0044] Compared with the prior art, the beneficial effects of the present invention include:

[0045] 1. This invention provides fault tolerance for online database backup and recovery in scenarios with large amounts of data, reducing costs and preventing the failure of the entire backup task due to the failure of a single table backup, thus avoiding the waste of corresponding time and resource costs.

[0046] 2. The solution of this invention allows for querying more detailed backup information at backup points. Since successful and failed backups of tables are recorded, the backup point information reveals which tables failed. The reasons for the failures can then be understood by querying the backup logs. If a table is corrupted (e.g., some files are missing), it can be promptly dumped to prevent subsequent data loss.

[0047] 3. The solution of the present invention can restore and specify the table that was successfully restored during backup and recovery. Attached Figure Description

[0048] Figure 1 This is a flowchart illustrating Embodiment 1 of the present invention;

[0049] Figure 2 This is a schematic diagram of the structure of Embodiment 2 of the present invention. Detailed Implementation

[0050] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other.

[0051] To make the objectives and features of the present invention more apparent and understandable, further explanation will be provided below in conjunction with the accompanying drawings and specific embodiments.

[0052] Example 1

[0053] The database online backup and recovery fault-tolerant method proposed in this embodiment, such as Figure 1 As shown, it includes:

[0054] S1. Configure the database to directly affect metadata and data files when data changes.

[0055] In existing technologies, various databases (such as MySQL, SQL Server, Oracle, etc.) generally employ a multi-versioning mechanism when data changes. Taking MySQL's InnoDB storage engine as an example, multi-versioning means that whenever data changes or is modified, InnoDB does not directly overwrite the original data. Instead, it maintains an undo log to record the changes and creates a new version of the data. Programs can choose the version of the data at any given time for easy rollback. This mechanism helps resolve read / write conflicts and improves concurrency performance. However, because multiple versions of the same data exist, over time, this can consume a significant amount of disk space, sometimes even reaching 90% disk usage, resulting in wasted disk space.

[0056] In this embodiment, a multi-version mechanism is not used. The database is set to directly affect the metadata and data files when data changes. A flag is used inside the database to prevent read-write conflicts. The flag must be checked before the database business is executed.

[0057] Because it does not use a multi-version mechanism, it does not require additional storage space, resulting in less disk pressure and lower operating costs.

[0058] S2. Before starting the online database backup, notify the database that the backup is about to begin.

[0059] Simultaneously with the notification, a flag to prevent read-write conflicts is enabled in the database. Once enabled, the system waits for all database transactions prior to the notification time to complete before initiating the backup task. New database transactions after the notification time will be performed using copy-on-write.

[0060] The database services mentioned in this step refer to business scenarios involving database operations. Taking relational databases as an example, database services are also known as SQL services, which use SQL statements to perform operations such as data querying, updating, and maintenance.

[0061] During online database backup, to ensure data consistency, data operations, especially modification operations, should be avoided as much as possible. Therefore, in this step, before starting the online database backup, the database is notified that a backup is about to begin. This notification is done through SQL interaction; for example, for a database backup, the command `backup database db_name start` is sent; similarly, for a table backup, the command `backup table tb_name start` is sent.

[0062] Simultaneously with the notification, enable the database's read / write conflict prevention flag. Once the flag is enabled, the database waits for all database transactions prior to the notification time to complete before initiating the backup task. This setting enables online backup functionality without using a multi-versioning mechanism.

[0063] To avoid impacting business operations, database services will not be stopped due to online database backup. Therefore, even after the database is notified that a backup is about to begin, new database services will still occur. In this step, for new database services after the notification time, if the flag bit is enabled, copy-on-write will be executed.

[0064] The copy-on-write technique refers to copying the changed metadata and the corresponding data tail blocks of the data file to a specified location when a new database business is executed. When the database performs online backup, it backs up the data at the specified location. When the data is restored, the data at the specified location is restored, thereby ensuring data consistency.

[0065] S3, the online database backup adopts a mode that tolerates backup failures of some tables, and copies content during write backup.

[0066] In this embodiment, the online database backup does not back up storage engine files, configuration files, and business logic. The backup content includes tables, stored procedures, user-defined functions, views, etc.

[0067] When performing online database backups, the backup task can be configured to tolerate backup failures for some tables. For example, if some tables have missing files, the table may not be able to be backed up normally. If a table cannot be backed up normally, the table will be recorded and then skipped, and the backup task will continue.

[0068] After all tables have been backed up, the tables that could not be backed up normally are retried. Since online backups copy content on write, this copy-on-write mechanism can ensure the data consistency of the tables during the retries.

[0069] If the backup fails to complete after three retries, the table backup has failed.

[0070] For backup results, as long as one table in the backup content is successfully backed up, the backup task is considered successful and can be used for restoration. Restoration is limited to the successfully backed-up content, such as successfully backed-up tables, stored procedures, user-defined functions, views, etc.

[0071] If a backup task is terminated at any time before its normal completion due to an abnormal reason or by the user, the backup task is considered a failure, no backup information is recorded, and any remaining data is cleared. For example, when backing up table data to a remote server, if the connection to the remote server is lost or the remote server's storage space is insufficient, the backup task is considered a failure.

[0072] S4. After a successful backup, stop the write-time replication of the new database business and record the backup information, including information on tables that were successfully backed up and information on tables that failed to be backed up.

[0073] Backup information is recorded in an info file, which includes backup cycle, backup point, backup time, and backup table summary. The backup table summary includes the number of tables that were successfully backed up, the number of tables that failed to be backed up, and the total number of tables backed up. It also records specific information about the tables that were successfully and failed to be backed up at each backup point.

[0074] In a specific embodiment, the backup information is recorded as a gcluster_info file, which is divided into A and B versions for distinguishing between read and write versions. These versions are named gcluster_info.A and gcluster_info.B, respectively. The A and B version switching method facilitates rollback operations and avoids errors in the gcluster_info file caused by rollback operations.

[0075] The summary information of the gcluster_info file record table backup can be recorded as a file record table backup summary information table, as shown in the following example:

[0076] Table 1. Summary of File Record Table Backup Information

[0077]

[0078] In the above summary information, cycle represents the backup cycle, point represents the backup point, time represents the backup time, and info(success / failure / total) represents the summary of backup tables, including the number of tables that were successfully backed up, the number of tables that failed to be backed up, and the total number of tables backed up.

[0079] The gcluster_info file records detailed information about the tables that were successfully and failed at each backup point. This information can be queried through links provided by each backup point, which links to the specific tables showing successful and failed backups for that particular backup point.

[0080] The gcluster_info file can also be used to query backup log files via backup point links, allowing analysis of the backup log content to determine the reasons for table backup failures. For example, information on successfully backed-up tables can be used to identify which tables failed backups, and the backup log content can be analyzed to determine why the backups failed. If table corruption is found, it can be detected promptly.

[0081] S5. When using backup data to restore data, restore according to the table information of the successfully backed-up data.

[0082] During data recovery, the recovery task is performed based on the number of tables successfully backed up at each backup point. A recovery is considered successful only if all tables are successfully restored. For example, when performing data recovery using the three backup points mentioned above, backup point 0 in backup cycle 0 successfully restored 38 tables; backup point 1 in backup cycle 0 successfully restored 10 tables; and backup point 2 in backup cycle 0 successfully restored 62 tables.

[0083] During recovery, you can specify a backup point or recover a specific table (this table must be one of the tables that were successfully backed up). That is, this embodiment supports recovering a portion of the tables from the backup. Because distributed databases have high availability features, it is not always necessary to perform a full database recovery; recovering a portion of the tables is sufficient. For example, if 10 tables were successfully backed up, recovering this backup point can successfully recover all 10 tables, or it can recover any one of those 10 tables.

[0084] The method used in this embodiment provides fault tolerance for online database backup and recovery in scenarios with large amounts of data, reduces costs, and prevents the failure of the entire backup task due to the failure of a single table backup, thus avoiding the waste of corresponding time and resource costs.

[0085] The method used in this embodiment allows for querying more detailed backup information at backup points. Because successful and failed backups are recorded, the backup point information reveals which tables failed. The reasons for the failures can then be understood by querying the backup logs. If a table is corrupted (e.g., some files are missing), it can be promptly dumped to prevent subsequent data loss.

[0086] The method used in this embodiment enables the restoration of tables that have been successfully restored during backup and recovery.

[0087] Example 2

[0088] This embodiment proposes a database online backup and recovery fault-tolerant device, including:

[0089] Database settings module: Configures the database so that changes to the data directly affect the metadata and data files;

[0090] Backup notification module: Before the online database backup begins, the module notifies the database that the backup is about to start and waits for all database operations performed before the notification time to be completed; new database operations performed after the notification time are copied on write.

[0091] Backup module: The online database backup adopts a mode that tolerates backup failures of some tables, and copies content when backing up;

[0092] Backup record module: After a successful backup, stop the write-time replication of the new database business, and record backup information, including table information that was successfully backed up and table information that failed to be backed up;

[0093] Data recovery module: When performing data recovery using backup data, the recovery is performed according to the table information of the successfully backed-up data.

[0094] The write-time copying in the backup notification module includes copying the metadata and data tail blocks corresponding to the data files that change during the execution of new database services to a specified location for backup during online database backup.

[0095] The backup module includes:

[0096] Skip a table: If a table cannot be backed up normally during the backup process, skip that table and continue with the backup task.

[0097] Retry Unit: After all other tables have been backed up, the table that could not be backed up normally will be retried for backup. The write-time copy ensures that the table has the latest data at the time of backup.

[0098] Failure confirmation unit: If the table still fails after multiple retries, it is confirmed that the table is a failed backup table;

[0099] Task success unit: As long as one table is successfully backed up, the backup task is considered successful and can be used for data recovery.

[0100] Also includes:

[0101] Task Failure Unit: If a backup task is terminated due to an abnormal reason or by the user before it is completed normally, the backup task is considered to have failed and no related backup information is recorded.

[0102] The backup information recorded in the backup recording module includes:

[0103] The records are stored in an info file, which includes the backup cycle, backup point, backup time, and backup table summary. The backup table summary includes the number of tables that were successfully backed up, the number of tables that failed to be backed up, and the total number of tables backed up. It also records specific information about the tables that were successfully and failed to be backed up at each backup point.

[0104] The info file contains detailed information about the success and failure of backups at each backup point, accessible through a table linked to the backup point.

[0105] The info file allows users to query backup log files via backup point links and analyze the backup log content to determine the reasons for backup failures.

[0106] In the data recovery module, data recovery is considered successful once all backed-up tables have been successfully restored.

[0107] The database online backup and recovery fault-tolerant device proposed in this embodiment can realize the database online backup and recovery fault-tolerant method proposed in Embodiment 1, and has the same technical effect as Embodiment 1.

[0108] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A database on-line backup recovery fault-tolerant method, characterized in that, include: S1. Configure the database to directly affect metadata and data files when data changes occur; A flag is used internally in the database to prevent read / write conflicts; the flag is checked before database business operations are executed. S2. Before the online database backup begins, notify the database that the backup is about to begin and wait for all database services executed before the notification time to be completed; at the same time as the notification, enable the flag bit. New database transactions following the notification time will be replicated on write. S3. The online database backup adopts a mode that tolerates backup failures of some tables, and copies the content when backing up the data. S4. After a successful backup, stop the write-time replication of the new database business and record the backup information, including the table information of the successfully backed-up tables and the table information of the failed backups. S5. When using backup data to restore data, restore according to the table information of the successfully backed-up data; The modes in step S3 that tolerate partial table backup failures include: If a table cannot be backed up normally during the backup process, skip that table and continue with the backup task. After all other tables have been backed up, the tables that cannot be backed up normally are retried for backup. The write-time replication ensures the data consistency of the tables during the retry. If the attempt fails after multiple retries, the table is confirmed as a failed backup. If even one table is successfully backed up, the backup task is considered successful and can be used for data recovery.

2. The database online backup and recovery fault-tolerant method according to claim 1, characterized in that, The write-time copying mentioned in step S2 includes copying the metadata and data tail blocks corresponding to the data files that change during the execution of new database business to a specified location for backup during online database backup.

3. The database online backup and recovery fault-tolerant method according to claim 1, characterized in that, Also includes: If the backup task is terminated due to an abnormal reason or by the user at any time before the backup task is completed normally, the backup task will be considered a backup failure and no related backup information will be recorded.

4. The database online backup and recovery fault-tolerant method according to claim 1, characterized in that, The backup information recorded in step S4 includes: The records are stored in an info file, which includes the backup cycle, backup point, backup time, and backup table summary. The backup table summary includes the number of tables that were successfully backed up, the number of tables that failed to be backed up, and the total number of tables backed up. It also records specific information about the tables that were successfully and failed to be backed up at each backup point.

5. The database online backup and recovery fault-tolerant method according to claim 4, characterized in that, The info file contains detailed information about the success and failure of backups at each backup point, accessible through a table linked to the backup point.

6. The database online backup and recovery fault-tolerant method according to claim 4, characterized in that, The info file allows users to query backup log files via backup point links and analyze the backup log content to determine the reasons for backup failures.

7. The database online backup and recovery fault-tolerant method according to claim 1, characterized in that, In step S5, once all backed-up tables have been successfully restored, the data recovery is considered successful.

8. A database online backup and recovery fault-tolerant device, characterized in that, include: Database settings module: Configures the database so that changes to the data directly affect the metadata and data files; A flag is used internally in the database to prevent read / write conflicts; the flag is checked before database business operations are executed. Backup notification module: Before the online database backup begins, the module notifies the database that the backup is about to start and waits for all database operations to be completed before the notification time; the module also enables the flag bit at the same time as the notification. New database transactions following the notification time will be replicated on write. Backup module: The online database backup adopts a mode that tolerates backup failures of some tables, and copies content when backing up; Backup record module: After a successful backup, stop the write-time replication of the new database business, and record backup information, including table information that was successfully backed up and table information that failed to be backed up; Data recovery module: When performing data recovery using backup data, the recovery is performed according to the table information of the successfully backed-up data; The backup module includes: Skip a table: If a table cannot be backed up normally during the backup process, skip that table and continue with the backup task. Retry Unit: After all other tables have been backed up, the table that could not be backed up normally will be retried for backup. The write-time copy ensures that the table has the latest data at the time of backup. Failure confirmation unit: If the table still fails after multiple retries, it is confirmed that the table is a failed backup table; Task success unit: As long as one table is successfully backed up, the backup task is considered successful and can be used for data recovery.

Images