A method and system for database backup in a duty log system

Abstract

This invention discloses a method and system for database backup in a value log system. The method includes the following steps: generating a checkpoint and obtaining the LSN corresponding to the current checkpoint in the value log system; obtaining an LSM tree snapshot, collecting each subtree of the LSM index tree, and then generating a linked list of SSTables to be backed up; in the LSM index tree, one SSTable corresponds to one persistent index file; one SSTable is retrieved from the linked list of SSTables to be backed up, the data of the index file corresponding to the SSTable is backed up, and stored in the backup file, and so on, retrieving the next SSTable for backup until the linked list of SSTables to be backed up is empty; obtaining the value log file of the value log system, copying it from the beginning to the LSN offset corresponding to the checkpoint, and storing it in the backup file, thus completing the backup of the value log system; the system includes: a backup device, a checkpoint module, an SSTable subtree collection module, an LSM subtree backup module, and a value log backup module. This invention can ensure the integrity of user data and restore user data using backup.

Description

Technical Field

[0001] This invention relates to the field of database technology, and in particular to a method and system for database backup in a value log system. Background Technology

[0002] Database backups must ensure no data loss or minimize data loss. Database backups are mainly divided into cold backups and hot backups. A hot backup is performed while the database is running, having no impact on ongoing database operations. A cold backup is performed when the database is down; this type of backup is the simplest, generally only requiring the copying of the relevant physical database files. Currently, systems are required to provide 24 / 7 online service, and backups are usually performed online; therefore, most backups are hot backups, and the following descriptions refer to hot backups.

[0003] Traditional database systems store several types of data: user data and indexes of user data. Indexes are usually organized using B-trees, skiplists, or heaps. Both user data and index data are stored in data files. REDO logs record data changes and are mainly used for exception recovery, consistency, and persistence. UNDO logs related to transactions are used to roll back essential data for transactions and ensure the consistency of database data.

[0004] The database checkpoint function targets a specific point in time (usually corresponding to the database's log sequence number LSN). The checkpoint ensures that all data changes prior to this LSN value have been flushed to disk.

[0005] Traditional data backup first performs a checkpoint operation (flushing updated data pages from the buffer to disk), obtains the LSN corresponding to the checkpoint, and records it as Start_LSN; secondly, it backs up the database's user data files (including user data and index data); next, it backs up the database's UNDO log files; after all the previous files have been backed up, it performs another checkpoint, obtains the LSN corresponding to the checkpoint, and records it as End_LSN; finally, it backs up the REDO logs from Begin_LSN to End_LSN from the REDO logs.

[0006] In traditional databases, user data and index data are stored together. To ensure data consistency, REDO and UNDO logs are introduced. During backup, checkpoints require flushing a large number of updated data pages to disk, which takes a considerable amount of time. Depending on the implementation of different checkpoints, this may cause user services to be interrupted for a period of time. In addition, backups not only need to back up index data and user data, but also REDO and UNDO logs, increasing the space consumption of backups and prolonging the backup time. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a method and system for database backup in a value log system, which can ensure the integrity of user data and restore user data using backup.

[0008] To address the aforementioned technical problems, this invention provides a method for database backup in a value log system, comprising the following steps:

[0009] (1) Generate a checkpoint and obtain the LSN corresponding to the current checkpoint in the value log system;

[0010] (2) To obtain an LSM tree snapshot, first traverse the LSM index tree and collect each subtree, and then generate a linked list of SSTables to be backed up.

[0011] (3) In the LSM index tree, one SSTable corresponds to one persistent index file. Take an SSTable from the SSTable chain to be backed up, read the data of the index file corresponding to the SSTable, package it and store it in the backup file; and so on, take the next SSTable to back up until the SSTable chain to be backed up is empty.

[0012] (4) Obtain the value log file of the value log system, read it from the beginning to the LSN offset corresponding to the checkpoint, pack the value log data, store it in the backup file, and complete the backup of the value log system.

[0013] Preferably, in step (1), LSN is the incremental log sequence number written to the database log file, or the offset address at the end of the value log file in the value log system. In this case, the offset address plays the same role as the LSN in a traditional database.

[0014] Preferably, in step (2), each subtree of the LSM index tree includes all index files generated by all read-only SSTables (Store String Tables) frozen in memory during checkpoint execution, and all index files that already exist on disk before the checkpoint is generated.

[0015] Correspondingly, a system for a database backup method in a value log system includes: a backup device, a checkpoint module, an SSTable subtree collection module, an LSM subtree backup module, and a value log backup module. The value log system backup device receives a backup request and notifies the checkpoint module, which generates a checkpoint operation and obtains the corresponding LSN. The value log system backup device calls the SSTable subtree collection module, which generates an LSM tree snapshot and collects the SSTables to be backed up and stores them in a linked list. The LSM subtree backup module retrieves an SSTable from the linked list of SSTables to be backed up, reads the data from the corresponding index file of the SSTable, and packages it into a backup file. After the backup is complete, it retrieves the next SSTable from the linked list to be backed up until the SSTable linked list is empty, completing the backup of the index LSM snapshot. The value log backup module reads data from offset 0 of the value log file, packages it, and stores it in a backup file until the offset address corresponding to the checkpoint LSN is reached, completing the backup of the value log system.

[0016] The beneficial effects of the present invention are: (1) ensuring the data security of the value log system, and the data of the value log system can be restored by using backup; (2) compared with traditional databases, eliminating the backup of REDO logs and UNDO logs saves storage space and shortens the backup time; (3) the value log system can avoid the business system lag caused by checkpoints during online backup of traditional databases and shorten the backup time. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0018] Figure 2 This is a schematic diagram of the system structure of the present invention. Detailed Implementation

[0019] like Figure 1 As shown, a method for database backup in a value log system includes the following steps:

[0020] S101, the value log backup worker thread receives a backup request, executes a checkpoint, and retrieves the checkpoint LSN (which can be the current offset address of the value log file);

[0021] S102, the value log backup worker thread, obtains a snapshot of the current index LSM tree, collects each subtree SSTable, and stores it in the SSTable linked list to be backed up;

[0022] S103, the value log backup worker thread retrieves a subtree SSTable from the SSTable list to be backed up. Each SSTable actually corresponds to a data file (the data organization can be B-tree, B+ tree, or heap, etc.). Only the data in the data file needs to be backed up. Then, the next subtree SSTable is retrieved and backed up, until the SSTable list to be backed up is empty;

[0023] S104, after backing up the subtree of the LSM, the value log backup worker thread starts backing up the value log file (starting from offset address 0 and continuing to the offset address corresponding to the LSN in S101), thus completing the backup work of the value log system.

[0024] like Figure 2 As shown, a system corresponding to a database backup method in a value log system includes: a backup device, a checkpoint module, an SSTable subtree collection module, an LSM subtree backup module, and a value log backup module. The value log system backup device receives a backup request and notifies the checkpoint module, which performs a checkpoint operation and obtains the corresponding LSN. The value log system backup device calls the SSTable subtree collection module, which generates an LSM tree snapshot and collects the SSTables to be backed up and stores them in a linked list. The LSM subtree backup module retrieves SSTables from the linked list to be backed up and backs them up. After the backup is complete, it retrieves the next SSTable from the linked list until the SSTable linked list is empty, completing the backup of the index LSM snapshot. The value log backup module backs up from offset 0 of the value log file until the offset address corresponding to the LSN, completing the backup of the value log system.

[0025] In this invention, the backup of the value log ensures the data security of the value log and the data can be restored using the backup; the backup of the value log system can save storage space, eliminating the need to back up redo and undo logs; the value log system can avoid business system lag caused by checkpoints during traditional online database backups, thus reducing backup time.

Claims

1. A method for database backup in a value log system, characterized in that, Includes the following steps: (1) Generate a checkpoint and obtain the LSN corresponding to the current checkpoint in the value log system; the LSN is the incrementing log sequence number of the database's write log file, or the offset address of the end of the value log file in the value log system. (2) To obtain an LSM tree snapshot, first traverse the LSM index tree and collect each subtree, and then generate a linked list of SSTables to be backed up; each subtree of the LSM index tree includes all index files generated by all read-only SSTables frozen in memory during checkpoint execution, and all index files that already exist on disk before the checkpoint is generated. (3) In the LSM index tree, one SSTable corresponds to one persistent index file. Take an SSTable from the SSTable list to be backed up, read the data of the index file corresponding to the SSTable, package it and store it in the backup file; Continue this process to back up the next SSTable, until the SSTable list to be backed up is empty; (4) Obtain the value log file of the value log system, read it from the beginning to the LSN offset corresponding to the checkpoint, pack the value log data, store it in the backup file, and complete the backup of the value log system.

2. A system corresponding to the database backup method in the value log system as described in claim 1, characterized in that, include: Backup device, checkpoint module, SSTable subtree collection module, LSM subtree backup module and value log backup module; When the backup device receives a backup request, it notifies the checkpoint module, which then generates a checkpoint operation and obtains the corresponding LSN. The backup device calls the SSTable subtree collection module, which generates an LSM tree snapshot and collects the SSTables to be backed up and stores them in a linked list. The LSM subtree backup module takes an SSTable from the linked list of SSTables to be backed up, reads the data of the index file corresponding to the SSTable, and packages it into a backup file. After the backup is complete, it takes the next SSTable from the linked list of SSTables to be backed up and backs it up until the linked list of SSTables is empty, thus completing the backup of the index LSM snapshot. The value log backup module reads data from offset 0 of the value log file, packages and stores the backup file, and continues until the offset address corresponding to the checkpoint LSN is reached, thus completing the backup of the value log system.

Images