Memory database recovery method and system based on snapshot backup

Abstract

The invention provides a memory database recovery method and system based on snapshot backup. The method comprises the steps of obtaining a database recovery instruction; according to the database recovery instruction, screening and determining a target snapshot from a plurality of snapshot data, the plurality of snapshot data being obtained by performing snapshot backup construction on a memory database at different time points; loading a plurality of corresponding table data to each node in the current memory database according to the target snapshot so as to obtain an initial memory database; according to a recovery time point corresponding to the target snapshot, extracting a plurality of thing records after the recovery time point from a command log; and sequentially submitting the plurality of transaction records to the initial memory database according to a time sequence, so that the initial memory database sequentially executes each transaction record to obtain a recovered memory database, and the data recovery efficiency and the data recovery accuracy of the memory database are improved.

Description

Technical Field

[0001] This application relates to the field of in-memory database technology, and in particular to a method and system for restoring in-memory databases based on snapshot backup. Background Technology

[0002] With the widespread application of big data, artificial intelligence, and the Internet of Things (IoT) technologies, enterprises' demand for real-time data processing has grown dramatically. In-memory databases, with their excellent low latency and high throughput capabilities, are gradually becoming a crucial infrastructure supporting core business systems. Compared to traditional disk-based databases, in-memory databases reside the entire dataset or frequently accessed datasets in memory, completely eliminating the performance bottleneck caused by disk I / O. They can achieve transaction processing capabilities of up to millions of transactions per second, fully meeting the demands of modern applications for extreme responsiveness. However, the volatility of memory also presents a fundamental challenge to data reliability—system crashes or power outages can lead to the instantaneous loss of in-memory data. Therefore, how to ensure data persistence and continuous service availability while enjoying the performance benefits of memory has become one of the core issues in the design and practice of in-memory databases.

[0003] Current mainstream data backup and recovery technologies mainly include logical backup, physical backup, incremental synchronization based on log replication, and snapshot technology. While logical backup is intuitive to operate, performing a full export on large-scale, high-concurrency clusters is time-consuming, and without applying global locks during the export, it's difficult to guarantee transaction consistency at the backup point in time. Physical backup, although it can directly mirror the memory state, is complex and significantly disrupts system operation; even slight errors can lead to state instability. While log change data capture methods theoretically possess incremental recovery capabilities, their practicality and efficiency are significantly limited by the high complexity of log parsing, format conversion, and semantic restoration. In contrast, snapshot technology periodically solidifies the complete state of the database at a specific moment, significantly reducing the amount of logs that need to be replayed during recovery, effectively lowering the recovery time target. However, traditional snapshot generation mostly relies on database-level shutdowns or table-level locking, which is an unacceptable interruption in enterprise scenarios requiring 24 / 7 uninterrupted service.

[0004] Despite this, existing backup and recovery technologies still expose many deep-seated problems when facing high-concurrency, distributed in-memory databases. Firstly, in terms of performance, the continuous high-intensity writing of command logs can easily become a system I / O bottleneck. Without effective snapshots, disaster recovery requires replaying massive amounts of historical logs from scratch, resulting in recovery times that can easily reach several hours, severely impacting business continuity. Secondly, the operation is highly complex. Setting and adjusting snapshot strategies heavily relies on human experience and lacks adaptability. Furthermore, the recovery process requires manually matching snapshots and log fragments, which is not only tedious but also prone to errors. Even more challenging are the issues of data consistency and integrity. Traditional snapshot mechanisms may create inconsistency windows due to brief execution intervals. In addition, backup files generally lack automated verification methods, making them highly susceptible to data corruption or partial loss in the event of storage media failure or network anomalies. Moreover, in geographically distributed architectures, frequent transmission of full snapshots or unprocessed raw logs can impose significant network and storage cost pressures. Summary of the Invention

[0005] To address the aforementioned technical issues, this application provides a method and system for recovering in-memory databases based on snapshot backups, thereby improving the efficiency and accuracy of data recovery for in-memory databases.

[0006] In a first aspect, embodiments of this application provide a method for restoring a memory database based on snapshot backup, comprising: Get database recovery commands; According to the database recovery instructions, the target snapshot is determined by filtering from several snapshot data. Based on the target snapshot, the corresponding table data is loaded into each node of the current in-memory database to obtain the initial in-memory database; Based on the recovery time point corresponding to the target snapshot, extract several transaction records after the recovery time point from the command log; The transaction records are submitted to the initial memory database in chronological order, so that the initial memory database executes each transaction record in turn to obtain the restored memory database. The snapshot data is obtained by backing up the memory database with snapshots at different points in time, and the command log is recorded and updated in real time according to the various operation instructions on the memory database.

[0007] This application provides a snapshot-based in-memory database recovery method. By generating snapshots of the in-memory database at multiple points in time and combining them with command log records, efficient database recovery is achieved, significantly improving the efficiency and accuracy of in-memory database data recovery. Compared with existing technologies, this embodiment does not require replaying the entire transaction log from scratch during database recovery. Instead, it first loads the most recent full snapshot to quickly rebuild the basic state, and then replays the incremental command logs after that point in time, thereby restoring the database to its precise state before the failure. This greatly shortens the recovery time and ensures the accuracy of data recovery. Furthermore, since a snapshot is a complete record of the state of the in-memory database at a specific moment, combining snapshots with command logs for database recovery avoids data corruption caused by accumulated log errors or inconsistencies over a long period. Finally, the command log only records calls to transactions, not the results. Recording only calls keeps the command log size to a minimum, reducing the impact of disk I / O on performance. Snapshots themselves do not require significant storage resources; therefore, this application effectively reduces the consumption of system resources.

[0008] Furthermore, the step of selecting and determining the target snapshot from several snapshot data according to the database recovery instruction includes: According to the database recovery instruction, each snapshot data is scanned sequentially in order of its closest to the current time. During the scanning process, for any current snapshot data, the integrity of the current snapshot data and its compatibility with the command log are verified to determine whether the current snapshot data is a usable snapshot; If the current snapshot data is a usable snapshot, the current snapshot data is sent to each node in the current memory database so that each node can verify the compatibility of the current snapshot data with its own data. When all nodes pass the verification of the current snapshot data, the current snapshot data is determined to be the target snapshot, and the scanning of each snapshot data is stopped.

[0009] This application provides a method for selecting target snapshots. It verifies the integrity and compatibility of snapshots one by one in reverse chronological order, with all nodes participating in the confirmation process to select the target snapshots. This mechanism significantly improves the accuracy and reliability of snapshot selection. Traditional snapshot recovery often relies on static checks of a single snapshot file, easily overlooking its applicability in dynamic environments. This method, through multi-layered verification, ensures that the selected snapshot is not only structurally intact but also accurately integrated with the current command log and various nodes in the database. This avoids recovery failures due to outdated snapshots or incompatibility between a node and the snapshot, ensuring consistency across nodes during the database recovery process and improving the efficiency and accuracy of data recovery for in-memory databases.

[0010] In one possible implementation, the step of loading corresponding table data into various nodes of the current in-memory database based on the target snapshot to obtain the initial in-memory database includes: Create a corresponding first synchronization barrier node for each node in the current memory database; By scanning the target snapshot, the corresponding recovery information is determined. The recovery information includes the system status of the in-memory database, data partition configuration, serial number of each table data, and setting parameters. Based on the recovery information, modify the system state and partition configuration of the current memory database to obtain the modified memory database; Based on the recovery information, the table data in the target snapshot is loaded into each node of the modified in-memory database. When any node completes the loading of table data, the corresponding first synchronization barrier node is deleted. The initial memory database is obtained when all first synchronization barrier nodes are deleted.

[0011] This application provides a snapshot recovery method. First, recovery information is read from the target snapshot, and then the current in-memory database is reset based on this information. The recovery information encompasses multi-dimensional metadata such as system status, data partition configuration, and table sequence numbers. This makes recovery no longer a simple data copy, but a comprehensive reconstruction with context awareness. By adjusting the partition configuration and system status of the current in-memory database and precisely aligning them with the snapshot data, this embodiment can maximize the isomorphism of the environment before and after recovery, preventing data recovery errors caused by configuration differences. Furthermore, this embodiment introduces a synchronization barrier node mechanism to ensure strong consistency in the recovery process, eliminating data corruption or logical conflicts caused by premature progress of some nodes before they are ready, thereby improving the stability and success rate of the entire recovery process.

[0012] Furthermore, the step of extracting several transaction records after the recovery time point from the command log based on the recovery time point corresponding to the target snapshot includes: The corresponding recovery time point is determined based on the transaction ID and timestamp in the target snapshot; Based on the recovery time point, extract the command log segment after the recovery time point from the command log to obtain several transaction records after the recovery time point.

[0013] This application provides a method for extracting transaction records from command logs. By utilizing the transaction ID and timestamp built into the snapshot, the recovery time point can be accurately located, and newly added operation records can be extracted from the log. This approach not only ensures the logical continuity of the recovery but also avoids the replay of redundant transactions. Compared to the fuzzy time interval matching commonly used in traditional methods, this embodiment has higher timing accuracy and controllability, effectively preventing data loss caused by time drift or log overwriting, and improving the data recovery efficiency and accuracy of in-memory databases.

[0014] Furthermore, the initial in-memory database sequentially executes each of the transaction records to obtain the restored in-memory database, including: Create a corresponding second synchronization barrier node for each node in the current memory database; The initial memory database executes each transaction record sequentially. For any current transaction record, the transaction record is sent to several corresponding execution nodes according to the data partition corresponding to the current transaction record, so that each execution node executes the current transaction record and updates the data in the initial memory database. When any node has completed executing all corresponding transaction records, the corresponding second synchronization barrier node is deleted. Once all secondary synchronization barrier nodes are deleted, the recovered in-memory database is obtained.

[0015] This application provides a method for replaying transaction records. It schedules corresponding nodes for processing based on data partitions, fully leveraging the parallel advantages of distributed systems to accelerate data updates. Simultaneously, this application introduces a second synchronization barrier node during the transaction record execution phase. This provides crucial eventual consistency guarantees for the recovery process of the distributed in-memory database. The entire recovery process is only officially terminated after the last node has completed all the transaction records it needed to process and deleted its corresponding barrier node. This mechanism ensures that in the recovered in-memory database, every node is at the exact same transaction point in time, thus providing extremely strong data consistency and improving the efficiency and accuracy of in-memory database data recovery.

[0016] In one possible implementation, the in-memory database recovery method further includes performing a snapshot backup of the in-memory database at the backup time point to obtain corresponding snapshot data, specifically: The global system state and data partition configuration of the in-memory database are captured at the backup time point and converted into corresponding metadata; An initial snapshot is generated based on the metadata, and a unique identifier is assigned to the initial snapshot; Based on the data distribution of the memory database, assign corresponding snapshot tasks to each node in the memory database; Each node extracts the corresponding table data from the memory database according to the corresponding snapshot task, serializes the table data, and writes it into the initial snapshot through the corresponding data stream to obtain the snapshot data.

[0017] This application provides a backup process based on snapshot data, emphasizing the unified capture of global system state and partition configuration, and solidifying it in the form of metadata. By assigning a unique identifier to each snapshot and distributing independent snapshot tasks to each node based on data distribution, this invention achieves efficient snapshot generation. Each node serializes the table data it is responsible for and writes it into a unified snapshot file. This not only improves the parallelism of backup operations but also lays a good data structure foundation for the subsequent recovery stage, improving the efficiency and accuracy of subsequent in-memory database data recovery.

[0018] Furthermore, the memory database recovery method also includes, when a truncation snapshot instruction is obtained, determining the corresponding baseline snapshot data according to the truncation snapshot instruction, and deleting several transaction records in the command log prior to the backup time point corresponding to the baseline snapshot data.

[0019] This application introduces a log cleanup mechanism after snapshot generation, which proactively deletes older transaction records earlier than the snapshot date after successful snapshot creation. This frees up storage resources, alleviating the problem of excessive log file growth over long periods, and simplifies the log retrieval path during subsequent recovery, indirectly improving overall operational efficiency. In enterprise practice, retaining complete logs for extended periods not only consumes significant space but can also introduce additional risks to recovery due to log file corruption. Therefore, this embodiment ensures recoverability while also considering the rational utilization of system resources, further reducing the storage cost pressure of backup data.

[0020] Secondly, embodiments of this application provide a memory database recovery system based on snapshot backup, including an acquisition module, a snapshot filtering module, a snapshot recovery module, a transaction record extraction module, and a transaction replay module; The acquisition module is used to acquire database recovery instructions; The snapshot filtering module is used to filter and determine the target snapshot from several snapshot data according to the database recovery instruction; The snapshot recovery module is used to load the corresponding table data into each node of the current in-memory database according to the target snapshot, thereby obtaining the initial in-memory database; The transaction record extraction module is used to extract several transaction records from the command log after the recovery time point corresponding to the target snapshot. The transaction replay module is used to submit the several transaction records to the initial memory database in chronological order, so that the initial memory database executes each of the transaction records in sequence to obtain the restored memory database. The snapshot data is obtained by backing up the memory database with snapshots at different points in time, and the command log is recorded and updated in real time according to the various operation instructions on the memory database.

[0021] Furthermore, the snapshot filtering module, based on the database recovery instruction, filters and determines the target snapshot from several snapshot data sets, including: According to the database recovery instruction, each snapshot data is scanned sequentially in order of its closest to the current time. During the scanning process, for any current snapshot data, the integrity of the current snapshot data and its compatibility with the command log are verified to determine whether the current snapshot data is a usable snapshot; If the current snapshot data is a usable snapshot, the current snapshot data is sent to each node in the current memory database so that each node can verify the compatibility of the current snapshot data with its own data. When all nodes pass the verification of the current snapshot data, the current snapshot data is determined to be the target snapshot, and the scanning of each snapshot data is stopped.

[0022] Furthermore, the in-memory database recovery system also includes a snapshot backup module, which is used to perform snapshot backup construction on the in-memory database at the backup time point to obtain corresponding snapshot data, including: The global system state and data partition configuration of the in-memory database are captured at the backup time point and converted into corresponding metadata; An initial snapshot is generated based on the metadata, and a unique identifier is assigned to the initial snapshot; Based on the data distribution of the memory database, assign corresponding snapshot tasks to each node in the memory database; Each node extracts the corresponding table data from the memory database according to the corresponding snapshot task, serializes the table data, and writes it into the initial snapshot through the corresponding data stream to obtain the snapshot data. Attached Figure Description

[0023] Figure 1 A flowchart illustrating a snapshot-backup-based in-memory database recovery method provided in this application embodiment; Figure 2 A schematic diagram of the architecture of the shuangzhaoDB database for applying the snapshot backup-based in-memory database recovery method provided in the embodiments of this application; Figure 3 A schematic diagram of command log recording in a snapshot-backed in-memory database recovery method provided in this application embodiment; Figure 4 This is a schematic diagram illustrating command log recovery based on a snapshot in a snapshot-based memory database recovery method provided in this application embodiment. Figure 5 A schematic diagram of a recovery process for a memory database recovery method based on snapshot backup, provided in an embodiment of this application; Figure 6 A schematic diagram illustrating the process of selecting target snapshots in a snapshot-based in-memory database recovery method provided in this application embodiment; Figure 7 A schematic diagram illustrating the snapshot recovery process in a snapshot-backup-based memory database recovery method provided in this application embodiment; Figure 8 A schematic diagram illustrating the process of transaction replay in a snapshot-backed in-memory database recovery method provided in this application embodiment; Figure 9 A schematic diagram illustrating the process of converting a database from a recovery state to a normal operating state in a snapshot-backup-based memory database recovery method provided in this application embodiment; Figure 10 A schematic diagram illustrating the process of creating a snapshot in a snapshot-based memory database recovery method provided in this application embodiment; Figure 11 A schematic diagram illustrating the architecture of an in-memory database recovery method based on snapshot backup, provided in an embodiment of this application, showing the automatic snapshot management of each component. Figure 12 A schematic diagram illustrating the process of truncating a snapshot in a snapshot-based memory database recovery method provided in this application embodiment; Figure 13 This is a schematic diagram of the structure of a memory database recovery system based on snapshot backup, provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0025] It should be noted that the step numbers in this document are only for the convenience of explaining the specific embodiments and are not intended to limit the order in which the steps are performed. In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of that feature.

[0026] Example 1: like Figure 1 As shown, Embodiment 1 provides a method for restoring a memory database based on snapshot backup, including steps S1-S5: Step S1: Obtain database recovery instructions; Step S2: Based on the database recovery instruction, select and determine the target snapshot from several snapshot data; Step S3: Based on the target snapshot, load the corresponding table data into each node of the current in-memory database to obtain the initial in-memory database; Step S4: Based on the recovery time point corresponding to the target snapshot, extract several transaction records after the recovery time point from the command log; Step S5: Submit the several transaction records to the initial memory database in chronological order, so that the initial memory database executes each transaction record in sequence to obtain the restored memory database; The snapshot data is obtained by backing up the memory database with snapshots at different points in time, and the command log is recorded and updated in real time according to the various operation instructions on the memory database.

[0027] This application provides a snapshot-based in-memory database recovery method. By generating snapshots of the in-memory database at multiple points in time and combining them with command log records, efficient database recovery is achieved, significantly improving the efficiency and accuracy of in-memory database data recovery. Compared with existing technologies, this embodiment does not require replaying the entire transaction log from scratch during database recovery. Instead, it first loads the most recent full snapshot to quickly rebuild the basic state, and then replays the incremental command logs after that point in time, thereby restoring the database to its precise state before the failure. This greatly shortens the recovery time and ensures the accuracy of data recovery. Furthermore, since a snapshot is a complete record of the state of the in-memory database at a specific moment, combining snapshots with command logs for database recovery avoids data corruption caused by accumulated log errors or inconsistencies over a long period. Finally, the command log only records calls to transactions, not the results. Recording only calls keeps the command log size to a minimum, reducing the impact of disk I / O on performance. Snapshots themselves do not require significant storage resources; therefore, this application effectively reduces the consumption of system resources.

[0028] In a preferred embodiment, the in-memory database recovery method provided in this application is mainly applied to distributed in-memory databases such as shuangzhaoDB. shuangzhaoDB's design philosophy is "memory priority + persistence guarantee." Its backup mechanism is not the traditional "periodic full backup + incremental log," but rather built around snapshots and command logs, aiming to achieve high-throughput, low-latency online backups and precise point-in-time recovery (PITR). Figure 2 As shown, shuangzhaoDB employs a shared-nothing, distributed memory architecture, storing data in memory rather than on disk. This allows it to avoid the I / O bottlenecks of traditional disk databases. The database is horizontally divided into several partitions, each bound to a fixed physical thread. All transactions accessing the same partition are sorted in the task queue of the corresponding thread and executed sequentially. This deterministic execution strategy ensures the atomicity and isolation of transactions, eliminating the need for traditional locking mechanisms.

[0029] The system supports two data distribution strategies: partitioned and replicated. Users can specify the partition column (e.g., region) for a table, and data is horizontally distributed across different partitions according to the value of that column; unpartitioned tables are globally replicated to all nodes. This flexible data distribution method allows the database to be optimized for different loads, while also providing a foundation for backup and recovery. Frequently used terms in this application's embodiments are explained below: Node: A physical or virtual server, which is the most basic hardware unit in a cluster.

[0030] Partition: A logical data shard, which is the basic unit for horizontal sharding of a database table.

[0031] Site: A Java thread. Each node in the cluster runs multiple execution sites (e.g., one per CPU core). Each partition on a node is assigned to one such site.

[0032] After applying the in-memory database recovery method provided in this application embodiment, shuangzhaoDB can automatically or manually generate data snapshots (full) periodically to provide a recovery baseline, and continuously record transaction command logs (incremental) to ensure zero data loss. During recovery, the most recent full snapshot is loaded first to quickly rebuild the basic state, and then the incremental command logs after that point in time are replayed, thereby restoring the database to its precise state before the failure. Therefore, this application embodiment actually includes two main processes: backup and recovery, and the specific process is as follows: Backup: Snapshots save the current state of the database before the cluster is shut down and restore the database after the cluster comes back online. Whether it's a physical failure or a client application error corrupting the logical content of the database, regular data backups provide a fallback in case of unexpected failures. shuangzhaoDB provides shell commands, system procedures, and automatic snapshot functionality to support these operations.

[0033] Recovery: Snapshots create a data protection mechanism by making a copy of the database at a specific point in time, but they cannot save transactions that occur between snapshots. For example... Figure 3 As shown, the command log provides a complete solution for the persistence and availability of the shuangzhaoDB database, retaining a record of each transaction (i.e., stored procedure) during execution. Regardless of the reason for server failure, such as... Figure 4 As shown, the database can restore the last snapshot and "replay" subsequent command logs to rebuild the entire database. The key to the command log is that it only records calls to transactions, not their results. Recording only calls keeps the command log size to a minimum, reducing the performance impact of disk I / O.

[0034] After applying the memory database recovery method proposed in the embodiments of this application, the data recovery process of shuangzhaoDB mainly includes the following two stages: 1) Recover data from the most recent valid snapshot.

[0035] 2) Replay the transaction from the command log to restore any changes that occurred after the snapshot was taken.

[0036] The recovery process includes the following key components that work together to rebuild the database state: 1) RestoreAgent: Coordinates the selection and recovery order of snapshots among distributed nodes to ensure data consistency and high availability of the cluster.

[0037] 2) SnapshotRestore: Performs specific operations for snapshot restoration, including scanning snapshot files, loading data into tables, handling uniqueness constraints, and coordinating the data distribution of distributed nodes.

[0038] 3) CommandLogReinitiator: Handles command log replay operations during fault recovery or cluster restart to ensure data consistency and transaction continuity.

[0039] 4) ZooKeeper: The core component for distributed coordination, providing a reliable distributed state management and coordination mechanism to ensure cluster consistency and high availability.

[0040] like Figure 5 As shown, the recovery process is coordinated across multiple nodes using ZooKeeperBarrier to ensure that all nodes in the cluster successfully complete each stage of the recovery before proceeding to the next stage. The detailed recovery process is as follows: 1. Create a recovery barrier to ensure that all nodes are synchronized during the recovery process and generate a recovery plan.

[0041] 2. Negotiate snapshot selection: Reach consensus through ZooKeeper to ensure that all nodes agree to use the same snapshot.

[0042] 3. Start the snapshot recovery process.

[0043] 4. Wait at the barrier for all nodes to complete the snapshot restoration.

[0044] 5. Replay the command log.

[0045] 6. Complete command log replay and restore normal operation.

[0046] Furthermore, in step S2, the step of selecting and determining the target snapshot from several snapshot data according to the database recovery instruction includes: According to the database recovery instruction, each snapshot data is scanned sequentially in order of its closest to the current time. During the scanning process, for any current snapshot data, the integrity of the current snapshot data and its compatibility with the command log are verified to determine whether the current snapshot data is a usable snapshot; If the current snapshot data is a usable snapshot, the current snapshot data is sent to each node in the current memory database so that each node can verify the compatibility of the current snapshot data with its own data. When all nodes pass the verification of the current snapshot data, the current snapshot data is determined to be the target snapshot, and the scanning of each snapshot data is stopped.

[0047] This application provides a method for selecting target snapshots. It verifies the integrity and compatibility of snapshots one by one in reverse chronological order, with all nodes participating in the confirmation process to select the target snapshots. This mechanism significantly improves the accuracy and reliability of snapshot selection. Traditional snapshot recovery often relies on static checks of a single snapshot file, easily overlooking its applicability in dynamic environments. This method, through multi-layered verification, ensures that the selected snapshot is not only structurally intact but also accurately integrated with the current command log and various nodes in the database. This avoids recovery failures due to outdated snapshots or incompatibility between a node and the snapshot, ensuring consistency across nodes during the database recovery process and improving the efficiency and accuracy of data recovery for in-memory databases.

[0048] In a preferred embodiment, such as Figure 6 As shown, the RestoreAgent component in the database is responsible for finding and selecting a suitable snapshot to restore from. The snapshot selection process is as follows: 1) Scan available snapshots 2) Verify the integrity of the snapshot. 3) Check compatibility with available command logs. 4) Negotiate with other nodes to reach an agreement on a single snapshot. The negotiation process is completed through the coordination mechanism of ZooKeeper. Available snapshots are discovered on each node, the latest and most complete snapshot is selected, and the final consistent and usable target snapshot is selected through the verification of each node.

[0049] In one possible implementation, in step S3, loading the corresponding table data into each node of the current in-memory database according to the target snapshot to obtain the initial in-memory database includes: Create a corresponding first synchronization barrier node for each node in the current memory database; By scanning the target snapshot, the corresponding recovery information is determined. The recovery information includes the system status of the in-memory database, data partition configuration, serial number of each table data, and setting parameters. Based on the recovery information, modify the system state and partition configuration of the current memory database to obtain the modified memory database; Based on the recovery information, the table data in the target snapshot is loaded into each node of the modified in-memory database. When any node completes the loading of table data, the corresponding first synchronization barrier node is deleted. The initial memory database is obtained when all first synchronization barrier nodes are deleted.

[0050] This application provides a snapshot recovery method. First, recovery information is read from the target snapshot, and then the current in-memory database is reset based on this information. The recovery information encompasses multi-dimensional metadata such as system status, data partition configuration, and table sequence numbers. This makes recovery no longer a simple data copy, but a comprehensive reconstruction with context awareness. By adjusting the partition configuration and system status of the current in-memory database and precisely aligning them with the snapshot data, this embodiment can maximize the isomorphism of the environment before and after recovery, preventing data recovery errors caused by configuration differences. Furthermore, this embodiment introduces a synchronization barrier node mechanism to ensure strong consistency in the recovery process, eliminating data corruption or logical conflicts caused by premature progress of some nodes before they are ready, thereby improving the stability and success rate of the entire recovery process.

[0051] In a preferred embodiment, such as Figure 7 As shown, after selecting a snapshot, the system begins data recovery from that snapshot. This process is handled by the SnapshotRestore system procedure, and the snapshot recovery process is as follows: 1) Read the snapshot summary to determine the table structure and settings. 2) Restore hash configuration to achieve correct data partitioning. 3) Restore export and DR (Database Replication) serial numbers 4) Load table data from snapshot file Furthermore, in step S4, extracting several transaction records from the command log after the recovery time point corresponding to the target snapshot includes: The corresponding recovery time point is determined based on the transaction ID and timestamp in the target snapshot; Based on the recovery time point, extract the command log segment after the recovery time point from the command log to obtain several transaction records after the recovery time point.

[0052] This application provides a method for extracting transaction records from command logs. By utilizing the transaction ID and timestamp built into the snapshot, the recovery time point can be accurately located, and newly added operation records can be extracted from the log. This approach not only ensures the logical continuity of the recovery but also avoids the replay of redundant transactions. Compared to the fuzzy time interval matching commonly used in traditional methods, this embodiment has higher timing accuracy and controllability, effectively preventing data loss caused by time drift or log overwriting, and improving the data recovery efficiency and accuracy of in-memory databases.

[0053] Furthermore, in step S5, the initial memory database sequentially executes each of the transaction records to obtain the restored memory database, including: Create a corresponding second synchronization barrier node for each node in the current memory database; The initial memory database executes each transaction record sequentially. For any current transaction record, the transaction record is sent to several corresponding execution nodes according to the data partition corresponding to the current transaction record, so that each execution node executes the current transaction record and updates the data in the initial memory database. When any node has completed executing all corresponding transaction records, the corresponding second synchronization barrier node is deleted. Once all secondary synchronization barrier nodes are deleted, the recovered in-memory database is obtained.

[0054] This application provides a method for replaying transaction records. It schedules corresponding nodes for processing based on data partitions, fully leveraging the parallel advantages of distributed systems to accelerate data updates. Simultaneously, this application introduces a second synchronization barrier node during the transaction record execution phase. This provides crucial eventual consistency guarantees for the recovery process of the distributed in-memory database. The entire recovery process is only officially terminated after the last node has completed all the transaction records it needed to process and deleted its corresponding barrier node. This mechanism ensures that in the recovered in-memory database, every node is at the exact same transaction point in time, thus providing extremely strong data consistency and improving the efficiency and accuracy of in-memory database data recovery.

[0055] In a preferred embodiment, such as Figure 8 As shown, after restoring from a snapshot, if command logging is enabled, the system will replay the transactions in the command log to restore any changes made after the snapshot. The command log replay process is as follows: 1) Generate replay plan based on snapshot transaction ID 2) Read command log segment from disk 3) Filter transactions, only replaying transactions that occurred after the snapshot. 4) Execute transactions in the original order. 5) The process is complete when all logs have been processed. Furthermore, such as Figure 9 As shown, after snapshot restoration and command log replay are complete, the system will complete the restoration process and enter normal operation. The process is as follows: 1. Each node signifies completion by deleting its barrier node in ZooKeeper, ensuring that all nodes reach the same state at the same time.

[0056] 2. The system waits for all nodes to recover completely. 3. ZooKeeper status has been cleared. 4. Restore normal database operations In one possible implementation, the in-memory database recovery method further includes performing a snapshot backup of the in-memory database at the backup time point to obtain corresponding snapshot data, specifically: The global system state and data partition configuration of the in-memory database are captured at the backup time point and converted into corresponding metadata; An initial snapshot is generated based on the metadata, and a unique identifier is assigned to the initial snapshot; Based on the data distribution of the memory database, assign corresponding snapshot tasks to each node in the memory database; Each node extracts the corresponding table data from the memory database according to the corresponding snapshot task, serializes the table data, and writes it into the initial snapshot through the corresponding data stream to obtain the snapshot data.

[0057] This application provides a backup process based on snapshot data, emphasizing the unified capture of global system state and partition configuration, and solidifying it in the form of metadata. By assigning a unique identifier to each snapshot and distributing independent snapshot tasks to each node based on data distribution, this invention achieves efficient snapshot generation. Each node serializes the table data it is responsible for and writes it into a unified snapshot file. This not only improves the parallelism of backup operations but also lays a good data structure foundation for the subsequent recovery stage, improving the efficiency and accuracy of subsequent in-memory database data recovery.

[0058] In a preferred embodiment, the shuangzhaoDB database using this method provides a mechanism for creating, managing, and restoring database backups. It supports point-in-time recovery, database cloning, and data archiving, creating consistent, transactionally correct database state snapshots while the database is running. These snapshots can be used for backup, recovery, or database migration. The system coordinates activities across all partitions to ensure snapshot consistency while minimizing performance impact. Snapshots are coordinated across all database partitions to ensure transaction consistency, integrate with other subsystems such as export and DR to maintain external consistency, and provide flexible options for different backup and recovery scenarios.

[0059] shuangzhaoDB supports several types of snapshots, including: 1. Manual snapshots: initiated by the user through the @SnapshotSave system procedure. 2. Automatic snapshots: scheduled and managed by SnapshotDaemon. 3. Truncated snapshots: used to truncate the command log after a recovery point is established.

[0060] The snapshot system consists of several key components that work together to manage the snapshot lifecycle: 1) SnapshotDaemon: Orchestrates automatic snapshots and manages snapshot requests across the cluster. A Leader is selected via ZooKeeper to coordinate cluster-wide snapshots.

[0061] 2) SnapshotSaveAPI: Provides APIs for initializing snapshots, handling initial setup, and coordinating with the site.

[0062] 3) SnapshotSiteProcessor: Manages snapshot processing for each individual site, responsible for extracting and serializing data.

[0063] 4) TableStreamer: Streams table data from the storage engine when creating a snapshot.

[0064] 5) SnapshotUtil: Provides practical functions for snapshot operations, including file operations and metadata management.

[0065] 6) SnapshotRestore: Handles snapshot restoration during recovery.

[0066] Specifically, such as Figure 10 As shown, the process of creating a snapshot includes the following steps: 1. Initialization: Initialize a snapshot using the @SnapshotSave system procedure or SnapshotDaemon.

[0067] 2. Coordination: The `snapshotsaveapi.startsnapshot()` method handles initial settings, including: 2.1 Generate a snapshot write plan.

[0068] 2.2. Capture exported and system states to maintain consistency.

[0069] 2.3 Use barriers to synchronize sites in SnapshotSiteProcessor to ensure that all sites start taking snapshots at the same logical point in time.

[0070] 2.4. Create a unique nonce (identifier) ​​for the snapshot.

[0071] 2.5 Assign snapshot tasks to each site and coordinate task execution.

[0072] 3. Execution: Each site handles its own snapshot portion: 3.1 SnapshotSiteProcessor starts a snapshot task for each table and allocates the buffer required to write snapshot data.

[0073] 3.2 TableStreamer extracts data from partitioned tables, serializes the table data, writes it to the stream in batches, and catches exceptions. If an exception occurs during serialization, it will cause the data file to fail to write. In this case, you can stop snapshot generation and record the exception result, or regenerate the snapshot.

[0074] 3.3 SnapshotDataTarget writes data to the snapshot file and releases resources (such as file handles, buffers, etc.). Specifically, during snapshot execution, shuangzhaoDB automatically collects current system status information, including ongoing transactions, the latest transaction ID, and system timestamps. The Hashinator is an algorithm used to determine which partition a data item should be assigned to. When executing a snapshot, the system also obtains the current Hashinator configuration, which typically includes boundary conditions or rules for each partition.

[0075] 3.4 Save metadata and hashinator configuration. This data is stored in the file directory of each partition snapshot backup.

[0076] 4. Completion: When all sites have completed their snapshots: 4.1 The snapshot file has been successfully saved, and its type and structure are as follows: a. Table file (.vpt): Contains serialized table data. • Naming pattern: [nonce]-[table_name]-[host_id]_[site_id].vpt • Each site creates a file for each table's partition. b. Digest file (.digest): • Includes snapshot metadata, such as transaction ID, timestamp, table list, and other metadata. • Naming pattern: [nonce]-host_[host_id].digest • Verify the integrity and consistency of backup files to prevent the recovery of damaged or tampered files. c. Hashinator configuration file (.hash): • Includes partition information • Naming pattern: [nonce]-host_[host_id].hash • Used to restore the partition scheme during recovery, ensuring data distribution consistency. d. Finished file (.finished): • Indicates snapshot complete • Naming pattern: [nonce].finished • A marker indicating the completion of the snapshot process, used by monitoring tools to determine the snapshot status. 4.2 Create a completed entry in ZooKeeper.

[0077] 4.3 Notify SnapshotCompletionMonitor (monitor snapshot completion status).

[0078] Furthermore, embodiments of this application support automatic snapshot generation, such as... Figure 11 As shown, the SnapshotDaemon component manages automatic snapshots and acts as the central coordinator for snapshot operations across the cluster. Its main responsibilities are: 1) Automatic snapshot scheduling is based on configuration. Snapshot generation is automatically triggered using two strategies: time interval and number of transactions. The specific code is as follows: <snapshot enabled="true" frequency="24h" retain="24" prefix="AUTOSNAP" path=" / snapshots" / > 2) Manage snapshot retention policies.

[0079] 3) Coordinate truncation snapshots of command logs.

[0080] 4) Handle snapshot requests initiated by users.

[0081] 5) Select a leader for cluster-wide snapshot coordination.

[0082] Regarding cluster-wide coordination, SnapshotDaemon uses ZooKeeper to elect a leader, which ensures that snapshot operations are coordinated throughout the cluster, even in the event of node failures. The specific steps are as follows: 1) Each SnapshotDaemon instance will attempt to create a temporary node in ZooKeeper.

[0083] 2) The instance that successfully creates the node becomes the leader.

[0084] 3) The leader monitors ZooKeeper's snapshot requests and manages scheduled snapshots.

[0085] 4) If the leader fails, another instance will take over when the ephemeral node disappears.

[0086] Furthermore, the memory database recovery method also includes, when a truncation snapshot instruction is obtained, determining the corresponding baseline snapshot data according to the truncation snapshot instruction, and deleting several transaction records in the command log prior to the backup time point corresponding to the baseline snapshot data.

[0087] This application introduces a log cleanup mechanism after snapshot generation, which proactively deletes older transaction records earlier than the snapshot date after successful snapshot creation. This frees up storage resources, alleviating the problem of excessive log file growth over long periods, and simplifies the log retrieval path during subsequent recovery, indirectly improving overall operational efficiency. In enterprise practice, retaining complete logs for extended periods not only consumes significant space but can also introduce additional risks to recovery due to log file corruption. Therefore, this embodiment ensures recoverability while also considering the rational utilization of system resources, further reducing the storage cost pressure of backup data.

[0088] In a preferred embodiment, a truncated snapshot feature is introduced. A truncated snapshot is a special type of snapshot that ensures the command log can be kept at a manageable size while still maintaining the ability to restore the database to any point in time. To enable command log truncation, see [example code]. Figure 12 As shown, the specific process for truncating a snapshot is as follows: 1) Command logs are truncated and snapshotted via ZooKeeper requests.

[0089] 2) SnapshotDaemonleader detects requests and initiates a snapshot.

[0090] 3) Once the snapshot is successfully completed, the command log can be truncated.

[0091] 4) Delete command log entries that are now being overwritten by snapshots.

[0092] Example 2: like Figure 13 As shown, Embodiment 2 provides a memory database recovery system based on snapshot backup, including an acquisition module 10, a snapshot filtering module 20, a snapshot recovery module 30, a transaction record extraction module 40, and a transaction replay module 50; The acquisition module 10 is used to acquire database recovery instructions; The snapshot filtering module 20 is used to filter and determine the target snapshot from a number of snapshot data according to the database recovery instruction; The snapshot recovery module 30 is used to load the corresponding table data into each node of the current in-memory database according to the target snapshot, thereby obtaining the initial in-memory database; The transaction record extraction module 40 is used to extract several transaction records after the recovery time point from the command log according to the recovery time point corresponding to the target snapshot; The transaction replay module 50 is used to submit the plurality of transaction records to the initial memory database in chronological order, so that the initial memory database executes each of the transaction records in sequence to obtain the restored memory database; The snapshot data is obtained by backing up the memory database with snapshots at different points in time, and the command log is recorded and updated in real time according to the various operation instructions on the memory database.

[0093] Furthermore, the snapshot filtering module 20, according to the database recovery instruction, filters and determines the target snapshot from several snapshot data, including: According to the database recovery instruction, each snapshot data is scanned sequentially in order of its closest to the current time. During the scanning process, for any current snapshot data, the integrity of the current snapshot data and its compatibility with the command log are verified to determine whether the current snapshot data is a usable snapshot; If the current snapshot data is a usable snapshot, the current snapshot data is sent to each node in the current memory database so that each node can verify the compatibility of the current snapshot data with its own data. When all nodes pass the verification of the current snapshot data, the current snapshot data is determined to be the target snapshot, and the scanning of each snapshot data is stopped.

[0094] In one possible implementation, the snapshot recovery module 30 loads corresponding table data into various nodes of the current in-memory database based on the target snapshot, thereby obtaining the initial in-memory database, including: Create a corresponding first synchronization barrier node for each node in the current memory database; By scanning the target snapshot, the corresponding recovery information is determined. The recovery information includes the system status of the in-memory database, data partition configuration, serial number of each table data, and setting parameters. Based on the recovery information, modify the system state and partition configuration of the current memory database to obtain the modified memory database; Based on the recovery information, the table data in the target snapshot is loaded into each node of the modified in-memory database. When any node completes the loading of table data, the corresponding first synchronization barrier node is deleted. The initial memory database is obtained when all first synchronization barrier nodes are deleted.

[0095] Furthermore, the transaction record extraction module 40 extracts several transaction records from the command log after the recovery time point corresponding to the target snapshot, including: The corresponding recovery time point is determined based on the transaction ID and timestamp in the target snapshot; Based on the recovery time point, extract the command log segment after the recovery time point from the command log to obtain several transaction records after the recovery time point.

[0096] In one possible implementation, the initial in-memory database is executed sequentially for each of the transaction records to obtain a restored in-memory database, including: Create a corresponding second synchronization barrier node for each node in the current memory database; The initial memory database executes each transaction record sequentially. For any current transaction record, the transaction record is sent to several corresponding execution nodes according to the data partition corresponding to the current transaction record, so that each execution node executes the current transaction record and updates the data in the initial memory database. When any node has completed executing all corresponding transaction records, the corresponding second synchronization barrier node is deleted. Once all secondary synchronization barrier nodes are deleted, the recovered in-memory database is obtained.

[0097] Furthermore, the in-memory database recovery system also includes a snapshot backup module, which is used to perform snapshot backup construction on the in-memory database at the backup time point to obtain corresponding snapshot data, including: The global system state and data partition configuration of the in-memory database are captured at the backup time point and converted into corresponding metadata; An initial snapshot is generated based on the metadata, and a unique identifier is assigned to the initial snapshot; Based on the data distribution of the memory database, assign corresponding snapshot tasks to each node in the memory database; Each node extracts the corresponding table data from the memory database according to the corresponding snapshot task, serializes the table data, and writes it into the initial snapshot through the corresponding data stream to obtain the snapshot data.

[0098] Furthermore, the memory database recovery system also includes a snapshot truncation module, which is used to determine the corresponding baseline snapshot data according to the snapshot truncation command when a snapshot truncation command is obtained, and delete several transaction records in the command log before the backup time point corresponding to the baseline snapshot data.

[0099] This application provides a snapshot-based in-memory database recovery system. By generating snapshots of the in-memory database at multiple points in time and combining them with command log records, it achieves efficient database recovery, significantly improving the efficiency and accuracy of in-memory database data recovery. Compared with existing technologies, this embodiment does not require replaying the entire transaction log from scratch during database recovery. Instead, it first loads the most recent full snapshot to quickly rebuild the basic state, and then replays the incremental command logs after that point in time, thereby restoring the database to its precise state before the failure. This greatly shortens the recovery time and ensures the accuracy of data recovery. Furthermore, since a snapshot is a complete record of the state of the in-memory database at a specific moment, combining snapshots with command logs for database recovery avoids data corruption caused by accumulated log errors or inconsistencies over a long period. Finally, the command log only records calls to transactions, not the results. Recording only calls keeps the command log size to a minimum, reducing the impact of disk I / O on performance. Snapshots themselves do not require significant storage resources; therefore, this application effectively reduces the consumption of system resources.

[0100] For a more detailed explanation of the working principle and procedures of this embodiment, please refer to the relevant description in Embodiment 1.

[0101] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A method for restoring a memory database based on a snapshot backup, characterized in that, include: Get database recovery commands; According to the database recovery instructions, the target snapshot is determined by filtering from several snapshot data. Based on the target snapshot, the corresponding table data is loaded into each node of the current in-memory database to obtain the initial in-memory database; Based on the recovery time point corresponding to the target snapshot, extract several transaction records after the recovery time point from the command log; The transaction records are submitted to the initial memory database in chronological order, so that the initial memory database executes each transaction record sequentially to obtain the restored memory database. The snapshot data is obtained by backing up the in-memory database at different points in time, and the command log is recorded and updated in real time according to the various operation instructions on the in-memory database.

2. The method of claim 1, wherein, The step of selecting and determining the target snapshot from several snapshot data according to the database recovery instruction includes: According to the database recovery instruction, each snapshot data is scanned sequentially in order of its closest to the current time. During the scanning process, for any current snapshot data, the integrity of the current snapshot data and its compatibility with the command log are verified to determine whether the current snapshot data is a usable snapshot; If the current snapshot data is a usable snapshot, the current snapshot data is sent to each node in the current memory database so that each node can verify the compatibility of the current snapshot data with its own data. When all nodes pass the verification of the current snapshot data, the current snapshot data is determined to be the target snapshot, and the scanning of each snapshot data is stopped.

3. The method of claim 1, wherein the snapshot-based backup of the in-memory database is performed by a backup server. The step of loading corresponding table data into various nodes of the current in-memory database based on the target snapshot to obtain the initial in-memory database includes: Create a corresponding first synchronization barrier node for each node in the current memory database; By scanning the target snapshot, the corresponding recovery information is determined. The recovery information includes the system status of the in-memory database, data partition configuration, serial number of each table data, and setting parameters. Based on the recovery information, modify the system state and partition configuration of the current memory database to obtain the modified memory database; Based on the recovery information, the table data in the target snapshot is loaded into each node of the modified in-memory database. When any node completes the loading of table data, the corresponding first synchronization barrier node is deleted. The initial memory database is obtained when all first synchronization barrier nodes are deleted.

4. The method of claim 1, wherein the snapshot-based backup of the in-memory database is performed by a backup server. The step of extracting several transaction records from the command log after the recovery time point corresponding to the target snapshot includes: The corresponding recovery time point is determined based on the transaction ID and timestamp in the target snapshot; Based on the recovery time point, extract the command log segment after the recovery time point from the command log to obtain several transaction records after the recovery time point.

5. The method of claim 1, wherein, The initial in-memory database is executed sequentially for each transaction record to obtain the restored in-memory database, including: Create a corresponding second synchronization barrier node for each node in the current memory database; The initial memory database executes each transaction record sequentially. For any current transaction record, the transaction record is sent to several corresponding execution nodes according to the data partition corresponding to the current transaction record, so that each execution node executes the current transaction record, thereby updating the data in the initial memory database. When any node has completed executing all corresponding transaction records, the corresponding second synchronization barrier node is deleted. Once all secondary synchronization barrier nodes are deleted, the recovered in-memory database is obtained.

6. The method of claim 1-5, wherein, The memory database recovery method further includes performing a snapshot backup of the memory database at the backup time point to obtain the corresponding snapshot data, specifically: The global system state and data partition configuration of the in-memory database are captured at the backup time point and converted into corresponding metadata; An initial snapshot is generated based on the metadata, and a unique identifier is assigned to the initial snapshot; Based on the data distribution of the memory database, assign corresponding snapshot tasks to each node in the memory database; Each node extracts the corresponding table data from the memory database according to the corresponding snapshot task, serializes the table data, and writes it into the initial snapshot through the corresponding data stream to obtain the snapshot data.

7. The method of claim 6, wherein the snapshot backup-based memory database recovery method is characterized by, The memory database recovery method further includes, when a truncation snapshot instruction is obtained, determining the corresponding baseline snapshot data according to the truncation snapshot instruction, and deleting several transaction records in the command log before the backup time point corresponding to the baseline snapshot data.

8. A snapshot backup based in-memory database recovery system, characterized in that, It includes an acquisition module, a snapshot filtering module, a snapshot recovery module, a transaction record extraction module, and a transaction replay module; The acquisition module is used to acquire database recovery instructions; The snapshot filtering module is used to filter and determine the target snapshot from several snapshot data according to the database recovery instruction; The snapshot recovery module is used to load the corresponding table data into each node of the current in-memory database according to the target snapshot, thereby obtaining the initial in-memory database; The transaction record extraction module is used to extract several transaction records from the command log after the recovery time point corresponding to the target snapshot. The transaction replay module is used to submit the several transaction records to the initial memory database in chronological order, so that the initial memory database executes each of the transaction records in sequence to obtain the restored memory database. The snapshot data is obtained by backing up the memory database with snapshots at different points in time, and the command log is recorded and updated in real time according to the various operation instructions on the memory database.

9. The snapshot backup based in-memory database recovery system of claim 8, wherein, The snapshot filtering module, based on the database recovery instruction, filters and determines the target snapshot from several snapshot data sets, including: According to the database recovery instruction, each snapshot data is scanned sequentially in order of its closest to the current time. During the scanning process, for any current snapshot data, the integrity of the current snapshot data and its compatibility with the command log are verified to determine whether the current snapshot data is a usable snapshot; If the current snapshot data is a usable snapshot, the current snapshot data is sent to each node in the current memory database so that each node can verify the compatibility of the current snapshot data with its own data. When all nodes pass the verification of the current snapshot data, the current snapshot data is determined to be the target snapshot, and the scanning of each snapshot data is stopped.

10. A memory database recovery system based on snapshot backup as described in claim 8 or 9, characterized in that, The in-memory database recovery system also includes a snapshot backup module, which is used to perform snapshot backups of the in-memory database at backup points to obtain corresponding snapshot data, including: The global system state and data partition configuration of the in-memory database are captured at the backup time point and converted into corresponding metadata; An initial snapshot is generated based on the metadata, and a unique identifier is assigned to the initial snapshot; Based on the data distribution of the memory database, assign corresponding snapshot tasks to each node in the memory database; Each node extracts the corresponding table data from the memory database according to the corresponding snapshot task, serializes the table data, and writes it into the initial snapshot through the corresponding data stream to obtain the snapshot data.

Images