Data recovery method and apparatus

Abstract

The present disclosure relates to the technical field of big data, and particularly relates to a data recovery method and device. The method comprises the following steps: in the case that the database system is in the fault recovery and the starting instruction for the database system is received, a redo log file closest to the current time in the redo log file in the file system is loaded; according to the scanning sequence corresponding to at least one redo log file in the redo log file and the redo log file, each redo log file corresponding page in the at least one redo log file is read from the disk to the memory, and the modified page is written back to the disk according to each redo log file in the memory; in the case that the modified page and the undo log meet the recovery requirements, the data dictionary metadata is recovered to obtain the recovered data dictionary metadata; in the case that the modified page and the undo log meet the recovery requirements, the record is recovered to obtain the recovered record. The present disclosure can improve the data recovery speed.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, and in particular to a data recovery method and apparatus. Background Technology

[0002] Within the entire hardware and software stack of a database system, both hardware and software can fail. In a single-machine database system, system failures require significant processing time (due to long restart times or the need for manual intervention), resulting in prolonged database service unavailability. High-availability database systems, through redundant database nodes and automatic disaster recovery mechanisms, can achieve automatic handling and rapid recovery from database failures. A high-availability database system can, for example, consist of 1 primary and N backup nodes (N>=1). The primary and backup nodes in a high-availability database system have identical hardware configurations (including CPU, memory, and disk), i.e., a peer-to-peer configuration. The primary node provides read and write services, while the backup nodes maintain data synchronization with the primary node. When the primary node fails, the management system selects one backup node and promotes it to primary status within a short time (industry experience suggests 30 seconds), restoring it to provide services. To ensure rapid fault recovery, multiple backup hosts are typically used, but this increases the hardware cost of the database system and the fault recovery time. Summary of the Invention

[0003] This disclosure provides a data recovery method and apparatus, which can improve data recovery speed. The technical solution of this disclosure is as follows:

[0004] According to a first aspect of the present disclosure, a fault recovery method is provided, comprising:

[0005] In the event of database system failure recovery and receipt of a startup command for the database system, the physical redo log file in the file system that is closest to the current time is loaded, wherein the database system includes a database node;

[0006] Based on the scan order corresponding to at least one physical redo log file in the physical log redo log files and the physical redo log file closest to the current time, read the data page corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory, modify the data page in the memory according to each physical redo log file, and rewrite the modified data page back to the disk;

[0007] If the modified data page and the undo log meet the recovery requirements, the data dictionary metadata is recovered to obtain the recovered data dictionary metadata, wherein the undo log includes the records in the modified data page;

[0008] If the modified data page and undo log meet the recovery requirements, the record is recovered, and the recovered record is obtained.

[0009] According to some embodiments, the step of reading the data pages corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory based on the scan order corresponding to at least one physical redo log file in the physical redo log file and the physical redo log file closest to the current time includes:

[0010] Obtain the scan start point corresponding to the physical log redo log file, wherein the scan start point is the checkpoint corresponding to the physical log redo log file closest to the current time;

[0011] According to the scan points and scan order, the physical log redo log file is scanned. During the scan, if an unscanned physical log redo log file is found, the log sequence number (LSN) is updated. The scan order is from the furthest to the nearest time from the current time.

[0012] If the scan operation meets the scan completion condition, it is determined that at least one physical log redo log file has been obtained, wherein the at least one physical log redo log file is used to recover data pages;

[0013] Read the data pages corresponding to each physical redo log file from the disk into memory.

[0014] According to some embodiments, reading the data pages corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory includes:

[0015] Based on the scan start point corresponding to the physical log redo log file, it is determined that there is modification information of data page in the record of the at least one physical log redo log file, and during the scanning process, if an unscanned physical log redo log file is found, the log sequence number LSN is updated.

[0016] Based on the modified information, the data page corresponding to the modified information is read from the data file on the disk and loaded into memory.

[0017] According to some embodiments, the method further includes:

[0018] The at least one physical redo log file is obtained using a multi-threaded or multi-process approach.

[0019] or,

[0020] Pre-read data pages from at least one physical log redo file;

[0021] or;

[0022] The at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file.

[0023] According to some embodiments, the step of restoring the data dictionary metadata and obtaining the restored data dictionary metadata includes:

[0024] Based on the recovery starting point, data pages, including data dictionary metadata, are read from the disk into memory;

[0025] Use the records in the physical log redo log file to obtain the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point;

[0026] Based on the physical log redo log file and the change information, the data dictionary metadata is restored. During the data dictionary metadata restoration process, the data dictionary metadata is verified. Once the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

[0027] According to some embodiments, the step of recovering the record and obtaining the recovered record includes:

[0028] Read the data page containing the target record from the disk into memory;

[0029] Traverse the undo log to find the transactions that were in the target state when the failure occurred;

[0030] Based on the undo log records corresponding to each transaction in the target state, undo the changes made by each transaction in the target state to the target record.

[0031] During the target record recovery process, the target record is verified, and upon completion of the verification, the recovered record is obtained.

[0032] According to some embodiments, the method further includes:

[0033] Receive a Structured Query Language (SQL) statement sent by the client, and read the target data page from the file system according to the SQL statement;

[0034] The target data page is placed into the data page cache pool, and if the target data page exists in the data page cache pool, the redo log-data page refresh operation and the linked list update operation are executed.

[0035] Determine the record to be read corresponding to the target data page, and determine the target record based on the processing status of the suspended transaction.

[0036] According to some embodiments, the method further includes:

[0037] Read the redo log file that is closest to the current time from the redo log files in the file system;

[0038] The redo log file closest to the current time is parsed to obtain the page number of the data page of the redo log file closest to the current time;

[0039] Store the data page number, and store the redo log file corresponding to the same data page number in the file system into a singly linked list corresponding to the same data page number;

[0040] The target data page is read from the data page file of the file system, and then each unidirectional linked list is queried according to the target data page pg to obtain and use the redo log corresponding to the target data page pg.

[0041] According to a second aspect of the present disclosure, a data recovery apparatus is provided, comprising:

[0042] The log loading unit is used to load the physical redo log file in the file system that is closest to the current time when the database system is recovering from a failure and a startup command for the database system is received. The database system includes a database node.

[0043] The data page modification unit is used to read the data page corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory according to the scan order corresponding to each physical redo log file in the physical redo log file and the physical redo log file closest to the current time, modify the data page in the memory according to each physical redo log file, and write the modified data page back to the disk;

[0044] The data recovery unit is used to recover the data dictionary metadata and obtain the recovered data dictionary metadata when the modified data page and the undo log meet the recovery requirements. The undo log includes the records in the modified data page.

[0045] The data recovery unit is also used to recover records and obtain the recovered records if the modified data page and undo log meet the recovery requirements.

[0046] According to a third aspect of the present disclosure, a network device is provided, comprising:

[0047] processor;

[0048] Memory used to store the processor's executable instructions;

[0049] The processor is configured to execute the instructions to implement the data recovery method described in any one of the preceding aspects.

[0050] According to a fourth aspect of the present disclosure, a storage medium is provided that, when instructions in the storage medium are executed by a processor of a network device, enables the network device to perform the data recovery method described in any one of the preceding aspects.

[0051] According to a fifth aspect of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the method described in any one of the preceding aspects.

[0052] The technical solutions provided by the embodiments of this disclosure have at least the following beneficial effects:

[0053] In some or related embodiments, upon failure recovery of the database system and receipt of a startup command for the database system, the system loads the physical redo log file in the file system that is most recent in time. The database system includes a database node. Based on the scan order of at least one physical redo log file and the most recent physical redo log file, data pages corresponding to each physical redo log file are read from the disk into memory. The data pages are then modified in memory according to the physical redo log files, and the modified data pages are written back to the disk. If the modified data pages and undo logs meet the recovery requirements, the data dictionary metadata is restored to obtain the restored data dictionary metadata. The undo logs include records from the modified data pages. Finally, if the modified data pages and undo logs meet the recovery requirements, the records are restored to obtain the restored records. Therefore, database instance recovery can be performed in database systems with only a single host and no backup nodes. This means that data pages, data dictionary metadata, and records can be recovered to the same version as before the failure. This does not require a high-availability database system or additional database hardware costs. Data pages, data dictionary metadata, and records can be recovered according to the data recovery order, which can reduce omissions during recovery and improve data recovery speed while significantly reducing the implementation hardware cost of the database system.

[0054] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0055] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0056] Figure 1 This is a flowchart illustrating a data recovery method according to an exemplary embodiment;

[0057] Figure 2This is a flowchart illustrating a data recovery method according to an exemplary embodiment;

[0058] Figure 3 This is a schematic diagram illustrating an example of the internal structure of a database system according to an exemplary embodiment;

[0059] Figure 4 This is an example schematic diagram illustrating a full data synchronization method according to an exemplary embodiment;

[0060] Figure 5 This is a flowchart illustrating a data recovery method according to an exemplary embodiment;

[0061] Figure 6 This is a block diagram illustrating a data recovery apparatus according to an exemplary embodiment;

[0062] Figure 7 This is a block diagram illustrating a network device according to an exemplary embodiment. Detailed Implementation

[0063] To enable those skilled in the art to better understand the technical solutions of this disclosure, the technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0064] This disclosure provides data recovery methods and apparatus. In some embodiments, the terms "data recovery method" and "information processing method" and "communication method" can be used interchangeably; the terms "data recovery apparatus" and "information processing apparatus" and "communication apparatus" can be used interchangeably; and the terms "information processing system" and "communication system" can be used interchangeably.

[0065] This disclosure is not exhaustive, but merely illustrative of some embodiments, and is not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment can be arbitrarily interchanged. Furthermore, the optional implementation methods in a particular embodiment can be arbitrarily combined; moreover, the embodiments can be arbitrarily combined, for example, some or all steps of different embodiments can be arbitrarily combined, and a particular embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0066] In each of the disclosed embodiments, unless otherwise specified or in case of logical conflict, the terminology and / or descriptions of the embodiments are consistent and can be referenced by each other. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships.

[0067] The terminology used in the embodiments of this disclosure is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure.

[0068] In this embodiment of the disclosure, unless otherwise stated, elements expressed in the singular form, such as "a," "an," "the," "the," "the," "the," "the," "the," "this," etc., can mean "one and only one," or "one or more," "at least one," etc. For example, when using articles such as "a," "an," "the," etc. in translation, the noun following the article can be understood as either a singular expression or a plural expression.

[0069] In the embodiments disclosed herein, "multiple" refers to two or more.

[0070] In some embodiments, the terms “at least one of”, “one or more”, “a plurality of”, “multiple”, etc., may be used interchangeably.

[0071] The prefixes "first," "second," etc., used in the embodiments of this disclosure are merely for distinguishing different descriptive objects and do not impose restrictions on the position, order, priority, quantity, or content of the descriptive objects. The description of the descriptive objects is found in the claims or the context of the embodiments, and the use of prefixes should not constitute unnecessary restrictions. For example, if the descriptive object is a "field," the ordinal numbers preceding "field" in "first field" and "second field" do not restrict the position or order of the "fields." "First" and "second" do not restrict whether the "fields" they modify are in the same message, nor do they restrict the order of "first field" and "second field." Similarly, if the descriptive object is a "level," the ordinal numbers preceding "level" in "first level" and "second level" do not restrict the priority between "levels." Furthermore, the number of descriptive objects is not limited by ordinal numbers and can be one or more. For example, in "first device," the number of "devices" can be one or more. Furthermore, the objects modified by different prefixes can be the same or different. For example, if the object being described is "device", then "first device" and "second device" can be the same device or different devices, and their types can be the same or different. Similarly, if the object being described is "information", then "first information" and "second information" can be the same information or different information, and their content can be the same or different.

[0072] In some embodiments, "terminal" or "terminal device" may be referred to as "user equipment (UE)," "user terminal," "mobile station (MS)," "mobile terminal (MT)," "subscriber station," "mobile unit," "subscriber unit," "wireless unit," "remote unit," "mobile device," "wireless device," "wireless communication device," "remote device," "mobile subscriber station," "access terminal," "mobile terminal," "wireless terminal," "remote terminal," "handset," "user agent," "mobile client," "client," etc.

[0073] In some embodiments, data, information, etc., may be obtained with the user's consent.

[0074] It should be noted that the terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0075] Figure 1 This is a flowchart illustrating a data recovery method according to an exemplary embodiment, such as... Figure 1 As shown, this data recovery method can be used to determine the root cause information corresponding to abnormal information when abnormal information occurs in the system, and includes the following steps:

[0076] In step S11, after the database system recovers from a failure and a startup command for the database system is received, the physical redo log file in the file system that is closest to the current time is loaded, wherein the database system includes a database node;

[0077] According to some embodiments, the execution subject of this disclosure is a network device. The name of the network device is not limited. For example, the network device may also be called a server, or a server cluster, etc.

[0078] According to some embodiments, a database system may be, for example, a system composed of a database and its management software. This database system may be, for example, a highly available database system, such as a MySQL master cluster, Oracle DataGuard, etc. In this embodiment, the database system may contain only one database node, which separates computation and storage. The computation layer may use a single cloud host or Docker, and the storage layer may be distributed storage (such as a distributed file system or distributed block storage).

[0079] In some embodiments, the startup command may be a command received after a database system failure has been repaired, used to start the database system. This startup command is not specifically a single, fixed command. For example, it may be a voice startup command, a timed startup command, or a click-to-start command.

[0080] In some embodiments, a file system is, for example, the file subsystem of an operating system. Database data pages are ultimately persistently stored in the file system's data page file. A file system is an important mechanism in a computer for organizing, storing, and accessing data. It serves as the interface between the operating system and storage devices (such as hard drives and flash memory), providing users and applications with a unified and abstract way to access and manage data. There are various types of file systems, each with its own characteristics and applicable scenarios. For example, FAT (File Allocation Table) is commonly used in Windows systems and is characterized by its simplicity, ease of implementation, and good compatibility.

[0081] In some embodiments, the physical redo log file may be generated by the database system for each modification operation on a data page. The physical redo log file can record the modification information of the data page by a transaction. The physical redo log file may include at least one physical redo log file. This physical redo log file is not specifically a fixed file. For example, when the number of physical redo log files included in the physical redo log file changes, the physical redo log file may also change accordingly. For example, when the modification operations on the data page before the failure occurred change, the physical redo log file may also change accordingly.

[0082] In some implementations, the Redo Log is an important logging mechanism in database systems, primarily used to record all operations that modify the database, such as inserts, updates, and deletions, to ensure data integrity and consistency. In relational databases like MySQL, the Redo Log plays a crucial role, including: ensuring transaction durability: when a database failure occurs, for example, if dirty pages (dirty data) in the buffer pool have not yet been written to disk, the Redo Log can identify the data records that need to be rewritten to the disk file, thereby restoring the data to its pre-failure state and improving transaction commit speed. In some implementations, directly flushing data from the buffer pool to disk is a random I / O operation, which is inefficient. Therefore, recording data from the buffer pool to the Redo Log is a sequential I / O operation, which can significantly improve transaction commit speed.

[0083] According to some embodiments, the physical log redo file most recent to the current time may, for example, refer to the physical log redo file corresponding to the last modification to a data page.

[0084] In some embodiments, upon recovery from a database system failure and receipt of a startup command for the database system, the physical redo log file most recent to the current time is loaded from the physical redo log file in the file system, wherein the database system includes a database node.

[0085] In step S12, based on the scan order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, the data page corresponding to each physical log redo log file in the at least one physical log redo log file is read from the disk into memory, and the data page is modified in the memory according to each physical log redo log file, and the modified data page is rewritten back to the disk;

[0086] According to some embodiments, the scan order can be used, for example, to indicate the order in which at least one physical log redo log file is scanned. This scan order is not specifically defined by a fixed sequence. For example, the scan order could be scanning from oldest to newest, starting with a checkpoint, or scanning from newest to oldest, starting with a checkpoint.

[0087] In some instances, a disk can refer to a storage device that uses magnetic recording technology to store data. In this embodiment of the disclosure, the redo log will always be flushed to the disk when a transaction is committed.

[0088] According to some embodiments, memory may be used, for example, to store at least one physical log redo log file.

[0089] In some embodiments, based on the scan order corresponding to at least one physical redo log file in the physical redo log files and the physical redo log file closest to the current time, the data page corresponding to each physical redo log file in the at least one physical redo log file is read from the disk into memory, the data page is modified in the memory according to each physical redo log file, and the modified data page is rewritten back to the disk.

[0090] In step S13, if the modified data page and the undo log meet the recovery requirements, the data dictionary metadata is recovered to obtain the recovered data dictionary metadata, wherein the undo log includes the records in the modified data page;

[0091] In some embodiments, the undo log can be, for example, the undo log generated when the database system inserts, modifies, or deletes each record. This undo log does not refer to a specific, fixed undo log. For instance, when a record changes, the undo log can also change accordingly.

[0092] In some embodiments, the recovery requirement may be, for example, determining whether the data dictionary metadata can be repaired before doing so. This recovery requirement may, for example, be restoring consistency between the modified data page and the undo log.

[0093] In some embodiments, the data dictionary is used to store metadata information for tables / indexes. This metadata information includes, but is not limited to, table / index definitions, the correspondence between tables / indexes and their internal data structures (B+ trees, tablespaces, files, etc.), and globally unique IDs. The data managed and maintained by the data dictionary is the database table index metadata. The metadata itself exists in record form and is stored in pages.

[0094] According to some implementations, a data dictionary is a collection that defines and describes data items, data structures, data flows, data storage, and processing logic. Its purpose is to provide detailed explanations of each element in the data flow diagram. Alternatively, a data dictionary can be a collection of information describing data, a set of definitions for all data elements used in the system. When a database system fails, to ensure that the metadata of the data dictionary remains consistent with the time of the failure and to maintain transaction-level integrity, recovery is typically performed in conjunction with data pages and undo logs.

[0095] In some embodiments, metadata itself exists in the form of records and is stored in pages. By implementing consistency recovery for records and pages in both the record and page modules, metadata recovery can be achieved. Therefore, the data dictionary fault recovery problem can be reduced to a record and page fault recovery problem.

[0096] According to some embodiments, if the modified data page and the undo log meet the recovery requirements, the data dictionary metadata is recovered to obtain the recovered data dictionary metadata, wherein the undo log includes the records in the modified data page.

[0097] In step S14, if the modified data page and undo log meet the recovery requirements, the record is recovered to obtain the recovered record.

[0098] According to some implementations, when a database system fails, in order to ensure that the records remain consistent with the time of the failure and to maintain transaction-level integrity, recovery can be performed by combining data pages and undo logs.

[0099] In some embodiments, if the modified data page and undo log meet the recovery requirements, the record is recovered to obtain the recovered record.

[0100] In some or related embodiments, upon failure recovery of the database system and receipt of a startup command for the database system, the system loads the physical redo log file in the file system that is most recent in time. The database system includes a database node. Based on the scan order of at least one physical redo log file and the most recent physical redo log file, data pages corresponding to each physical redo log file are read from the disk into memory. The data pages are then modified in memory according to the physical redo log files, and the modified data pages are written back to the disk. If the modified data pages and undo logs meet the recovery requirements, the data dictionary metadata is restored to obtain the restored data dictionary metadata. The undo logs include records from the modified data pages. Finally, if the modified data pages and undo logs meet the recovery requirements, the records are restored to obtain the restored records. Therefore, database instance recovery can be performed in database systems with only a single host and no backup nodes. This means that data pages, data dictionary metadata, and records can be recovered to the same version as before the failure. This does not require a high-availability database system or additional database hardware costs. Data pages, data dictionary metadata, and records can be recovered according to the data recovery order, which can reduce omissions during recovery and improve data recovery speed while significantly reducing the implementation hardware cost of the database system.

[0101] Figure 2 This is a flowchart illustrating a data recovery method according to an exemplary embodiment, such as... Figure 2 As shown, this data recovery method can be used in data recovery scenarios and includes the following steps:

[0102] In step S21, after the database system recovers from a failure and a startup command for the database system is received, the physical redo log file in the file system that is closest to the current time is loaded, wherein the database system includes a database node;

[0103] According to some embodiments, the execution subject of this disclosure is a network device. The name of the network device is not limited. For example, the network device may also be called a server, or a server cluster, etc.

[0104] In some embodiments, Figure 3 This is an exemplary schematic diagram illustrating the internal structure of a database system according to an exemplary embodiment, such as... Figure 3 As shown, the internal structure of a disk-based database system may include, for example:

[0105] 1. SQL module: Receives SQL statements from clients, parses them into a syntax tree, translates them into an execution plan tree, and finally delivers the execution plan to the executor.

[0106] 2. Record Module: When the SQL module's executor executes an SQL statement, it calls the record module to read and write the records that the SQL statement needs to process. The main task of the record module is to perform a series of management and maintenance operations around the record data object;

[0107] 3. Page Module: In a database system, records are stored in pages. When the record module needs to read a record from disk, it requests the page module to contain the records for that page. The main task of the page module is to perform a series of management and maintenance operations around the page data object.

[0108] 4. File System: The file system is the file subsystem of the operating system. Database pages will ultimately be persistently stored in page files within the file system.

[0109] 5. Data Dictionary: Stores metadata information for tables / indexes. This metadata information includes, but is not limited to: table / index definitions, the correspondence between tables / indexes and internal data structures (B+ trees, tablespaces, files, etc.); globally unique IDs, etc.

[0110] 6. Transaction Module: Transactions are a crucial feature of database systems. After a transaction is committed, the database system must guarantee that all modifications made to the data by that transaction are completed; partial modifications are not permitted. Similarly, after a transaction is rolled back, the database system must guarantee that all modifications made by that transaction are undone; partial rollbacks are not allowed. Furthermore, the modifications made by a transaction must be persisted to the file system.

[0111] In a database system, the transaction module is responsible for ensuring the atomicity and durability of transactions. These guarantees are achieved through two mechanisms:

[0112] Mechanism 1: Redo log mechanism for the page module: For each page modification operation, the database system generates a corresponding redo log, which records the modifications made to the page by the transaction. For example, if a transaction performs a +1 operation on a field of a record in a page, the generated redo log records the result of that +1. When the transaction is committed, the database system flushes the redo log to the file system, ensuring that all modifications made to the page by the transaction are persisted to the file system.

[0113] Mechanism 2: Undo log mechanism for the record module: When the database system inserts, modifies, or deletes each record, a corresponding undo log is generated. The undo log records the version of the record before the transaction operation. For example, if a transaction performs a +1 operation on a field of a record, an undo log is generated recording the result of that field before the +1 operation. When the transaction is committed, these undo logs are deleted; when the transaction is rolled back, the database system uses these undo logs to roll back the modified record, undoing the transaction's modifications to the record.

[0114] Undo logs are also stored in pages. The page that stores undo logs is called an undologpage. Therefore, there are two types of pages in a database system: recordpages and undo logpages. Both types of pages can be used for recovery using redo logs. Unless otherwise specified, these two types of pages will be referred to as "pages" below.

[0115] In the fault recovery operation after the database system restarts following a failure, since the data in the SQL module is cached data that does not need to be persisted, it is not necessary to restore the data in the SQL module during fault recovery.

[0116] According to some embodiments, the latest redo log can be loaded from the redo log file in the file system, ensuring that the redo log in memory remains consistent with the point of failure. The latest redo log could be, for example, the redo log most recent than the current time.

[0117] In some embodiments, the generation and release process of the Redo Log may include, for example, the following: As a transaction operation is executed, a Redo Log may be generated; upon transaction commit, the generated Redo Log is written to the Log Buffer instead of immediately writing to the disk file. Once the dirty pages of the transaction operation are written to disk, the Redo Log's mission is complete, and the space it occupied can be overwritten, i.e., reused.

[0118] In some implementations, the Redo Log works with the goal of ensuring transaction durability. When a database failure occurs, the Redo Log is used to redo data, ensuring that data not yet written to disk is persisted. Specifically, the Redo Log file content is written sequentially in a circular fashion; when a file is full, it backtracks to the first file to overwrite it.

[0119] In some implementations, the latest redo log is loaded from the redo log file in the file system to ensure that the redo log records in memory are consistent with the state at the time of the failure. This is the basis of the recovery process because only by knowing which modifications have not yet been applied to the data files can the data be recovered correctly.

[0120] In step S22, the scan start point corresponding to the physical log redo log file is obtained, wherein the scan start point is the checkpoint corresponding to the physical log redo log file closest to the current time;

[0121] The specific process is as described above and will not be repeated here.

[0122] According to some implementations, the scan start point corresponding to the physical redo log file can be obtained. This scan start point can also be referred to as the scan start point or the last checkpoint. A checkpoint is the point in time when InnoDB periodically flushes dirty pages (pages that have been modified but not yet written to disk) from memory to disk. Specifically, when starting up after a system crash, the scan needs to begin from a position before the crash, which can rely on the log position recorded by the crash recovery mechanism.

[0123] According to some embodiments, the scan start point corresponding to the physical log redo log file is obtained, wherein the scan start point is the checkpoint corresponding to the physical log redo log file closest to the current time.

[0124] In step S23, the physical log redo log file is scanned according to the scan point and scan order. During the scan, if an unscanned physical log redo log file is found, the log sequence number LSN is updated. The scan order is from the furthest to the nearest time.

[0125] The specific process is as described above and will not be repeated here.

[0126] In some implementations, scanning the redo log is a critical step in the database recovery process, especially when using a storage engine such as InnoDB in MySQL. The redo log is written cyclically and contains physical modifications to the database, used to recover data page changes that were not committed to disk in the event of a system crash.

[0127] In some implementations, the redo log can be stored, for example, in files on disk, which can be opened and read. Depending on the InnoDB configuration, there may be multiple redo log files that form a circularly written log group.

[0128] In some implementations, the redo log is sorted by Log Sequence Number (LSN), a monotonically increasing number representing the position of a log entry in the log file. During scanning, modifications can be applied in LSN order to ensure the sequential nature of operations. Error handling or recovery strategies can be implemented if any discontinuous or corrupted log entries are encountered.

[0129] In step S24, if the scanning operation meets the scan completion condition, it is determined that at least one physical log redo log file has been obtained, wherein the at least one physical log redo log file is used to recover data pages;

[0130] The specific process is as described above and will not be repeated here.

[0131] In step S25, the data pages corresponding to each physical redo log file in the at least one physical log file are read from the disk into memory;

[0132] The specific process is as described above and will not be repeated here.

[0133] According to some embodiments, reading the data pages corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory includes:

[0134] Based on the scan start point corresponding to the physical log redo log file, it is determined that there is modification information of data page in the record of the at least one physical log redo log file, and during the scanning process, if an unscanned physical log redo log file is found, the log sequence number LSN is updated.

[0135] Based on the modified information, the data page corresponding to the modified information is read from the data file on the disk and loaded into memory.

[0136] In some embodiments, the currently processed LSN can be tracked during the scan. When a new or previously unscanned log entry is encountered, the current LSN is updated, which can improve the accuracy of determining the scan completion time, such as reaching the end of the log file or reaching a specific recovery point.

[0137] In some implementations, a scan can be considered complete when the end of the log file is reached or a specific recovery point is achieved. At this point, a checkpoint may need to be performed to flush dirty pages in memory to disk and update the checkpoint location.

[0138] According to some embodiments, the method further includes:

[0139] The at least one physical redo log file is obtained using a multi-threaded or multi-process approach.

[0140] or,

[0141] Pre-read data pages from at least one physical log redo file;

[0142] or;

[0143] The at least one physical redo log file is compressed to obtain at least one processed physical redo log file. Using a multi-threaded or multi-process approach to obtain the at least one physical redo log file allows for parallel processing of the physical redo log files, which can improve recovery speed. However, this embodiment does not limit the choice between the multi-threaded or multi-process approach.

[0144] According to some embodiments, during the various processes described above, data pages recorded in at least one physical log (redo log file) can be pre-read, which can reduce random I / O operations and improve recovery efficiency. The pre-read method is not limited; any method that enables pre-read operations is acceptable.

[0145] According to some embodiments, the at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file. This can increase the compression of redo logs, reduce storage space and I / O overhead, and the specific compression method is not limited.

[0146] According to some embodiments, Figure 4 This is an example schematic diagram illustrating a full data synchronization method according to an exemplary embodiment, such as... Figure 4 As shown, full data synchronization can be performed during the initial stage of redo log file recovery, or when incremental synchronization is interrupted or encounters data errors. During full synchronization, incremental synchronization stops. At the source end, Redis's own mechanism packages the data into files and transmits them to the synchronization proxy component at the destination. Upon receiving the full data, the synchronization proxy component first writes the data to a separate Redis database (the back buffer). The Redis database currently used by the application is called the front buffer. The access addresses of the two are exchanged, thus completing the full synchronization process. During incremental synchronization, the synchronization proxy component masquerades as a slave node at the source end, receiving the incrementally modified data after the full synchronization and then writing it to the destination end.

[0147] In step S26, the data page is modified in the memory according to the redo log files of each physical log, and the modified data page is rewritten back to the disk;

[0148] The specific process is as described above and will not be repeated here.

[0149] According to some embodiments, the corresponding page can be read from the disk into memory based on the records in the redo log, and the corresponding page can be modified in memory according to the contents of the redo log.

[0150] According to some implementations, when a database system recovers data from the redo log, it reads and modifies data pages based on the records in the redo log. Specific examples may include:

[0151] 1. Initiating the recovery process: This determines which position in which redo log file the recovery should begin. For example, it could be the checkpoint position written during the last normal shutdown, or the last redo log position written when the system crashed.

[0152] 2. Open and read redo log files: You can open physical files containing redo logs and read records starting from a defined starting point.

[0153] 3. Parse redo log records to obtain modified data pages and their modifications: For each redo log record read, it can be parsed to determine which data pages were modified and the specific modifications made to those pages. The parsed information typically includes the tablespace ID, page number, modification type, and modified data.

[0154] 4. Read data pages from disk: Based on the parsed tablespace ID and page number, read the corresponding data pages from the data file on disk into memory. If there is not enough space in memory to store these data pages, a page replacement strategy (such as the LRU algorithm) may be used to manage the data pages in memory.

[0155] 5. Modifying data pages in memory: Once data pages are loaded into memory, they can be updated based on changes recorded in the redo log. This may include modifying specific data on the page, updating page metadata (such as page status, transaction ID, etc.), or performing other necessary operations. Modified data pages become "dirty pages" in memory, meaning their content differs from that on disk, and they need to be flushed back to disk at some future point in time.

[0156] 6. Track and update LSN: When processing each redo log record, the currently processed LSN (Log Sequence Number) can be updated to ensure that modifications are applied in the correct order. The LSN is a monotonically increasing sequence number in the redo log used to identify the order and position of log records.

[0157] 7. Loop processing: Continue reading and parsing the next record in the redo log, and repeat steps 3-6 above until all redo log records have been processed or a certain recovery goal has been reached (such as recovering to a specific point in time).

[0158] 8. Recovery process complete: Once all redo log records have been processed, the recovery process is complete. At this point, the data pages have been updated to the state at the time of the crash, and the database can be reopened for read and write operations.

[0159] In some implementations, the modified page is written back to disk, ensuring that the data on disk reflects the state before the failure. Since the undo log is typically stored in data pages, it is automatically restored to its pre-failure state when the data pages are recovered.

[0160] In step S27, if the modified data page and the undo log meet the recovery requirements, the data dictionary metadata is recovered to obtain the recovered data dictionary metadata, wherein the undo log includes the records in the modified data page;

[0161] The specific process is as described above and will not be repeated here.

[0162] According to some embodiments, the step of restoring the data dictionary metadata and obtaining the restored data dictionary metadata includes:

[0163] Based on the recovery starting point, data pages, including data dictionary metadata, are read from the disk into memory;

[0164] Use the records in the physical log redo log file to obtain the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point;

[0165] Based on the physical log redo log file and the change information, the data dictionary metadata is restored. During the data dictionary metadata restoration process, the data dictionary metadata is verified. Once the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

[0166] According to some embodiments, restoring data dictionary metadata may include, for example:

[0167] 1. Determine the recovery starting point: This allows you to determine from which point in time or log position to begin recovery. For example, the recovery starting point can be determined using checkpoint information or the log position of the last successful commit.

[0168] 2. Read Data Pages: Read data pages containing data dictionary metadata from disk into memory. Data pages may include system tables, directory tables, indexes, etc.

[0169] 3. Apply redo log: Use the records in the redo log to reapply all changes made to the data dictionary metadata since the last checkpoint; that is, obtain change information, which may include, for example, insert, update, and delete operations. Through the redo log, the data dictionary metadata can be restored to a consistent state before the failure.

[0170] 4. Handling incomplete transactions: Traverse the undo log to find transactions that were active (i.e., uncommitted) at the time of the failure. For each uncommitted transaction, undo its changes to data dictionary metadata based on its undo log record. This ensures that only changes from committed transactions are preserved.

[0171] 5. Verify data consistency: During the recovery process, verification steps can be performed to ensure the consistency of data dictionary metadata. These steps may include checking referential integrity and triggering constraint checks.

[0172] 6. Complete Recovery: Once all necessary changes have been applied and data consistency has been verified, the recovery process is complete. At this point, the data dictionary metadata should be consistent with the state at the time of the failure and maintain transaction-level integrity.

[0173] In step S28, if the modified data page and undo log meet the recovery requirements, the record is recovered to obtain the recovered record.

[0174] The specific process is as described above and will not be repeated here.

[0175] According to some embodiments, the step of recovering the record and obtaining the recovered record includes:

[0176] Read the data page containing the target record from the disk into memory;

[0177] Traverse the undo log to find the transactions that were in the target state when the failure occurred;

[0178] Based on the undo log records corresponding to each transaction in the target state, undo the changes made by each transaction in the target state to the target record.

[0179] During the target record recovery process, the target record is verified, and upon completion of the verification, the recovered record is obtained.

[0180] Specifically, the process of restoring the record may include, for example:

[0181] 1. Read data page: Read the data page containing the target record from the disk into memory.

[0182] 2. Apply redo log: If there are redo log records that have not been written to disk when the failure occurs, these records can be applied to ensure that changes from all transactions committed before the failure are reflected in the data pages.

[0183] 3. Handling incomplete transactions: Traverse the undo log to find transactions that were active (i.e., uncommitted) at the time of the failure. The target state can be either active or uncommitted. For each uncommitted transaction, based on its undo log record, undo its changes to the target record, rolling the record back to its state before the transaction began.

[0184] 4. Verify data consistency: During the recovery process, verification steps can be performed to ensure the consistency and integrity of records. This may include checking foreign key constraints, triggers, etc.

[0185] 5. Complete Recovery: Once all necessary changes have been applied and data consistency has been verified, the recovery process is complete. At this point, the target records should be consistent with those at the time of the failure and maintain transaction-level integrity.

[0186] Different processes can be used for different database systems; that is, the order of steps S21 to S27 is not fixed. Some database systems may provide more advanced recovery tools and mechanisms, such as point-in-time recovery or log-based recovery.

[0187] According to some embodiments, the method further includes:

[0188] Receive a Structured Query Language (SQL) statement sent by the client, and read the target data page from the file system according to the SQL statement;

[0189] The target data page is placed into the data page cache pool, and if the target data page exists in the data page cache pool, the redo log-data page refresh operation and the linked list update operation are executed.

[0190] Determine the record to be read corresponding to the target data page, and determine the target record based on the processing status of the suspended transaction.

[0191] According to some embodiments, the SQL module can receive SQL statements sent by the client to the SQL module. The SQL module generates an execution plan for the SQL statement and executes it. During the execution process, the record module is called to obtain the record, and the record module calls the page module to obtain the page. Since the database system has just been initialized, the page module does not have the page in its page cache pool. Therefore, the page module sends a request to the file system to read the page. The file system can return an older version of the page and put it into the page cache pool. When the page cache pool contains a newly added page from the file system, the redolog-page refresh device is started and a linked list update operation is performed. The redolog-page refresh device refreshes the newly added page in the page cache pool and puts the latest version of the refreshed page into the page cache pool. The page module retrieves the page from the page cache pool and returns it to the record module. The record module then checks if there are any pending transactions for the record to be read in the page. If not, it means the record is correct and can be returned directly to the SQL module. If there are pending transactions, it means the record is the latest version, and it is not returned to the SQL module. Ultimately, the record obtained by the SQL module is the latest and correct version of the record.

[0192] When a newly added page from the file system exists in the page cache pool, the redo log-page refresh mechanism is activated, and a linked list update operation is performed. Specifically, this may include reading the latest redo log from the redo log file in the file system, parsing the read redo log to obtain its page number, saving the parsing result (redo logs with the same page number are stored in the same singly linked list), reading older (historical) versions of pages from the file system's page file, querying each singly linked list based on the page number, finding and using the corresponding redo log, and refreshing the page to the latest version.

[0193] In one or related embodiments, the scan start point corresponding to the physical log redo log file is obtained, wherein the scan start point is the checkpoint corresponding to the physical log redo log file closest to the current time; according to the scan point and scan order, the physical log redo log file is scanned, and during the scan, if an unscanned physical log redo log file is found, the log sequence number (LSN) is updated, wherein the scan order is from oldest to youngest from the current time; if the scan operation meets the scan completion condition, it is determined that at least one physical log redo log file has been obtained, wherein the at least one physical log redo log file is used to recover data pages; the data pages corresponding to each physical log redo log file in the at least one physical log redo log file are read from the disk into memory. Scanning by scan point and scan order reduces the probability of data loss during recovery and improves the integrity of data recovery. In addition, updating by log sequence number (LSN) allows for sequential modification of references, improving the accuracy and speed of data recovery.

[0194] Figure 5 This is a flowchart illustrating a data recovery method according to an exemplary embodiment, such as... Figure 5 As shown, this data recovery method can be used in the fault recovery process of a single-host database system, where the database system's data pages, records, and metadata are all restored to be consistent with the state at the time of the fault, and external clients can read the data. The method includes the following steps:

[0195] In step S31, the redo log file most recent to the current time is read from the redo log files in the file system;

[0196] The specific details have been described above and will not be repeated here.

[0197] According to some embodiments, the redo log file most recent to the current time can be read from the redo log files in the file system.

[0198] In some implementations, redo log files in a database system may be written in a circular fashion, for example, switching to the next log file to continue writing once one log file is full. The currently being written to or the most recently written to redo log file can be determined based on the writing order. This can be achieved, for example, by querying database status information, reading configuration files, or log files.

[0199] In some implementations, redo log files are opened using file access APIs (such as POSIX API, Windows API, etc.). When the contents of the redo log are stored in binary format, the file can be opened in binary mode.

[0200] According to some embodiments, the header information of Redo Log entries can be read. Each Redo Log entry may include header information, including but not limited to the length, type, and sequence number of the log entry. This header information can be read first to determine how to parse the log content.

[0201] In some implementations, Redo Log entries can be parsed: based on the header information, the specific content of the Redo Log entry can be parsed. For example, binary data can be decoded to obtain information such as page number, modification type, and modified data. The specific parsing method depends on the database system's definition of the Redo Log format. For example, the parsing method for Redo Log entries can be determined by consulting relevant documentation or source code.

[0202] According to some embodiments, Redo Log entries can be processed: based on the parsed redo log entry content, corresponding operations can be performed to restore or redo these changes. For example, this may include loading data pages into memory, modifying the content of data pages, etc.

[0203] In some embodiments, error handling and verification can be performed: throughout the process, errors that occur, such as file read errors and parsing errors, can be handled. Simultaneously, the validity and consistency of the parsed redo log entries can be verified.

[0204] In some embodiments, a redo log index may also be constructed: before parsing the redo log, an index or data structure is needed to quickly locate redo log entries associated with a specific page number. This index can be based on log sequence numbers (LSNs), because redo logs are typically written in LSN order, and each page change has a corresponding LSN range.

[0205] In some implementations, the file is closed: after reading and parsing the redo log file is complete, the file is closed to release resources.

[0206] In step S32, the redo log file closest to the current time is parsed to obtain the page number (pageno) of the data page of the redo log closest to the current time;

[0207] According to some embodiments, the redo log file most recent to the current time is parsed to obtain the page number of the data page of the most recent redo log file. That is, the latest redo log file can be parsed to obtain the page number of its data page. The page number is used to identify a specific data page in the database.

[0208] According to some implementations, the Log Header can be located. A redo log file may have one or more header sections, which include metadata information about the log file. The header information can be located and parsed first to obtain the overall structure and format of the log file.

[0209] According to some implementations, log entries are parsed: the redo log file consists of a series of log entries, each recording one or more changes to a data page. Log entries can be traversed and parsed to obtain the relevant page number and other information.

[0210] In some implementations, the page number is parsed: when parsing log entries, the page number field can be located and extracted. The specific location and format of this field depend on the format of the redo log. Generally, the page number is stored as a fixed-length byte and may be accompanied by other related metadata (such as block size, tablespace ID, etc.).

[0211] According to some embodiments, variable length and encoding are handled: In some cases, page numbers or other fields may be stored in a way that specifies the variable length or a particular encoding. This encoding can be understood and processed to ensure that page numbers are parsed correctly.

[0212] According to some implementation examples, verification and testing are recommended: When writing parsing code, it is also advisable to write test cases to verify the correctness of the parsing logic. The code can be tested using known data and scenarios, and the results can be checked to see if they meet expectations.

[0213] Specifically, specialized tools or APIs provided by the database system are used to parse the redo log file. Since the tools used have been optimized and tested, the accuracy of parsing the redo log file and extracting relevant information can be ensured.

[0214] In step S33, the data page number is stored, and the redo log file corresponding to the same data page number in the file system is stored in a singly linked list corresponding to the same data page number;

[0215] In step S34, the target data page is read from the data page file of the file system, and then each unidirectional linked list is queried according to the target data page pag to obtain and use the redo log corresponding to the target data page pag.

[0216] According to some implementations, after all relevant redo log changes have been applied, the consistency and integrity of the data pages can be verified. If the data pages are valid, they can be written back to the data file or the target location in memory for later use.

[0217] In one or related embodiments, the most recent redo log file is read from the redo log file in the file system; the most recent redo log file is parsed to obtain the data page number of the most recent redo log; the data page number is stored, and the redo log files corresponding to the same data page number in the file system are stored in a singly linked list corresponding to the same data page number; a target data page is read from the data page file in the file system, and then each singly linked list is queried according to the target data page pg to obtain and use the redo log corresponding to the target data page pg. Therefore, after the pages, records, and metadata in the database system are recovered, any page read from the file system can be refreshed to the latest version using the redo log corresponding to the stored page and placed into the page cache pool. Since the page version is now the latest, it can be provided to upper-layer business logic or external clients, continuing the data and state at the time of the master node failure, thereby providing services externally and improving service quality.

[0218] A block diagram of a data recovery apparatus is shown according to an exemplary embodiment. (Refer to...) Figure 6 The device 600 includes:

[0219] Log loading unit 601 is used to load the physical redo log file in the file system that is closest to the current time when the database system is recovering from a failure and a startup command for the database system is received. The database system includes a database node.

[0220] The data page modification unit 602 is used to read the data page corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory according to the scan order corresponding to each physical redo log file and the physical redo log file closest to the current time, modify the data page in the memory according to each physical redo log file, and rewrite the modified data page back to the disk;

[0221] Data recovery unit 603 is used to recover data dictionary metadata and obtain recovered data dictionary metadata when the modified data page and undo log meet the recovery requirements, wherein the undo log includes records in the modified data page;

[0222] The data recovery unit 603 is also used to recover the record and obtain the recovered record if the modified data page and undo log meet the recovery requirements.

[0223] According to some embodiments, when the data page modification unit 602 reads the data page corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory based on the scan order corresponding to at least one physical redo log file in the physical redo log file and the physical redo log file closest to the current time, it is specifically used for:

[0224] Obtain the scan start point corresponding to the physical log redo log file, wherein the scan start point is the checkpoint corresponding to the physical log redo log file closest to the current time;

[0225] According to the scan points and scan order, the physical log redo log file is scanned. During the scan, if an unscanned physical log redo log file is found, the log sequence number (LSN) is updated. The scan order is from the furthest to the nearest time from the current time.

[0226] If the scan operation meets the scan completion condition, it is determined that at least one physical log redo log file has been obtained, wherein the at least one physical log redo log file is used to recover data pages;

[0227] Read the data pages corresponding to each physical redo log file from the disk into memory.

[0228] According to some embodiments, the data page modification unit 602, when reading the data page corresponding to each physical redo log file from the disk into memory, is specifically used for:

[0229] Based on the scan start point corresponding to the physical log redo log file, it is determined that there is modification information of data page in the record of the at least one physical log redo log file, and during the scanning process, if an unscanned physical log redo log file is found, the log sequence number LSN is updated.

[0230] Based on the modified information, the data page corresponding to the modified information is read from the data file on the disk and loaded into memory.

[0231] According to some embodiments, the data page modification unit 602 is also specifically used for:

[0232] The at least one physical redo log file is obtained using a multi-threaded or multi-process approach.

[0233] or,

[0234] Pre-read data pages from at least one physical log redo file;

[0235] or;

[0236] The at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file.

[0237] According to some embodiments, the data recovery unit 603 is used to recover data dictionary metadata. Specifically, when obtaining the recovered data dictionary metadata, it is used for:

[0238] Based on the recovery starting point, data pages, including data dictionary metadata, are read from the disk into memory;

[0239] Use the records in the physical log redo log file to obtain the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point;

[0240] Based on the physical log redo log file and the change information, the data dictionary metadata is restored. During the data dictionary metadata restoration process, the data dictionary metadata is verified. Once the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

[0241] According to some embodiments, the data recovery unit 603 is used to recover a record, and when obtaining the recovered record, it is specifically used for:

[0242] Read the data page containing the target record from the disk into memory;

[0243] Traverse the undo log to find the transactions that were in the target state when the failure occurred;

[0244] Based on the undo log records corresponding to each transaction in the target state, undo the changes made by each transaction in the target state to the target record.

[0245] During the target record recovery process, the target record is verified, and upon completion of the verification, the recovered record is obtained.

[0246] According to some embodiments, the data recovery unit 603 is also specifically used for:

[0247] Receive a Structured Query Language (SQL) statement sent by the client, and read the target data page from the file system according to the SQL statement;

[0248] The target data page is placed into the data page cache pool, and if the target data page exists in the data page cache pool, the redo log-data page refresh operation and the linked list update operation are executed.

[0249] Determine the record to be read corresponding to the target data page, and determine the target record based on the processing status of the suspended transaction.

[0250] According to some embodiments, the data recovery unit 603 is also used for:

[0251] Read the redo log file that is closest to the current time from the redo log files in the file system;

[0252] The redo log file closest to the current time is parsed to obtain the page number of the data page of the redo log file closest to the current time;

[0253] Store the data page number, and store the redo log file corresponding to the same data page number in the file system into a singly linked list corresponding to the same data page number;

[0254] The target data page is read from the data page file of the file system, and then each unidirectional linked list is queried according to the target data page pg to obtain and use the redo log corresponding to the target data page pg.

[0255] Regarding the apparatus in the above embodiments, the specific methods by which each module performs its operations have been described in detail in the embodiments related to the method, and will not be elaborated upon here. The names of the various virtual devices are not limited; for example, the log loading unit can also be called a reading module, etc.

[0256] In some or related embodiments, a log loading unit is used to load the physical redo log file most recent in time from the physical log file in the file system when the database system is recovering from a failure and a startup command for the database system is received. The database system includes a database node. A data page modification unit is used to read data pages corresponding to each physical redo log file from the disk into memory according to the scan order of at least one physical redo log file and the most recent physical redo log file, and modify the data pages in memory according to each physical redo log file, then write the modified data pages back to the disk. A data recovery unit is used to recover data dictionary metadata and obtain the recovered data dictionary metadata when the modified data pages and undo logs meet the recovery requirements. The undo logs include records in the modified data pages. The data recovery unit is also used to recover data dictionary metadata in the modified data pages and undo logs. If the log meets the recovery requirements, the record is recovered to obtain the recovered record. Therefore, database instance recovery can be performed only in database systems with a single host and no backup node. This means that data pages, data dictionary metadata, and record data can be recovered to the version consistent with the state before the failure. This does not require a high-availability database system or additional database hardware costs. Data pages, data dictionary metadata, and record data can be recovered according to the data recovery order, reducing omissions during recovery and significantly reducing the implementation hardware costs of the database system while improving data recovery speed.

[0257] Figure 7 This is a block diagram of a network device 700 provided in an embodiment of this disclosure. For example, network device 700 can be provided as a network device. See also... Figure 7 The network device 700 includes a processing component 722, which further includes at least one processor, and memory resources represented by memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform any of the methods described above applied to the network device.

[0258] Network device 700 may also include a power supply component 727 configured to perform power management of network device 700, a wired or wireless network interface 750 configured to connect network device 700 to a network, and an input / output (I / O) interface 758. Network device 700 can operate on an operating system stored in memory 732, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0259] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0260] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0261] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0262] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0263] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), the Internet, and blockchain networks.

[0264] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0265] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0266] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data recovery method, characterized in that, include: In the event of database system failure recovery and receipt of a startup command for the database system, the physical redo log file in the file system that is closest to the current time is loaded, wherein the database system includes a database node; Based on the scan order corresponding to at least one physical redo log file in the physical log redo log files and the physical redo log file closest to the current time, read the data page corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory, modify the data page in the memory according to each physical redo log file, and rewrite the modified data page back to the disk; If the modified data page and the undo log meet the recovery requirements, the data dictionary metadata is recovered to obtain the recovered data dictionary metadata, wherein the undo log includes the records in the modified data page; If the modified data page and undo log meet the recovery requirements, the record is recovered and the recovered record is obtained. The method further includes: Receive a Structured Query Language (SQL) statement sent by the client, and read the target data page from the file system according to the SQL statement; The target data page is placed into the data page cache pool, and if the target data page exists in the data page cache pool, the redo log-data page refresh operation and the linked list update operation are executed. Determine the record to be read corresponding to the target data page, and determine the target record based on the processing status of the suspended transaction.

2. The method according to claim 1, characterized in that, The step of reading data pages corresponding to each physical redo log file from the disk into memory based on the scan order of at least one physical redo log file and the physical redo log file closest to the current time includes: Obtain the scan start point corresponding to the physical log redo log file, wherein the scan start point is the checkpoint corresponding to the physical log redo log file closest to the current time; According to the scan points and scan order, the physical log redo log file is scanned. During the scan, if an unscanned physical log redo log file is found, the log sequence number (LSN) is updated. The scan order is from the furthest to the nearest time from the current time. If the scan operation meets the scan completion condition, it is determined that at least one physical log redo log file has been obtained, wherein the at least one physical log redo log file is used to recover data pages; Read the data pages corresponding to each physical redo log file from the disk into memory.

3. The method according to claim 1 or 2, characterized in that, The step of reading the data pages corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory includes: Based on the scan start point corresponding to the physical log redo log file, it is determined that there is modification information of data page in the record of at least one physical log redo log file. During the scanning process, if an unscanned physical log redo log file is found, the log sequence number LSN is updated. Based on the modified information, the data page corresponding to the modified information is read from the data file on the disk and loaded into memory.

4. The method according to claim 3, characterized in that, The method further includes: The at least one physical redo log file is obtained using a multi-threaded or multi-process approach. or, Pre-read data pages from at least one physical log redo file; or; The at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file.

5. The method according to claim 1, characterized in that, The process of restoring the data dictionary metadata and obtaining the restored data dictionary metadata includes: Based on the recovery starting point, data pages, including data dictionary metadata, are read from the disk into memory; Use the records in the physical log redo log file to obtain the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point; Based on the physical log redo log file and the change information, the data dictionary metadata is restored. During the data dictionary metadata restoration process, the data dictionary metadata is verified. Once the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

6. The method according to claim 1, characterized in that, The process of restoring the record and obtaining the restored record includes: Read the data page containing the target record from the disk into memory; Traverse the undo log to find the transactions that were in the target state when the failure occurred; Based on the undo log records corresponding to each transaction in the target state, undo the changes made by each transaction in the target state to the target record. During the target record recovery process, the target record is verified, and upon completion of the verification, the recovered record is obtained.

7. The method according to claim 1, characterized in that, The method further includes: Read the redo log file that is closest to the current time from the redo log files in the file system; The redo log file closest to the current time is parsed to obtain the page number of the data page of the redo log file closest to the current time; Store the data page number, and store the redo log file corresponding to the same data page number in the file system into a singly linked list corresponding to the same data page number; The target data page is read from the data page file of the file system, and then each unidirectional linked list is queried according to the target data page pg to obtain and use the redo log corresponding to the target data page pg.

8. A data recovery device, characterized in that, include: The log loading unit is used to load the physical redo log file in the file system that is closest to the current time when the database system is recovering from a failure and a startup command for the database system is received. The database system includes a database node. The data page modification unit is used to read the data page corresponding to each physical redo log file in the at least one physical redo log file from the disk into memory according to the scan order corresponding to each physical redo log file and the physical redo log file closest to the current time, modify the data page in the memory according to each physical redo log file, and write the modified data page back to the disk; The data recovery unit is used to recover the data dictionary metadata and obtain the recovered data dictionary metadata when the modified data page and the undo log meet the recovery requirements. The undo log includes the records in the modified data page. The data recovery unit is also used to recover the record and obtain the recovered record if the modified data page and undo log meet the recovery requirements. The data recovery unit is further configured to: Receive a Structured Query Language (SQL) statement sent by the client, and read the target data page from the file system according to the SQL statement; The target data page is placed into the data page cache pool, and if the target data page exists in the data page cache pool, the redo log-data page refresh operation and the linked list update operation are executed. Determine the record to be read corresponding to the target data page, and determine the target record based on the processing status of the suspended transaction.

9. A network device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the instructions to implement the data recovery method as described in any one of claims 1 to 7.

Images