Data recovery method and device

Abstract

The invention relates to the technical field of big data, in particular to a data recovery method and device. The method comprises the steps of loading a redo log file closest to the current time in redo log files in a file system under the conditions that a fault of a database system is recovered and a starting instruction for the database system is received; according to the scanning sequence corresponding to at least one redo log file in the redo log files and the redo log files, pages corresponding to all the redo log files in the at least one redo log file are read from the disk to a memory, the pages are modified in the memory according to all the redo log files, and the modified pages are rewritten back to the disk; under the condition that the modified page and the undo log meet the recovery requirement, recovering the metadata of the data dictionary, and obtaining the recovered metadata of the data dictionary; under the condition that the modified page and the undo log meet the recovery requirement, the record record is recovered, and the recovered record record is obtained. According to the invention, the data recovery speed can be improved.

Description

Technical Field

[0001] The present disclosure relates to the field of big data technology, and in particular to a data recovery method and device. Background Art

[0002] In the entire hardware and software stack where the database system is located, both hardware and software may fail. If it is a stand-alone database system, system failures require a long processing time (because the database system takes a long time to restart, or requires manual intervention), which causes the database service to be unavailable for a long time. For example, a high-availability database system can be used to automatically handle and quickly recover database failures through redundant database nodes and automatic disaster recovery mechanisms. Among them, a high-availability database system can be composed of 1 master and N backups (N>=1). The hardware configuration (including CPU, memory, and disk) of the master and backup nodes of the high-availability database system is the same, that is, the master and backup are configured in a peer-to-peer manner. The master node provides read and write services to the outside world, and the backup node is responsible for keeping data synchronized with the master node. When the master node is abnormal, the management and control system selects a backup node, upgrades the backup node to the master node in a short time (the industry experience value is 30s), and provides services to the outside world. In order to ensure the speed of fault recovery, multiple backup hosts are generally equipped, but the hardware cost and fault recovery time of the database system are increased accordingly. Summary of the invention

[0003] The present disclosure provides a data recovery method and device, which can improve the data recovery speed. The technical solution of the present disclosure is as follows:

[0004] According to a first aspect of an embodiment of the present disclosure, a fault recovery method is provided, including:

[0005] When a database system is recovering from a failure and a startup instruction for the database system is received, loading a physical log redo log file closest to a current time from among physical log redo log files in a file system, wherein the database system includes a database node;

[0006] According to the scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory, modify the data page page in the memory according to each physical log redo log file, and rewrite the modified data page page back to the disk;

[0007] When the modified data page page and the undo log undo log meet the recovery requirement, the data dictionary metadata is restored to obtain the restored data dictionary metadata, wherein the undo log undo log includes the record record in the modified data page page;

[0008] When the modified data page page and undo log undo log meet the recovery requirements, the record record is restored to obtain the restored record record.

[0009] According to some embodiments, reading data pages corresponding to each physical log redo log file in at least one physical log redo log file from a disk to a memory according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, comprises:

[0010] Obtain a scan start point corresponding to a physical log redo log file, wherein the scan start point is a checkpoint corresponding to a physical log redo log file closest to the current time;

[0011] According to the scanning point and the scanning order, the physical log redo log file is scanned, and in the scanning process, when a physical log redo log file that has not been scanned is scanned in the physical log redo log file, a log sequence number LSN is updated, wherein the scanning order is the order from far to near from the current time;

[0012] In a case where the scanning operation satisfies the scanning completion condition, determining to obtain the at least one physical log redo log file, wherein the at least one physical log redo log file is used to recover the data page page;

[0013] Read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory.

[0014] According to some embodiments, reading from the disk to the memory the data page corresponding to each physical log redo log file in the at least one physical log redo log file includes:

[0015] According to the scanning start point corresponding to the physical log redo log file, determine that the modification information of the data page page exists in the at least one physical log redo log file record, and in the scanning process, when the physical log redo log file that is not scanned is scanned in the physical log redo log file, update the log sequence number LS N;

[0016] According to the modification information, a data page corresponding to the modification information is read from a data file on the disk to the memory.

[0017] According to some embodiments, the method further comprises:

[0018] Acquire the at least one physical log redo log file by adopting a multi-threading mode or a multi-process mode;

[0019] or,

[0020] Pre-reading a data page page in the at least one physical log redo log file record;

[0021] or;

[0022] The at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file.

[0023] According to some embodiments, the restoring the data dictionary metadata and obtaining the restored data dictionary metadata includes:

[0024] According to the recovery starting point, read the data page including the data dictionary metadata from the disk into the memory;

[0025] Using the records in the physical log redo log file, obtaining the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point;

[0026] The data dictionary metadata is restored according to the physical log redo log file and the change information, and during the data dictionary metadata restoration process, the data dictionary metadata is verified, and when the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

[0027] According to some embodiments, the recovering the record record and obtaining the recovered record record includes:

[0028] Read the data page including the target record from disk into memory;

[0029] Traverse the undo log and find the transactions that were in the target state when the failure occurred;

[0030] Undo the change information of the target record made by each transaction in the target state according to the undo log record corresponding to each transaction in the target state;

[0031] During the target record recovery process, the target record is verified, and when the target record verification is completed, the recovered record record is obtained.

[0032] According to some embodiments, the method further comprises:

[0033] Receive a structured query language SQL statement sent by a client, and read a target data page page from a file system according to the structured query language SQL statement;

[0034] Put the target data page page into a data page cache pool, and when it is determined that the target data page page exists in the data page cache pool, perform a redolog-data page refresh operation and a linked list update operation;

[0035] Determine the record record to be read corresponding to the target data page page, and determine the target record record according to the processing status corresponding to the suspended transaction.

[0036] According to some embodiments, the method further comprises:

[0037] Read the redolog file closest to the current time from the redolog files in the file system;

[0038] Parse the redolog file closest to the current time to obtain the data page number of the redolog file closest to the current time;

[0039] The data page page number is stored, and the redolog file corresponding to the same data page page number in the file system is stored in a one-way linked list corresponding to the same data page page number;

[0040] The target data page page is read from the data page page file of the file system, and then the one-way linked lists are queried according to the target data page pag to obtain and use the redolog corresponding to the target data page pag.

[0041] According to a second aspect of an embodiment of the present disclosure, there is provided a data recovery device, including:

[0042] A log loading unit, configured to load a physical log redo log file that is closest to a current time from among physical log redo log files in a file system when a database system fails and a startup instruction for the database system is received, wherein the database system includes a database node;

[0043] a data page modification unit, configured to read data page pages corresponding to each physical log redo log file in at least one physical log redo log file from a disk to a memory according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, modify the data page pages in the memory according to each physical log redo log file, and rewrite the modified data page pages back to the disk;

[0044] a data recovery unit, configured to recover the data dictionary metadata and obtain the recovered data dictionary metadata when the modified data page page and the undo log undo log meet the recovery requirements, wherein the undo log undo log includes the record record in the modified data page page;

[0045] The data recovery unit is further used to recover the record record when the modified data page page and the undo log undo log meet the recovery requirements, and obtain the recovered record record.

[0046] According to a third aspect of an embodiment of the present disclosure, a network device is provided, including:

[0047] processor;

[0048] a memory for storing instructions executable by the processor;

[0049] The processor is configured to execute the instructions to implement the data recovery method described in any one of the aforementioned aspects.

[0050] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided. When instructions in the storage medium are executed by a processor of a network device, the network device is enabled to execute the data recovery method described in any one of the preceding aspects.

[0051] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program implements any one of the methods described in the preceding aspects when executed by a processor.

[0052] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0053] In some or related embodiments, during failure recovery of a database system and when a startup instruction for the database system is received, a physical log redo log file in a file system that is closest to a current time is loaded, wherein the database system includes a database node; according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file that is closest to the current time, a data page page corresponding to each physical log redo log file in the at least one physical log redo log file is read from a disk to a memory, and the data page page is modified in the memory according to each physical log redo log file, and the modified data page page is rewritten back to the disk; when the modified data page page and undo log meet recovery requirements, data dictionary metadata is restored to obtain restored data dictionary metadata, wherein the undo log includes a record record in the modified data page page; when the modified data page page and undo log meet recovery requirements, the record record is restored to obtain a restored record record. Therefore, in a database system with only a single host and no spare nodes, the database instance can be recovered, that is, the data page, data dictionary metadata, and record can be recovered to a version consistent with that before the failure occurred. There is no need for a high-availability database system, and there is no need to increase the database hardware cost. The data page, data dictionary metadata, and record can be recovered according to the data recovery order, which can reduce omissions during recovery and improve the data recovery speed while significantly reducing the implementation hardware cost of the database system.

[0054] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0056] Figure 1 is a flow chart of a data recovery method according to an exemplary embodiment;

[0057] Figure 2is a flow chart of a data recovery method according to an exemplary embodiment;

[0058] Figure 3 is a schematic diagram showing an example of the internal structure of a database system according to an exemplary embodiment;

[0059] Figure 4 is a schematic diagram showing an example of a full data synchronization method according to an exemplary embodiment;

[0060] Figure 5 is a flow chart of a data recovery method according to an exemplary embodiment;

[0061] Figure 6 is a block diagram of a data recovery device according to an exemplary embodiment;

[0062] Figure 7 The invention is a block diagram of a network device according to an exemplary embodiment. DETAILED DESCRIPTION

[0063] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0064] The embodiments of the present disclosure provide data recovery methods and devices. In some embodiments, the data recovery method and information processing method, communication method and other terms can be interchangeable, the data recovery device and information processing device, communication device and other terms can be interchangeable, and the information processing system and communication system and other terms can be interchangeable.

[0065] The embodiments of the present disclosure are not exhaustive, but are only illustrative of some embodiments, and are not intended to be a specific limitation on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a certain embodiment can also be implemented as an independent embodiment, and the order of the steps in a certain embodiment can be arbitrarily exchanged. In addition, the optional implementation methods in a certain embodiment can be arbitrarily combined; in addition, the embodiments can be arbitrarily combined, for example, some or all of the steps of different embodiments can be arbitrarily combined, and a certain embodiment can be arbitrarily combined with the optional implementation methods of other embodiments.

[0066] In each embodiment of the present disclosure, unless otherwise specified or there is a logical conflict, the terms and / or descriptions between the embodiments are consistent and can be referenced to each other, and the technical features in different embodiments can be combined to form a new embodiment based on their internal logical relationships.

[0067] The terms used in the embodiments of the present disclosure are only for the purpose of describing specific embodiments and are not intended to limit the present disclosure.

[0068] In the embodiments of the present disclosure, unless otherwise specified, elements expressed in the singular form, such as "a", "an", "the", "above", "said", "aforementioned", "this", etc., may mean "one and only one", or "one or more", "at least one", etc. For example, when using articles such as "a", "an", "the" in English in translation, the noun after the article may be understood as a singular expression or a plural expression.

[0069] In the embodiments of the present disclosure, “plurality” refers to two or more.

[0070] In some embodiments, the terms “at least one,” “one or more,” “a plurality of,” “multiple,” etc. may be used interchangeably.

[0071] The prefixes such as "first" and "second" in the embodiments of the present disclosure are only used to distinguish different description objects, and do not constitute restrictions on the position, order, priority, quantity or content of the description objects. The statement of the description object refers to the description in the context of the claims or embodiments, and should not constitute unnecessary restrictions due to the use of prefixes. For example, if the description object is a "field", the ordinal number before the "field" in the "first field" and the "second field" does not limit the position or order between the "fields", and the "first" and "second" do not limit whether the "fields" they modify are in the same message, nor do they limit the order of the "first field" and the "second field". For another example, if the description object is a "level", the ordinal number before the "level" in the "first level" and the "second level" does not limit the priority between the "levels". For another example, the number of description objects is not limited by the ordinal number, and can be one or more. Taking the "first device" as an example, the number of "devices" can be one or more. In addition, the objects modified by different prefixes may be the same or different. For example, if the description object is "device", then the "first device" and the "second device" may be the same device or different devices, and their types may be the same or different. For another example, if the description object is "information", then the "first information" and the "second information" may be the same information or different information, and their contents may be the same or different.

[0072] In some embodiments, "terminal" or "terminal device" can be referred to as "user equipment (UE)", "user terminal" "mobile station (MS)", "mobile terminal (MT)", subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless terminal, remote terminal, handset, user agent, mobile client, client, etc.

[0073] In some embodiments, data, information, etc. may be obtained with the user's consent.

[0074] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0075] Figure 1 is a flow chart of a data recovery method according to an exemplary embodiment. Figure 1 As shown, the data recovery method can be used in a scenario where, when abnormal information appears in the system, the root cause information corresponding to the abnormal information is determined, and includes the following steps:

[0076] In step S11, when a database system is recovered from a failure and a startup instruction for the database system is received, a physical log redo log file closest to the current time in the physical log redo log files in the file system is loaded, wherein the database system includes a database node;

[0077] According to some embodiments, the execution subject of the embodiments of the present disclosure is a network device. The name of the network device is not limited. For example, the network device can also be called a server, or a server cluster, etc.

[0078] According to some embodiments, the database system may be, for example, a system consisting of a database and its management software. The database system may be, for example, a high-availability database system, such as a MySQL master cluster, Oracle Data Guard, etc. Among them, the database system of the embodiment of the present disclosure may, for example, include only one database node, and the database node is separated from computing and storage. The computing layer may use a single cloud host or docker, and the storage layer is distributed storage (such as a distributed file system or distributed block storage).

[0079] In some embodiments, the startup instruction may be, for example, an instruction received after the database system fault is repaired and used to start the database system. The startup instruction does not specifically refer to a fixed instruction. The startup instruction may be, for example, a voice startup instruction, a timed startup instruction, or a click startup instruction.

[0080] According to some embodiments, the file system is, for example, a file subsystem of an operating system. The data page page of the database will eventually be persistently stored in the data page page file of the file system. The file system is an important mechanism for organizing, storing and accessing data in a computer. It is an interface between the operating system and storage devices (such as hard disks, flash memories, etc.), providing users and applications with a unified and abstract way to access and manage data. There are many types of file systems, each with its own characteristics and applicable scenarios. For example, FAT (File Allocation Table) is commonly used in Windows systems and has the characteristics of simplicity, easy implementation and good compatibility.

[0081] In some embodiments, the physical log redo log file may be generated by the database system for each modification operation of a data page, and the physical log redo log file may record the modification information of the transaction on the data page. The physical log redo log file may include at least one physical log redo log file. The physical log redo log file does not specifically refer to a fixed file. For example, when the number of physical log redo log files included in the physical log redo log file changes, the physical log redo log file may also change accordingly. For example, when the modification operation on the data page page changes before the failure occurs, the physical log redo log file may also change accordingly.

[0082] According to some embodiments, Redo Log is an important log mechanism in the database system, which is mainly used to record all operations that modify the database, such as insert, update and delete, to ensure the integrity and consistency of the data. In relational databases such as MySQL, Redo Log plays a key role, including: ensuring the persistence of transactions: when a database failure occurs, for example, when the dirty pages (dirty data) in the buffer pool have not been written to the disk, the data records that need to be replayed to the disk file can be found through Redo Log, thereby restoring the data to the state before the failure, which can increase the speed of transaction submission. In some embodiments, refreshing the data in the buffer pool directly to the disk is a random IO operation with low efficiency, so recording the data in the buffer pool to the Redo Log is a sequential IO operation, which can significantly increase the speed of transaction submission.

[0083] According to some embodiments, the physical log redo log file closest to the current time may refer to, for example, a physical log redo log file corresponding to the last modification of the data page.

[0084] In some embodiments, when a database system fails and a startup instruction for the database system is received, a physical log redo log file closest to the current time in the physical log redo log files in the file system is loaded, wherein the database system includes a database node.

[0085] In step S12, according to the scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, the data page page corresponding to each physical log redo log file in the at least one physical log redo log file is read from the disk to the memory, and the data page page is modified in the memory according to each physical log redo log file, and the modified data page page is rewritten back to the disk;

[0086] According to some embodiments, the scanning order may be used to indicate the order in which at least one physical log redo log file is scanned. The scanning order does not specifically refer to a fixed order. For example, the scanning order may be scanning from the old to the new starting from the checkpoint, or scanning from the new to the old starting from the checkpoint.

[0087] In some examples, a disk may be used to refer to a memory that stores data using magnetic recording technology. In the embodiment of the present disclosure, the redo log must be flushed to the disk when the transaction is committed.

[0088] According to some embodiments, the memory may be used, for example, to store at least one physical log redo log file.

[0089] In some embodiments, according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, a data page page corresponding to each physical log redo log file in the at least one physical log redo log file is read from a disk to a memory, and the data page page is modified in the memory according to each physical log redo log file, and the modified data page page is written back to the disk.

[0090] In step S13, when the modified data page page and the undo log undo log meet the recovery requirement, the data dictionary metadata is restored to obtain the restored data dictionary metadata, wherein the undo log undo log includes the record record in the modified data page page;

[0091] According to some embodiments, the undo log may be, for example, a corresponding undo log generated when a database system inserts, modifies, or deletes each record. The undo log does not specifically refer to a fixed undo log. For example, when a record changes, the undo log may also change accordingly.

[0092] In some embodiments, the recovery requirement may be, for example, a condition for determining whether the data dictionary metadata can be repaired before the data dictionary metadata is repaired. The recovery requirement may be, for example, that the modified data page page is consistent with the undo log undo log recovery.

[0093] In some embodiments, the data dictionary is used to store metadata information of tables / indexes. The metadata information includes, but is not limited to, the definition of tables / indexes, the correspondence between tables / indexes and internal data structures (B+ trees, table spaces, files, etc.), globally unique IDs, etc. Among them, the data managed and maintained by the data dictionary is the library table index metadata. The metadata itself exists in the form of records and is stored in pages.

[0094] According to some embodiments, a data dictionary is a collection of definitions and descriptions of data items, data structures, data flows, data storage, processing logic, etc., and its purpose is to provide a detailed description of each element in a data flow diagram. Alternatively, a data dictionary is a collection of information describing data, which is a collection of definitions of all data elements used in the system. When a database system fails, in order to ensure that the metadata of the data dictionary is consistent with the time of failure and to maintain the integrity of the transaction level, recovery is usually performed in combination with data pages and undo logs.

[0095] In some embodiments, the metadata itself exists in the form of records and is stored in pages. If the consistency recovery of records and pages can be achieved in the record module and the page module, the recovery of metadata can be achieved. Therefore, the problem of data dictionary failure recovery can be reduced to the problem of record and page failure recovery.

[0096] According to some embodiments, when the modified data page page and undo log undo log meet the recovery requirements, the data dictionary metadata is restored to obtain the restored data dictionary metadata, wherein the undo log undo log includes the record record in the modified data page page.

[0097] In step S14, when the modified data page page and undo log undo log meet the recovery requirement, the record record is restored to obtain the restored record record.

[0098] According to some embodiments, when a database system fails, in order to ensure that the record is consistent with the time of the failure and to maintain the integrity of the transaction level, recovery can be performed in combination with data pages and undo logs.

[0099] In some embodiments, when the modified data page page and the undo log undo log meet the recovery requirement, the record record is restored to obtain the restored record record.

[0100] In some or related embodiments, during failure recovery of a database system and when a startup instruction for the database system is received, a physical log redo log file in a file system that is closest to a current time is loaded, wherein the database system includes a database node; according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file that is closest to the current time, a data page page corresponding to each physical log redo log file in the at least one physical log redo log file is read from a disk to a memory, and the data page page is modified in the memory according to each physical log redo log file, and the modified data page page is rewritten back to the disk; when the modified data page page and undo log meet recovery requirements, data dictionary metadata is restored to obtain restored data dictionary metadata, wherein the undo log includes a record record in the modified data page page; when the modified data page page and undo log meet recovery requirements, the record record is restored to obtain a restored record record. Therefore, in a database system with only a single host and no spare nodes, the database instance can be recovered, that is, the data page, data dictionary metadata, and record can be recovered to a version consistent with that before the failure occurred. There is no need for a high-availability database system, and there is no need to increase the database hardware cost. The data page, data dictionary metadata, and record can be recovered according to the data recovery order, which can reduce omissions during recovery and improve the data recovery speed while significantly reducing the implementation hardware cost of the database system.

[0101] Figure 2 is a flow chart of a data recovery method according to an exemplary embodiment. Figure 2 As shown, the data recovery method can be used in a data recovery scenario, including the following steps:

[0102] In step S21, when a database system is recovered from a failure and a startup instruction for the database system is received, a physical log redo log file closest to the current time in the physical log redo log files in the file system is loaded, wherein the database system includes a database node;

[0103] According to some embodiments, the execution subject of the embodiments of the present disclosure is a network device. The name of the network device is not limited. For example, the network device can also be called a server, or a server cluster, etc.

[0104] In some embodiments, Figure 3 is a schematic diagram showing an example of the internal structure of a database system according to an exemplary embodiment. Figure 3 As shown, the internal structure of the disk storage-based database system may include, for example:

[0105] 1. SQL module: receives SQL statements sent by the client, parses them into a syntax tree, translates them into an execution plan tree, and finally sends them to the executor to execute the execution plan;

[0106] 2.Record module: When the SQL module executor executes SQL statements, it will call the record module to read and write the records that the SQL statements need to process. The main task of the record module is to perform a series of management and maintenance operations around the record data object;

[0107] 3. Page module: In the database system, records are stored in pages. When the record module needs to read a record from the disk, it will request the record containing the page from the page module. The main task of the page module is to perform a series of management and maintenance operations around the page data object;

[0108] 4. File system: The file system is the file subsystem of the operating system. The database pages will eventually be persistently stored in the page files of the file system;

[0109] 5. Data dictionary: stores metadata information of tables / indexes. Such metadata information includes but is not limited to: table / index definitions, correspondence between tables / indexes and internal data structures (B+ trees, tablespaces, files, etc.); globally unique IDs, etc.

[0110] 6. Transaction module: Transaction is an important function of the database system. After a transaction is committed in the database system, the database system must ensure that all modification operations of the transaction on the data in the database system are completed. It is not allowed that some modification operations are completed and some operations are not completed. After a transaction is rolled back in the database system, the database system must ensure that all modification operations of the transaction on the database system are completely undone. It is not allowed that some operations are not undone and some operations are undone. At the same time, the modification operations of the transaction on the database system must be persisted to the file system.

[0111] In the database system, the transaction module is responsible for ensuring the atomicity and persistence of transactions. The atomicity and persistence of transactions are achieved through two mechanisms:

[0112] Mechanism 1: redo log mechanism for page module: The database system generates a corresponding redo log for each page modification operation. The redo log records the transaction's modification of the page. For example, if a transaction performs a +1 operation on a field of a record in a page, the generated redo log records the result after the field is +1. When the transaction is committed, the database system will flush the redo log to the file system to ensure that the transaction's modifications to the page are persisted in the file system.

[0113] Mechanism 2: Undo log mechanism for record module: When the database system inserts, modifies, or deletes each record, it generates a corresponding undo log. The undo log records the version of the record before the transaction operation. For example, if a transaction performs a +1 operation on a field of a record, an undo log is generated to record the result before the field is +1. When the transaction is committed, these undo logs will be deleted; when the transaction is rolled back, the database system will use these undo logs to roll back the modified record and undo the modification of the record by the transaction.

[0114] Undo log is also stored in page. The page storing undo log is called undologpage. So there are two kinds of pages in the database system, one is recordpage and the other is undo logpage. Both pages can be restored by redo log. Unless otherwise specified, these two pages are referred to as page.

[0115] Among them, in the fault recovery operation when the database system is restarted after a fault, since the data of the SQL module are cache data that does not need to be persisted, there is no need to restore the SQL module data during fault recovery.

[0116] According to some embodiments, the latest redo log can be loaded from the redo log file of the file system, so that the redo log in the memory can be consistent with the time of the failure. The latest redo log can be, for example, the redo log closest to the current time.

[0117] In some embodiments, the generation and release process of Redo Log may include, for example: as the transaction operation is executed, Redo Log may be generated, and when the transaction is committed, the generated Redo Log will be written to the Log Buffer instead of being written to the disk file immediately. After the dirty page of the transaction operation is written to the disk, the mission of Redo Log is completed, and the space occupied by it can be overwritten, that is, reused.

[0118] According to some embodiments, the working principle of Redo Log is to achieve transaction persistence. When a database failure occurs, redo is performed based on Redo Log to ensure that data not written to the disk is persisted. The content of the Redo Log file is written to the file in a sequential loop. When the file is full, it will go back to the first file for overwriting.

[0119] In some embodiments, the latest redo log is loaded from the redo log file of the file system to ensure that the redo log record in the memory is consistent with the state when the failure occurs. This is the basis of the recovery process because only by knowing which modifications have not been applied to the data file can the data be correctly recovered.

[0120] In step S22, a scan start point corresponding to the physical log redo log file is obtained, wherein the scan start point is a checkpoint corresponding to the physical log redo log file closest to the current time;

[0121] The specific process is as described above and will not be repeated here.

[0122] According to some embodiments, the scanning start point corresponding to the physical log redo log file can be obtained, wherein the scanning start point can also be referred to as the scanning starting point, or the last checkpoint. The checkpoint is the time point when InnoDB regularly refreshes the dirty pages (pages that have been modified but not yet written to the disk) in the memory to the disk. Among them, when starting after a system crash, it is necessary to start scanning from a certain position before the crash, which can rely on the log position recorded by the crash recovery mechanism.

[0123] According to some embodiments, a scan start point corresponding to a physical log redo log file is obtained, wherein the scan start point is a checkpoint corresponding to a physical log redo log file that is closest to a current time.

[0124] In step S23, the physical log redo log file is scanned according to the scanning point and the scanning order, and in the scanning process, when a physical log redo log file that has not been scanned is scanned in the physical log redo log file, the log sequence number LSN is updated, wherein the scanning order is the order from far to near from the current time;

[0125] The specific process is as described above and will not be repeated here.

[0126] According to some embodiments, scanning the redo log is a key step in the database recovery process, especially when using a storage engine such as MySQL's InnoDB. The redo log is written cyclically and contains physical modifications to the database, and is used to recover data page changes that are not committed to disk when the system crashes.

[0127] In some embodiments, redo logs may be stored in files on disk, for example, and these files may be opened and read. Depending on the configuration of InnoDB, there may be multiple redo log files, which form a log group that is written in a loop.

[0128] In some embodiments, the redo log is sorted by log sequence number (LSN), where LSN is a monotonically increasing number that indicates the position of the log entry in the log file. During the scanning process, modifications can be applied in the order of LSN to ensure the order of operations. If any discontinuous or corrupted log entries are encountered, error handling or recovery strategies can be implemented.

[0129] In step S24, when the scanning operation satisfies the scanning completion condition, it is determined that the at least one physical log redo log file is acquired, wherein the at least one physical log redo log file is used to recover the data page page;

[0130] The specific process is as described above and will not be repeated here.

[0131] In step S25, the data page corresponding to each physical log redo log file in the at least one physical log redo log file is read from the disk to the memory;

[0132] The specific process is as described above and will not be repeated here.

[0133] According to some embodiments, reading data pages corresponding to each physical log redo log file in the at least one physical log redo log file from a disk to a memory includes:

[0134] According to the scanning start point corresponding to the physical log redo log file, determine that the modification information of the data page page exists in the at least one physical log redo log file record, and in the scanning process, when the physical log redo log file that is not scanned is scanned in the physical log redo log file, update the log sequence number LS N;

[0135] According to the modification information, a data page corresponding to the modification information is read from a data file on the disk to the memory.

[0136] In some embodiments, during the scanning process, the currently processed LSN can be tracked. When new log entries or unscanned log entries are scanned, the current LSN is updated, which can improve the accuracy of determining the completion time of the scan, where the completion time of the scan can be, for example, reaching the end of the log file or reaching a specific recovery point.

[0137] In some embodiments, when the scan reaches the end of the log file or reaches a specific recovery point, the scan can be considered complete. At this point, it may be necessary to perform a checkpoint to refresh the dirty pages in the memory to the disk and update the position of the checkpoint.

[0138] According to some embodiments, the method further comprises:

[0139] Acquire the at least one physical log redo log file by adopting a multi-threading mode or a multi-process mode;

[0140] or,

[0141] Pre-reading a data page page in the at least one physical log redo log file record;

[0142] or;

[0143] The at least one physical log redo log file is compressed to obtain the at least one processed physical log redo log file. The at least one physical log redo log file is obtained by adopting a multi-threading method or a multi-process method, and the physical log redo log file can be processed in parallel, which can improve the recovery speed. The embodiment of the present disclosure does not limit the multi-threading method or the multi-process method.

[0144] According to some embodiments, in each of the above-mentioned execution processes, the data page page in the at least one physical log redo log file record can be pre-read, which can reduce random I / O operations and improve recovery efficiency. There is no limitation on the pre-reading method, that is, any pre-reading operation can be implemented.

[0145] According to some embodiments, the at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file, which can increase the compression of the redo log and reduce storage space and I / O overhead, wherein the specific compression method is not limited.

[0146] According to some embodiments, Figure 4 is an exemplary schematic diagram showing a method for synchronizing full data according to an exemplary embodiment. Figure 4 As shown in the figure, full data synchronization can be performed in the initial stage of redo log file recovery, or when incremental synchronization is interrupted or has data errors. Incremental synchronization will stop during full synchronization. The source uses Redis's own mechanism to package the data into files and then transmit them to the synchronization agent component of the destination segment. After receiving the full data, the synchronization agent component first writes the data to a separate Redis database (i.e., back buffer). The Redis database currently being used by the business is called the front buffer. The access addresses of the two are exchanged to complete the full synchronization process. During the incremental synchronization process, the synchronization agent component will disguise itself as a slave node on the source side, receive the incremental modified data after the full synchronization, and then write it to the destination side.

[0147] In step S26, the data page page is modified in the memory according to the physical log redo log files, and the modified data page page is rewritten back to the disk;

[0148] The specific process is as described above and will not be repeated here.

[0149] According to some embodiments, the corresponding page may be read from the disk to the memory according to the record in the redo log, and the corresponding page may be modified in the memory according to the content of the redo log.

[0150] According to some embodiments, when the database system recovers data from the redo log, it reads and modifies the data page according to the records in the redo log. Specifically, for example, it may include:

[0151] 1. Start the recovery process: You can determine which redo log file and which position to start recovery from, such as the checkpoint position written during the last normal shutdown, or the last redo log position written when the system crashed.

[0152] 2. Open and read redo log files: You can open the physical file containing the redo log and read the records from a certain starting point.

[0153] 3. Parse the redo log record to obtain the modified data pages and modification content: For each redo log record read, it can be parsed to determine which data pages (pages) have been modified and the specific modification content on the data page. The parsed information usually includes the tablespace ID, page number, modification type, and modification data.

[0154] 4. Read data pages from disk: Read the corresponding data pages from the data file on disk into memory according to the parsed tablespace ID and page number. If there is not enough space in memory to store these data pages, a page replacement strategy (such as the LRU algorithm) may be used to manage the data pages in memory.

[0155] 5. Modify data pages in memory: Once data pages are loaded into memory, they can be updated based on the modifications recorded in the redo log. For example, this may include modifying specific data on the page, updating the metadata of the page (such as the page status, transaction ID, etc.), or performing other necessary operations. The modified data pages become "dirty pages" in memory, meaning that their contents are different from those on disk and need to be flushed back to disk at some point in the future.

[0156] 6. Track and update LSN: When processing each redo log record, the currently processed LSN (Log Sequence Number) can be updated to ensure that modifications are applied in the correct order. LSN is a monotonically increasing sequence number in the redo log that is used to identify the order and position of log records.

[0157] 7. Loop processing: Continue to read and parse the next record in the redo log, and repeat the above steps 3-6 until all redo log records are processed or a certain recovery target is reached (such as recovery to a specific point in time).

[0158] 8. Complete the recovery process: When all redo log records have been processed, the recovery process is complete. At this point, the data page has been updated to the state at the time of the crash, and the database can be reopened for read and write operations.

[0159] According to some embodiments, the modified page is written back to disk to ensure that the data on the disk reflects the state before the failure. Since the undo log is usually stored in the data page, when the data page is restored, the undo log is automatically restored to the state before the failure.

[0160] In step S27, when the modified data page page and the undo log undo log meet the recovery requirement, the data dictionary metadata is restored to obtain the restored data dictionary metadata, wherein the undo log undo log includes the record record in the modified data page page;

[0161] The specific process is as described above and will not be repeated here.

[0162] According to some embodiments, the restoring the data dictionary metadata and obtaining the restored data dictionary metadata includes:

[0163] According to the recovery starting point, read the data page including the data dictionary metadata from the disk into the memory;

[0164] Using the records in the physical log redo log file, obtaining the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point;

[0165] The data dictionary metadata is restored according to the physical log redo log file and the change information, and during the data dictionary metadata restoration process, the data dictionary metadata is verified, and when the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

[0166] According to some embodiments, restoring the data dictionary metadata may include, for example:

[0167] 1. Determine the recovery starting point: You can determine the time point or log position from which to start recovery. For example, the recovery starting point can be determined by checkpoint information or the last successfully submitted log position.

[0168] 2. Read data pages: Read data pages containing data dictionary metadata from disk into memory. Data pages can include system tables, directory tables, indexes, etc.

[0169] 3. Apply redo log: Use the records in the redo log to reapply all changes made to the data dictionary metadata since the last checkpoint, that is, obtain change information, which can include insert, update, and delete operations. Through the redo log, the data dictionary metadata can be restored to a consistent state before the failure.

[0170] 4. Process unfinished transactions: Traverse the undo log and find transactions that were active (i.e., uncommitted) when the failure occurred. For each uncommitted transaction, undo its changes to the data dictionary metadata based on its undo log record. This ensures that only changes from committed transactions are retained.

[0171] 5. Verify data consistency: During the recovery process, you can perform verification steps to ensure the consistency of the data dictionary metadata. For example, this may include checking referential integrity, triggering constraint checks, etc.

[0172] 6. Complete recovery: Once all necessary changes have been applied and data consistency has been verified, the recovery process is complete. At this point, the data dictionary metadata should be consistent with the time of failure and maintain transaction-level integrity.

[0173] In step S28, when the modified data page page and undo log undo log meet the recovery requirement, the record record is restored to obtain the restored record record.

[0174] The specific process is as described above and will not be repeated here.

[0175] According to some embodiments, the recovering the record record and obtaining the recovered record record includes:

[0176] Read the data page including the target record from disk into memory;

[0177] Traverse the undo log and find the transactions that were in the target state when the failure occurred;

[0178] Undo the change information of the target record made by each transaction in the target state according to the undo log record corresponding to each transaction in the target state;

[0179] During the target record recovery process, the target record is verified, and when the target record verification is completed, the recovered record record is obtained.

[0180] Specifically, the process of recording the recovered record may include, for example:

[0181] 1. Read data page: Read the data page containing the target record from disk into memory.

[0182] 2. Apply redo log: If there are redo log records that have not been written to disk when the failure occurs, you can apply the redo log records that have not been written to disk to ensure that all changes of transactions that have been committed before the failure are reflected in the data pages.

[0183] 3. Process unfinished transactions: traverse the undo log to find transactions that were active (i.e., uncommitted) when the failure occurred. The target state may include, for example, an active state or an uncommitted state. For each uncommitted transaction, undo the changes made to the target record based on its undo log record, and roll back the record to the state before the transaction started.

[0184] 4. Verify data consistency: During the recovery process, verification steps can be performed to ensure the consistency and integrity of the records. For example, this can include checking foreign key constraints, triggers, etc.

[0185] 5. Complete recovery: Once all necessary changes have been applied and data consistency has been verified, the recovery process is complete. At this point, the target record should be consistent with the time of failure and maintain transactional integrity.

[0186] Different processes may be used for different database systems, that is, the order of step S21 to step S27 may not be fixed. Some database systems may provide more advanced recovery tools and mechanisms, such as point-in-time recovery or log-based recovery.

[0187] According to some embodiments, the method further comprises:

[0188] Receive a structured query language SQL statement sent by a client, and read a target data page page from a file system according to the structured query language SQL statement;

[0189] Put the target data page page into a data page cache pool, and when it is determined that the target data page page exists in the data page cache pool, perform a redolog-data page refresh operation and a linked list update operation;

[0190] Determine the record record to be read corresponding to the target data page page, and determine the target record record according to the processing status corresponding to the suspended transaction.

[0191] According to some embodiments, an SQL statement sent by a client to an SQL module can be received, the SQL module generates an execution plan for the SQL statement and executes it, the execution process calls the record module to obtain the record, and the record module calls the page module to obtain the page; because the database system has just been initialized, the page module does not have the page in the page cache pool, so the page module sends a request to the file system to read the page; the file system can return the old version of the page and put it into the page cache pool, when the page cache pool contains a new page put in by the file system, the redolog-page refresh device is started and a linked list update operation is performed, the page newly put into the page cache pool is refreshed by the redolog-page refresh device, and the refreshed latest version of the page is put into the page cache pool. The page module obtains the page in the page cache pool and returns it to the record module; the record module determines whether there are any pending transactions for the record to be read in the page. If not, it means that the record is the correct record and it can be returned directly to the SQL module; if there are any pending transactions, it means that the record is the latest version of the record. In this case, it is not returned to the SQL module. Finally, the record obtained by the SQL module is the latest and correct version of the record.

[0192] When there is a new page put into the file system in the page cache pool, the redolog-page refresh device is started and the linked list update operation is performed. Specifically, for example, the latest redolog is read from the redo log file in the file system, and the read redo log is parsed to obtain the page number of the redo log. The parsing result is saved, and the redologs of the same pageno are stored in the same one-way linked list. The old version, i.e., the historical version of the page is read from the page file of the file system, and each one-way linked list is queried according to the pageno to find and use the corresponding redolog to refresh the page to the latest version.

[0193] In one or related embodiments, a scan start point corresponding to a physical log redo log file is obtained, wherein the scan start point is a checkpoint corresponding to a physical log redo log file closest to a current time; a scan operation is performed on the physical log redo log file according to the scan point and the scan order, and during the scanning process, when a physical log redo log file that has not been scanned is scanned in the physical log redo log file, a log sequence number LSN is updated, wherein the scan order is an order from far to near from the current time; when the scan operation satisfies a scan completion condition, it is determined that at least one physical log redo log file is obtained, wherein the at least one physical log redo log file is used to recover a data page page; a data page page corresponding to each physical log redo log file in the at least one physical log redo log file is read from a disk to a memory, and the scan can be performed through the scan point and the scan order, thereby reducing the probability of data recovery loss and improving the integrity of data recovery. In addition, by updating the log sequence number LSN, reference modification can be performed in sequence, thereby improving the accuracy and speed of data recovery.

[0194] Figure 5 is a flow chart of a data recovery method according to an exemplary embodiment. Figure 5 As shown, the data recovery method can be used in the failure recovery process of a single-host database system. The data pages, records, and metadata of the database system are restored to be consistent with the failure time, and the external client reads data, including the following steps:

[0195] In step S31, the redolog file closest to the current time is read from the redolog files in the file system;

[0196] The specific related introduction has been mentioned above and will not be repeated here.

[0197] According to some embodiments, the redolog file closest to the current time may be read from the redolog files in the file system.

[0198] According to some embodiments, in a database system, a redo log file may be written cyclically, for example, when a log file is full, it may switch to the next log file to continue writing. The redo log file currently being written or most recently written may be determined according to the writing order. For example, this may be achieved by querying the status information of the database, reading a configuration file or a log file.

[0199] In some embodiments, a redolog file is opened using a file access API (such as a POSIX API, a Windows API, etc.) When the contents of the redo log are stored in a binary format, the file may be opened in a binary mode.

[0200] According to some embodiments, header information of a Redo Log entry may be read, wherein each Redo Log entry may include header information, including but not limited to the length, type, sequence number, etc. of the log entry. The header information may be read first to determine the parsing method of the log content.

[0201] In some embodiments, the redo log entry can be parsed: Based on the header information, the specific content of the redo log entry can be parsed. For example, binary data can be decoded to obtain information such as data page number, modification type, and modification data. The specific method of parsing depends on the definition of the redo log format by the database system. For example, the parsing method of the redo log entry can be determined by referring to relevant documents or source code.

[0202] According to some embodiments, Redo Log entries may be processed: Based on the parsed content of the redo log entries, corresponding operations may be performed to restore or redo the changes, which may include, for example, loading a data page into memory, modifying the content of a data page, and the like.

[0203] In some embodiments, error handling and verification may be performed: during the entire process, errors that occur may be handled, such as file read errors, parsing errors, etc. At the same time, it may also be verified whether the parsed redo log entries are valid and consistent.

[0204] In some embodiments, a redo log index may be constructed: before parsing the redo log, an index or data structure is required to quickly locate the redo log entry associated with a specific page number. This index may be based on the log sequence number (LSN), because the redo log is usually written in the order of LSN, and each page change has a corresponding LSN range.

[0205] In some embodiments, closing the file: After completing reading and parsing the redo log file, the file is closed to release resources.

[0206] In step S32, the redolog file closest to the current time is parsed to obtain the data page number (pageno) of the redolog closest to the current time;

[0207] According to some embodiments, the redolog file closest to the current time is parsed to obtain the data page number of the redolog closest to the current time. That is, the latest redolog file can be parsed to obtain the data page number of the redolog. The data page number is used to identify a specific data page in the database.

[0208] According to some embodiments, the Log Header can be located, wherein the redo log file has one or more header parts, which include metadata information about the log file. The header information can be located and parsed first to obtain the overall structure and format of the log file.

[0209] According to some embodiments, parsing log entries: Redo log files consist of a series of log entries, each of which records one or more changes to a data page. The log entries can be traversed and parsed to obtain relevant page numbers and other information.

[0210] According to some embodiments, parsing page numbers: When parsing log entries, the page number field can be found and extracted. The specific location and format of this field depends on the format of the redo log. Generally speaking, the page number is stored in a fixed byte length and may be accompanied by other related metadata (such as block size, tablespace ID, etc.).

[0211] According to some embodiments, variable length and encoding are handled: In some cases, the page number or other fields may be stored in a variable length or specific encoding. This encoding can be understood and handled to ensure that the page number is parsed correctly.

[0212] According to some embodiments, verification and testing: When writing parsing code, it is recommended to write some test cases to verify whether the parsing logic is correct. You can use known data and scenarios to test the code and check whether the results meet expectations.

[0213] Specifically, a special tool or API provided by the database system is used to parse the redo log file. Since the tool used has been optimized and tested, the accuracy of parsing the redo log file and extracting relevant information can be ensured.

[0214] In step S33, the data page page number is stored, and the redolog file corresponding to the same data page page number in the file system is stored in a one-way linked list corresponding to the same data page page number;

[0215] In step S34, the target data page page is read from the data page page file of the file system, and then the one-way linked lists are queried according to the target data page pag to obtain and use the redolog corresponding to the target data page pag.

[0216] According to some embodiments, after all relevant redo log changes have been applied, the consistency and integrity of the data page can be verified. If the data page is valid, it can be written back to the target location in the data file or memory for subsequent use.

[0217] In one or related embodiments, the redolog file closest to the current time is read from the redolog file in the file system; the redolog file closest to the current time is parsed to obtain the data page page number of the redolog closest to the current time; the data page page number is stored, and the redolog file corresponding to the same data page page number in the file system is stored in a one-way linked list corresponding to the same data page page number; the target data page page is read from the data page page file of the file system, and then the one-way linked lists are queried according to the target data page pag, and the redolog corresponding to the target data page pag is obtained and used. Therefore, after the page, record and metadata in the database system are restored, any page read from the file system can be refreshed to the latest version through the redo log corresponding to the stored page, and put into the page cache pool. Since the page version is already the latest at this time, it can be provided to the upper-level business logic or external client for access, and the data and status at the time of the main node failure can be continued, so that services can be provided to the outside world and the service quality can be improved.

[0218] A block diagram of a data recovery device is shown according to an exemplary embodiment. Figure 6 , the device 600 comprises:

[0219] The log loading unit 601 is used to load the physical log redo log file closest to the current time in the physical log redo log files in the file system when the database system recovers from a failure and a startup instruction for the database system is received, wherein the database system includes a database node;

[0220] The data page modification unit 602 is used to read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory according to the scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, and modify the data page page in the memory according to each physical log redo log file, and rewrite the modified data page page back to the disk;

[0221] The data recovery unit 603 is used to recover the data dictionary metadata and obtain the recovered data dictionary metadata when the modified data page page and the undo log undo log meet the recovery requirements, wherein the undo log undo log includes the record record in the modified data page page;

[0222] The data recovery unit 603 is further configured to recover the record record and obtain the recovered record record when the modified data page page and the undo log undo log meet the recovery requirement.

[0223] According to some embodiments, the data page modification unit 602 is used to read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory according to the scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, specifically to:

[0224] Obtain a scan start point corresponding to a physical log redo log file, wherein the scan start point is a checkpoint corresponding to a physical log redo log file closest to the current time;

[0225] According to the scanning point and the scanning order, the physical log redo log file is scanned, and in the scanning process, when a physical log redo log file that has not been scanned is scanned in the physical log redo log file, a log sequence number LSN is updated, wherein the scanning order is the order from far to near from the current time;

[0226] In a case where the scanning operation satisfies the scanning completion condition, determining to obtain the at least one physical log redo log file, wherein the at least one physical log redo log file is used to recover the data page page;

[0227] Read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory.

[0228] According to some embodiments, the data page modification unit 602 is used to read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory, specifically to:

[0229] According to the scanning start point corresponding to the physical log redo log file, determine that the modification information of the data page page exists in the at least one physical log redo log file record, and in the scanning process, when the physical log redo log file that is not scanned is scanned in the physical log redo log file, update the log sequence number LS N;

[0230] According to the modification information, a data page corresponding to the modification information is read from a data file on the disk to the memory.

[0231] According to some embodiments, the data page modification unit 602 is further specifically configured to:

[0232] Acquire the at least one physical log redo log file by adopting a multi-threading mode or a multi-process mode;

[0233] or,

[0234] Pre-reading a data page page in the at least one physical log redo log file record;

[0235] or;

[0236] The at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file.

[0237] According to some embodiments, the data recovery unit 603 is used to recover the data dictionary metadata, and when obtaining the recovered data dictionary metadata, it is specifically used to:

[0238] According to the recovery starting point, read the data page including the data dictionary metadata from the disk into the memory;

[0239] Using the records in the physical log redo log file, obtaining the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point;

[0240] The data dictionary metadata is restored according to the physical log redo log file and the change information, and during the data dictionary metadata restoration process, the data dictionary metadata is verified, and when the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

[0241] According to some embodiments, the data recovery unit 603 is used to recover the record record, and when obtaining the recovered record record, is specifically used to:

[0242] Read the data page including the target record from disk into memory;

[0243] Traverse the undo log and find the transactions that were in the target state when the failure occurred;

[0244] Undo the change information of the target record made by each transaction in the target state according to the undo log record corresponding to each transaction in the target state;

[0245] During the target record recovery process, the target record is verified, and when the target record verification is completed, the recovered record record is obtained.

[0246] According to some embodiments, the data recovery unit 603 is further specifically configured to:

[0247] Receive a structured query language SQL statement sent by a client, and read a target data page page from a file system according to the structured query language SQL statement;

[0248] Put the target data page page into a data page cache pool, and when it is determined that the target data page page exists in the data page cache pool, perform a redolog-data page refresh operation and a linked list update operation;

[0249] Determine the record record to be read corresponding to the target data page page, and determine the target record record according to the processing status corresponding to the suspended transaction.

[0250] According to some embodiments, the data recovery unit 603 is further configured to:

[0251] Read the redolog file closest to the current time from the redolog files in the file system;

[0252] Parse the redolog file closest to the current time to obtain the data page number of the redolog file closest to the current time;

[0253] The data page page number is stored, and the redolog file corresponding to the same data page page number in the file system is stored in a one-way linked list corresponding to the same data page page number;

[0254] The target data page page is read from the data page page file of the file system, and then the one-way linked lists are queried according to the target data page pag to obtain and use the redolog corresponding to the target data page pag.

[0255] Regarding the device in the above embodiment, the specific way in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here. The names of each virtual device are not limited, for example, the log loading unit can also be called a reading module.

[0256] In some or related embodiments, a log loading unit is used to load the physical log redo log file in the file system that is closest to the current time when a database system fails and a startup instruction for the database system is received, wherein the database system includes a database node; a data page modification unit is used to read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory according to the scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file that is closest to the current time, and modify the data page page in the memory according to the physical log redo log files, and rewrite the modified data page page back to the disk; a data recovery unit is used to restore the data dictionary metadata when the modified data page page and the undo log meet the recovery requirements, and obtain the restored data dictionary metadata, wherein the undo log includes the record record in the modified data page page; the data recovery unit is also used to restore the data dictionary metadata between the modified data page page and the undo log. When the log meets the recovery requirements, the record is restored to obtain the restored record. Therefore, in a database system with only a single host and no standby node, the database instance can be restored, that is, the data page, data dictionary metadata, and record can be restored to a version consistent with that before the failure. There is no need for a high-availability database system, and there is no need to increase the database hardware cost. The data page, data dictionary metadata, and record can be restored according to the data recovery order, which can reduce omissions during recovery and improve the data recovery speed while significantly reducing the implementation hardware cost of the database system.

[0257] Figure 7 700 is a block diagram of a network device 700 provided in an embodiment of the present disclosure. For example, the network device 700 may be provided as a network device. Figure 7 The network device 700 includes a processing component 722, which further includes at least one processor, and a memory resource represented by a memory 732 for storing instructions executable by the processing component 722, such as an application. The application stored in the memory 732 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute instructions to perform any of the methods described above in the network device.

[0258] The network device 700 may also include a power supply component 727 configured to perform power management of the network device 700, a wired or wireless network interface 750 configured to connect the network device 700 to a network, and an input / output (I / O) interface 758. The network device 700 may operate based on an operating system stored in the memory 732, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM or the like.

[0259] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0260] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0261] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0262] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0263] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0264] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0265] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0266] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A data recovery method, characterized in that: include: When a database system is recovering from a failure and a startup instruction for the database system is received, loading a physical log redo log file closest to a current time from among physical log redo log files in a file system, wherein the database system includes a database node; According to the scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory, modify the data page page in the memory according to each physical log redo log file, and rewrite the modified data page page back to the disk; When the modified data page page and the undo log undo log meet the recovery requirement, the data dictionary metadata is restored to obtain the restored data dictionary metadata, wherein the undo log undo log includes the record record in the modified data page page; When the modified data page page and undo log undo log meet the recovery requirements, restore the record record d to obtain the restored record record.

2. The method according to claim 1, characterized in that The step of reading data pages corresponding to each physical log redo log file in at least one physical log redo log file from a disk to a memory according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time includes: Obtain a scan start point corresponding to a physical log redo log file, wherein the scan start point is a checkpoint corresponding to a physical log redo log file closest to the current time; According to the scanning point and the scanning order, the physical log redo log file is scanned, and in the scanning process, when a physical log redo log file that has not been scanned is scanned in the physical log redo log file, a log sequence number LSN is updated, wherein the scanning order is the order from far to near from the current time; In a case where the scanning operation satisfies the scanning completion condition, determining to obtain the at least one physical log redo log file, wherein the at least one physical log redo log file is used to recover the data page page; Read the data page page corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory.

3. The method according to claim 1 or 2, characterized in that: The step of reading data pages corresponding to each physical log redo log file in the at least one physical log redo log file from the disk to the memory includes: According to the scanning start point corresponding to the physical log redo log file, determine that the modification information of the data page page exists in the at least one physical log redo log file record, and in the scanning process, when the physical log redo log file that is not scanned is scanned in the physical log redo log file, update the log sequence number LSN; According to the modification information, a data page corresponding to the modification information is read from a data file on the disk to the memory.

4. The method according to claim 3, characterized in that The method further comprises: Acquire the at least one physical log redo log file by adopting a multi-threading mode or a multi-process mode; or, Pre-reading a data page page in the at least one physical log redo log file record; or; The at least one physical log redo log file is compressed to obtain at least one processed physical log redo log file.

5. The method according to claim 1, characterized in that The step of restoring the data dictionary metadata and obtaining the restored data dictionary metadata includes: According to the recovery starting point, read the data page including the data dictionary metadata from the disk into the memory; Using the records in the physical log redo log file, obtaining the change information corresponding to the data dictionary metadata between the last recovery starting point and the recovery starting point; The data dictionary metadata is restored according to the physical log redo log file and the change information, and during the data dictionary metadata restoration process, the data dictionary metadata is verified, and when the data dictionary metadata verification is completed, the restored data dictionary metadata is obtained.

6. The method according to claim 1, characterized in that The recovering the record record and obtaining the recovered record record includes: Read the data page including the target record from disk into memory; Traverse the undo log and find the transactions that were in the target state when the failure occurred; Undo the change information of the target record made by each transaction in the target state according to the undo log record corresponding to each transaction in the target state; During the target record recovery process, the target record is verified, and when the target record verification is completed, the recovered record record is obtained.

7. The method according to claim 1, characterized in that The method further comprises: Receive a structured query language SQL statement sent by a client, and read a target data page page from a file system according to the structured query language SQL statement; Put the target data page page into a data page cache pool, and when it is determined that the target data page page exists in the data page cache pool, perform a redolog-data page refresh operation and a linked list update operation; Determine the record record to be read corresponding to the target data page page, and determine the target record record according to the processing status corresponding to the suspended transaction.

8. The method according to claim 1, characterized in that The method further comprises: Read the redolog file closest to the current time from the redolog files in the file system; Parse the redolog file closest to the current time to obtain the data page number of the redolog file closest to the current time; The data page page number is stored, and the redol og file corresponding to the same data page page number in the file system is stored in a one-way linked list corresponding to the same data page page number; The target data page page is read from the data page page file of the file system, and then the one-way linked lists are queried according to the target data page pag to obtain and use the redolog corresponding to the target data page pag.

9. A data recovery device, characterized in that: include: A log loading unit, configured to load a physical log redo log file that is closest to a current time from among physical log redo log files in a file system when a database system fails and a startup instruction for the database system is received, wherein the database system includes a database node; a data page modification unit, configured to read data page pages corresponding to each physical log redo log file in at least one physical log redo log file from a disk to a memory according to a scanning order corresponding to at least one physical log redo log file in the physical log redo log file and the physical log redo log file closest to the current time, modify the data page pages in the memory according to the physical log redo log files, and rewrite the modified data page pages back to the disk; a data recovery unit, configured to recover the data dictionary metadata and obtain the recovered data dictionary metadata when the modified data page page and the undo log undo log meet the recovery requirements, wherein the undo log undo log includes the record record in the modified data page page; The data recovery unit is further used to recover the record record when the modified data page page and the undo log undo log meet the recovery requirements, and obtain the recovered record record.

10. A network device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the data recovery method as described in any one of claims 1 to 7 or 8.

Images