A distributed log in-memory database parallel recovery method and system

Abstract

The application discloses a kind of distributed log's memory database parallel recovery method and system, belong to computer database technical field, the method is: the command log generated in memory database is stored by log partitioning and is physically isolated, and the metadata of each log partition is recorded;Based on pre-received recovery instruction and the metadata, the corresponding recovery plan is generated for each log partition;Parallelly start multiple recovery threads, and based on the recovery plan, the snapshot information pre-acquired in each recovery thread is loaded corresponding log partition, and according to the recovery plan and the snapshot information, the log fragment that needs to be replayed is determined, and the log fragment that needs to be replayed is replayed to corresponding log partition, to carry out parallel recovery to memory database.Therefore, by implementing the present application, the problem that the data recovery reliability of prior art distributed log memory data recovery process is complex and dependent on manual intervention can be solved.

Description

Technical Field

[0001] This invention relates to the field of computer database technology, and in particular to a method and system for parallel recovery of a distributed log-based in-memory database. Background Technology

[0002] In-memory databases, due to their persistent data reside in memory, possess extremely high transaction processing performance and have become a core infrastructure for real-time applications and online transactions. To ensure data persistence, in-memory databases typically employ a strategy combining snapshots and command logs. This involves periodically generating data snapshots and recording each data change as a command log. When a system failure occurs and recovery is needed, the most recent snapshot is loaded first, and then all command logs since that snapshot are replayed sequentially to reconstruct subsequent data changes and ultimately restore the system to its pre-failure state. However, in large-scale distributed environments, data is horizontally partitioned into multiple partitions distributed across different physical nodes. Each partition independently generates massive amounts of command logs, potentially accumulating in terabytes of log volume between snapshots. Traditional recovery methods require globally sorting and serially replaying these logs, resulting in recovery times of several hours. Furthermore, the recovery process often relies on operations personnel manually locating snapshots, collecting logs, and verifying integrity, making the process complex and error-prone, further exacerbating the unreliability of data recovery.

[0003] Currently, existing technologies mainly focus on optimizing log storage formats or using log compression techniques to improve I / O efficiency, but they have failed to overcome the serial computation bottleneck inherent in log replay itself. Some solutions attempt to introduce the concept of parallel recovery, but they usually rely on complex transaction dependency graph analysis, which is difficult to implement. Furthermore, when handling cross-partition transactions, they tend to degenerate into serial execution, failing to fully utilize multi-core parallel resources. It is evident that the existing distributed log memory data recovery process is complex and relies on manual intervention, resulting in poor data recovery reliability. Summary of the Invention

[0004] This invention proposes a parallel recovery method and system for distributed log in-memory databases, which solves the problem of poor data recovery reliability caused by the complexity of existing distributed log in-memory data recovery processes and reliance on manual intervention. This invention automates the parallel recovery process, reduces manual intervention, simplifies the recovery process, and improves data recovery reliability.

[0005] To achieve the above objectives, embodiments of the present invention provide a parallel recovery method for a distributed log-based in-memory database, comprising: physically isolating and storing command logs generated in the in-memory database according to log partitions, and recording metadata for each log partition; generating a corresponding recovery plan for each log partition based on a pre-received recovery instruction and the metadata; starting multiple recovery threads in parallel, loading pre-acquired snapshot information of the corresponding log partition in each recovery thread based on the recovery plan, determining the log fragments to be replayed according to the recovery plan and the snapshot information, and replaying the log fragments to be replayed to the corresponding log partitions, thereby performing parallel recovery of the in-memory database.

[0006] This invention proposes a parallel recovery method for in-memory databases using distributed logs. Command logs are physically isolated and stored according to log partitions, and a corresponding recovery plan is generated independently for each log partition. This allows multiple threads to be started in parallel during the recovery process. Each recovery thread only handles the loading of snapshot information and log fragment replay for its corresponding log partition, thus completing the parallel recovery of the in-memory database. Therefore, by starting an independent recovery thread for each log partition and executing a corresponding independent recovery plan within that thread, the parallel recovery process can be automated on a large scale, reducing manual intervention, simplifying the recovery process, and improving data recovery reliability.

[0007] Furthermore, the step of physically isolating and storing command logs generated in the memory database according to log partitions and recording the metadata of each log partition includes: capturing command logs generated during the execution of a preset transaction; setting partition identifiers for the command logs and embedding multi-dimensional data, and writing the multi-dimensional data into the log file or log stream of the corresponding log partition according to the partition identifiers; dividing the log file or log stream into several log fragments based on preset partitioning rules; obtaining the corresponding metadata in several log fragments and associating them with the global transaction sequence number and the logical sequence number within the log partition; and recording the metadata of each log partition according to preset partitioning rules based on the global transaction sequence number and the logical sequence number within the log partition.

[0008] In the above scheme, partition identifiers and multi-dimensional data are attached during the log capture phase and written to a dedicated log stream by partition to achieve physical isolation of log data. Then, log fragments are split and sequence numbers are constructed to provide a reliable location basis for the recovery phase. This ensures that the logs of each partition are strictly ordered internally, providing a reliable data foundation for conflict-free parallel recovery, reducing the complexity of recovery plan execution, and helping to improve the reliability of data recovery.

[0009] Furthermore, the step of recording the corresponding metadata in several log shards and associating them with the global transaction sequence number and the logical sequence number within the log partition includes: using the log partition start and end index range, the preset transaction association range, the complete hash value of the log shard, and the operation type summary of each log shard as the metadata of the corresponding log shard; and associating the global transaction sequence number and the logical sequence number within the log partition based on the preset transaction association range and the log partition start and end index range.

[0010] In the above scheme, the start and end index range, transaction association range, complete hash value, and operation type digest are used as metadata for log shards. Based on the transaction association range and start and end index, the global transaction sequence number and the logical sequence number within the log partition are associated to construct a rich global and local view. As a result, this metadata can be used for fast retrieval and intelligent decision-making. Global transaction tracking is achieved through the global transaction sequence number, and the order within the log partition is guaranteed through the logical sequence number within the log partition. The operation type digest is used for subsequent log pruning, which helps to improve the efficiency and accuracy of recovery plan generation, thereby improving the reliability of data recovery.

[0011] Furthermore, the step of generating a corresponding recovery plan for each log partition based on the pre-received recovery instructions and the metadata includes: obtaining the target recovery time node and the initial snapshot information of each log partition based on the pre-received recovery instructions, wherein the initial snapshot information includes a timestamp and is composed of the snapshot information of each log shard in each log partition; querying the initial snapshot information whose timestamp is less than or equal to the target recovery time node as the target snapshot information for each log partition; querying the starting global transaction sequence number or the starting logical sequence number within the starting log partition corresponding to each log partition based on the target snapshot information; obtaining the metadata that needs to be replayed between the starting global transaction sequence number and the target recovery time node in each log partition, or obtaining the metadata that needs to be replayed between the starting logical sequence number within the starting log partition and the target recovery time node in each log partition; constructing an initial list of log shards that need to be replayed for each log partition based on the metadata; and generating a recovery plan for each log partition based on the target snapshot information and the initial list of log shards that need to be replayed for each log partition.

[0012] In the above scheme, the snapshot timestamp is used for positioning and the starting global transaction sequence number or the starting logical sequence number within the log partition is used for querying. Combined with the target recovery time point, the log shard list of each partition is retrieved independently, which effectively avoids performing a global sorting operation. Since the logs within each partition are naturally ordered, only the metadata index needs to be queried to obtain the log list sorted by the logical sequence number within the log partition. This eliminates the risk of sorting errors caused by clock asynchrony or network latency, reduces the computational complexity of recovery plan generation, and helps improve the reliability of data recovery.

[0013] Furthermore, based on the target snapshot information corresponding to each log partition and the initial list of log shards requiring replay corresponding to each log partition, a recovery plan is generated for each log partition, including: obtaining the complete hash value of each log shard in the initial list of log shards requiring replay; obtaining the verification hash value of each log shard in the pre-distributed pre-check task based on the pre-distributed pre-check task; if the complete hash value of each log shard matches the verification hash value, then each log shard is marked as verified; if the complete hash value of any log shard does not match the verification hash value, then the log shard is marked as a corrupted log shard, several replica log shards corresponding to the corrupted log shard are obtained, and pre-check tasks are distributed to the several replica log shards in sequence to perform hash value verification. The process continues until the complete hash value of any of the replica log shards matches the verification hash value, thus obtaining the target replica log shard; the corrupted log shard is updated to the target replica log shard, and the target replica log shard is marked as verified; when all the log shards in the initial list of log shards to be replayed are marked as verified, a list of logs to be replayed is obtained; based on the pre-acquired recovery configuration file and the operation type summary of each log shard, the operation type of the list of logs to be replayed is verified, and log shards that do not meet the preset verification requirements are removed, thus obtaining the target list of logs to be replayed; based on the target snapshot information corresponding to each log partition and the target list of logs to be replayed corresponding to each log partition, a recovery plan corresponding to each log partition is constructed.

[0014] The above scheme introduces integrity pre-check verification of remote log fragments to detect damaged fragments in advance and dynamically replace them with replicas, realizing the transformation from passive failure to proactive repair. Semantic pruning optimization is also introduced, and irrelevant log fragments are filtered using operation type digests based on the recovery configuration file, reducing the amount of data to be transmitted and processed, improving recovery efficiency and success rate, and helping to improve the reliability of data recovery.

[0015] Furthermore, before executing the step of constructing a recovery plan for each log partition based on the target snapshot information and the target replay log list corresponding to each log partition, the method further includes: obtaining the start and end index range or preset transaction association range of the log partition corresponding to each log partition based on the target replay log list corresponding to each log partition; performing continuity verification on each log shard in the target replay log list corresponding to each log partition based on the start and end index range or preset transaction association range of the log partition corresponding to each log partition; if all log shards meet the preset continuity verification requirements, then executing the step of constructing a recovery plan for each log partition based on the target snapshot information and the target replay log list corresponding to each log partition; if any log shard does not meet the preset continuity verification requirements, then suspending the step of constructing a recovery plan for each log partition based on the target snapshot information and the target replay log list corresponding to each log partition.

[0016] In the above scheme, the log fragments of each partition are verified for continuity before the final solidification recovery plan is executed to ensure that the log chain within each log partition is complete and unbroken. This step serves as the final verification of the correctness of parallel recovery, which can effectively detect data gaps caused by metadata errors or unrecoverable damage, prevent data inconsistency caused by missing logs, and ensure that the recovery process within each log partition still strictly follows the original order of transactions when log partitions are recovered independently in parallel, thus maintaining global data consistency and helping to improve the reliability of data recovery.

[0017] Furthermore, multiple recovery threads are started in parallel, and based on the recovery plan, the snapshot information of the corresponding log partition is pre-acquired in each recovery thread. The log shards to be replayed are determined according to the recovery plan and the snapshot information, and the replayed log shards are replayed to the corresponding log partitions to perform parallel recovery of the in-memory database. This includes: starting multiple recovery threads in parallel and allocating a unique recovery plan to each recovery thread; based on the recovery plan, loading the snapshot information of the corresponding log partition in each recovery thread and re-establishing the initial state of the corresponding log partition to obtain the initialized log partition corresponding to each log partition; replaying the log shards to the corresponding initialized log partition according to the order of the log shards in the target log list to be replayed, obtaining the recovery log partition corresponding to each log partition; outputting a recovery thread completion report whenever all log shards have completed the replay operation; if all recovery threads output recovery thread completion reports, calculating the recovery hash value of the recovery log partition and performing a consistency check between the recovery hash value and a preset check value; and completing the parallel recovery of the in-memory database after the check passes.

[0018] In the above scheme, an independent recovery thread is allocated to each partition, and snapshot loading, log fragment verification and replay are executed sequentially within the thread to achieve large-scale parallel recovery; all threads run independently without sharing or synchronization, making full use of multi-core CPU resources; after recovery, a global hash verification is performed to ensure that all partition data is accurate, and finally the service is automatically brought online, realizing the automation of the parallel recovery process, reducing manual intervention and simplifying the recovery process, and improving the reliability of data recovery.

[0019] This invention also provides a parallel recovery system for a distributed log-based in-memory database, comprising: a data acquisition module, a recovery plan generation module, and a parallel recovery module; the data acquisition module is used to physically isolate and store command logs generated in the in-memory database according to log partitions, and record metadata for each log partition; the recovery plan generation module is used to generate a corresponding recovery plan for each log partition based on pre-received recovery instructions and the metadata; the parallel recovery module is used to start multiple recovery threads in parallel, load pre-acquired snapshot information of the corresponding log partition in each recovery thread based on the recovery plan, determine the log fragments that need to be replayed according to the recovery plan and the snapshot information, and replay the log fragments that need to be replayed to the corresponding log partitions to perform parallel recovery of the in-memory database.

[0020] This invention proposes a distributed log-based parallel recovery system for in-memory databases. Command logs are physically isolated and stored according to log partitions, and a corresponding recovery plan is generated independently for each log partition. This allows multiple threads to be started in parallel during the recovery process. Each recovery thread only handles the loading of snapshot information and log fragment replay for its corresponding log partition, thus completing the parallel recovery of the in-memory database. Therefore, by starting an independent recovery thread for each log partition and executing a corresponding independent recovery plan within that thread, the parallel recovery process can be automated on a large scale, reducing manual intervention, simplifying the recovery process, and improving data recovery reliability.

[0021] Furthermore, the data acquisition module is used to physically isolate and store the command logs generated in the memory database according to log partitions, and record the metadata of each log partition, including: a command log acquisition unit, a data preprocessing unit, a data segmentation unit, a sequence number construction unit, and a metadata recording unit; the command log acquisition unit is used to capture the command logs generated during the execution of a preset transaction; the data preprocessing unit is used to set partition identifiers for the command logs and embed multi-dimensional data, and write the multi-dimensional data into the log file or log stream of the corresponding log partition according to the partition identifiers; the data segmentation unit is used to segment the log file or the log stream into several log fragments based on preset segmentation rules; the sequence number construction unit is used to acquire the corresponding metadata in several log fragments and associate them with the global transaction sequence number and the logical sequence number within the log partition; the metadata recording unit is used to record the metadata of each log partition based on the global transaction sequence number and the logical sequence number within the log partition, according to preset partitioning rules.

[0022] Furthermore, the recovery plan generation module is used to generate corresponding recovery plans for each log partition based on the pre-received recovery instructions and the metadata, including: an initial snapshot information acquisition unit, a target snapshot information query unit, a sequence number query unit, a log shard list construction unit, and a recovery plan acquisition unit; the initial snapshot information acquisition unit is used to acquire the target recovery time node and the initial snapshot information of each log partition based on the pre-received recovery instructions, wherein the initial snapshot information includes a timestamp, and the initial snapshot information is composed of the snapshot information of each log shard in each log partition; the target snapshot information query unit is used to query the initial snapshot information with a timestamp less than or equal to the target recovery time node as the target snapshot information for each log partition; The sequence number query unit is used to query the starting global transaction sequence number or the logical sequence number within the starting log partition corresponding to each log partition based on the target snapshot information; the log shard list construction unit is used to obtain the metadata that needs to be replayed between the starting global transaction sequence number and the target recovery time node in each log partition, or obtain the metadata that needs to be replayed between the logical sequence number within the starting log partition and the target recovery time node in each log partition, and construct the initial log shard list that needs to be replayed for each log partition based on the metadata; the recovery plan acquisition unit is used to generate the recovery plan corresponding to each log partition based on the target snapshot information and the initial log shard list that needs to be replayed for each log partition. Attached Figure Description

[0023] To more clearly illustrate the technical solution of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 A flowchart illustrating the steps of a parallel recovery method for a distributed log memory database provided in a certain embodiment of the present invention; Figure 2 This is a schematic diagram of the module structure of a distributed log in-memory database parallel recovery system provided in one embodiment of the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0027] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0028] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0030] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0031] In the description of the embodiments of this application, unless otherwise expressly specified and limited, technical terms such as "installation," "connection," "joining," and "fixing" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. For those skilled in the art, the specific meaning of the above terms in the embodiments of this application can be understood according to the specific circumstances.

[0032] To explain the parallel recovery method for a distributed log memory database proposed in this embodiment of the invention, a recovery system architecture integrated with a memory database cluster is used for explanation. This recovery system architecture includes: a distributed log storage adapter, a metadata manager, a recovery controller, a parallel recovery engine, and a control center.

[0033] The distributed log storage adapter runs on each database node, intercepts the command logs generated by the node, and routes them to the correct location on the distributed storage according to the partition ID corresponding to the transaction. That is, during the execution of a transaction, the command logs generated are automatically tagged with their respective partition IDs before being written to persistent storage. The adapter uses this ID to write the log streams of different partitions to different logical paths or files in the distributed file system.

[0034] The metadata manager is used for centralized metadata storage and serves as the directory service for the entire system. It employs a highly available distributed coordination service to uniformly manage the metadata of all backup assets. The metadata includes: snapshot information, storage path of each log shard, start and end timestamps (or transaction sequence numbers LSN), partition ID, and data integrity hash value, forming a globally consistent and queryable backup directory.

[0035] The recovery controller is a service that can run independently or be integrated into the management node. It serves as the entry point and scheduling center for the automated recovery process, acting as the overall commander of the recovery process. When it receives a recovery instruction, it can automatically query the metadata manager and intelligently plan a recovery path for each partition, i.e., select which base snapshot and which log shards need to be replayed.

[0036] The parallel recovery engine is a dynamic pool of threads / processes managed by the recovery controller. It is the unit of work that actually executes recovery tasks. Specifically, it dynamically creates a corresponding number of recovery worker threads based on the number of database partitions. Each thread is assigned a partition and completes all the work set in the recovery plan independently and in parallel according to the recovery plan provided by the controller.

[0037] The control center is responsible for controlling the distributed log storage adapter, metadata manager, recovery controller, and parallel recovery engine to execute the in-memory database parallel recovery method for distributed logs proposed in this embodiment of the invention.

[0038] To address the issue of poor data recovery reliability due to the complexity of existing distributed log memory data recovery processes and their reliance on manual intervention, see [link to relevant documentation]. Figure 1 , Figure 1 This is a flowchart illustrating the steps of a parallel recovery method for a distributed log memory database according to a certain embodiment of the present invention. Figure 1 As shown in the figure, this embodiment of the invention proposes a parallel recovery method for a distributed log memory database, including steps 101 to 103, each step of which is as follows: Step 101: Physically isolate and store the command logs generated in the in-memory database according to log partitions, and record the metadata of each log partition; Step 102: Based on the pre-received recovery instructions and the metadata, generate a corresponding recovery plan for each log partition; Step 103: Start multiple recovery threads in parallel, load the snapshot information of the corresponding log partition in each recovery thread based on the recovery plan, determine the log fragments that need to be replayed according to the recovery plan and the snapshot information, and replay the log fragments that need to be replayed to the corresponding log partitions to perform parallel recovery of the in-memory database.

[0039] One possible implementation involves the following: During normal operation of the distributed in-memory database, each log partition is processed serially by an independent thread. After each write transaction is completed, a command log is generated. The distributed log storage adapter intercepts these logs, appends its partition identifier to each log, and embeds multi-dimensional data, including: Global Transaction ID (GTID), Partition-Local Sequence Number (PLSN), write node information, and an operation type summary and complete hash value. Then, the distributed log storage adapter writes the logs to a dedicated log file or stream in the distributed file system based on the partition identifier, for example, using a hierarchical structure like " / database root directory / logs / partition ID / transaction phase / log fragment identifier". For ease of management, the log stream of each partition is divided into several log fragments of fixed size or time intervals. Each time a log fragment is generated, its metadata is registered in the distributed metadata manager. Simultaneously, when a snapshot is generated, the snapshot metadata records the GTID and PLSN of each partition at that time.

[0040] Then, the recovery controller queries the metadata manager based on the target time point to obtain the latest snapshot of each partition and its recorded starting GTID or starting PLSN. Next, using the GTID log shard index and the PLSN index within the partition established by the metadata manager, the recovery controller quickly retrieves all log shard metadata for each partition from the starting point to the target recovery time point, and the returned list is already sorted by PLSN, eliminating the need for cross-partition sorting. Then, in the pre-check optimization phase, the recovery controller batch-fetches the integrity hash values ​​of each shard and distributes pre-check tasks to the parallel recovery engine. Worker threads read remote log shard data in a streaming manner and calculate hash values ​​in real time. If a match is found, it is marked as verified; if not, other replicas are queried and the verification is repeated until a valid replica is found. After all shards pass integrity verification, the recovery controller filters out shards that do not contain user data modification instructions based on the operation type summary according to the recovery configuration file, thereby generating a target log list to be replayed. Finally, the recovery controller performs PLSN continuity verification on the log list of each partition to ensure that the end_plsn of the previous shard and the start_plsn of the next shard are accurately connected. If a break is found, the recovery is stopped and an alarm is triggered. In this embodiment, end_plsn represents the end sequence number and start_plsn represents the start sequence number. After completing the above steps, the recovery controller constructs a recovery plan for each partition. The recovery plan includes: plan identifier, target partition identifier, recovery mode, snapshot information group, log replay sequence array, and termination condition object.

[0041] Finally, the recovery controller starts a corresponding number of recovery threads in parallel based on the total number of log partitions. Each thread receives a dedicated recovery plan. The recovery thread first parses the snapshot information in the recovery plan and loads the data blocks of its own log partition from the snapshot file in sequence, thereby reconstructing the initial state of the partition in memory. Subsequently, the recovery thread downloads the log fragments one by one according to the log replay sequence. The recovery thread replays each command log in the fragment serially, applying the data changes to the memory partition. When all logs that need to be replayed in the log partition have been processed, the recovery thread sends a completion report to the controller. The recovery controller waits for all threads to complete before triggering a global consistency check. For example, it instructs each partition to calculate the hash value of the current data and compare it with the check value pre-stored during backup, or it executes business logic checks. After the check passes, the controller announces that the database cluster has been successfully recovered and resumes providing services.

[0042] This invention proposes a parallel recovery method for in-memory databases using distributed logs. Command logs are physically isolated and stored according to log partitions, and a corresponding recovery plan is generated independently for each log partition. This allows multiple threads to be started in parallel during the recovery process. Each recovery thread only handles the loading of snapshot information and log fragment replay for its corresponding log partition, thus completing the parallel recovery of the in-memory database. Therefore, by starting an independent recovery thread for each log partition and executing a corresponding independent recovery plan within that thread, the parallel recovery process can be automated on a large scale, reducing manual intervention, simplifying the recovery process, and improving data recovery reliability.

[0043] In a preferred embodiment, the step of physically isolating and storing command logs generated in the in-memory database according to log partitions and recording metadata of each log partition includes: capturing command logs generated during the execution of a preset transaction; setting partition identifiers for the command logs and embedding multi-dimensional data, and writing the multi-dimensional data into the log file or log stream of the corresponding log partition according to the partition identifiers; dividing the log file or log stream into several log fragments based on preset partitioning rules; obtaining the corresponding metadata in several log fragments and associating them with the global transaction sequence number and the logical sequence number within the log partition; and recording the metadata of each log partition according to the preset partitioning rules based on the global transaction sequence number and the logical sequence number within the log partition.

[0044] For example, during the normal operation of the distributed in-memory database, each log partition is processed serially by an independent thread. After each write transaction is completed, a command log is generated. The distributed log storage adapter intercepts these logs, appends the partition identifier to each log, and embeds multi-dimensional data, including: Global Transaction ID (GTID), Partition-Local Sequence Number (PLSN), write node information, operation type summary, and complete hash value of the log fragment. Then, the distributed log storage adapter writes the log to a log file or stream in the distributed file system that is dedicated to that partition, based on the partition identifier. For example, it uses a hierarchical structure of " / database root directory / logs / partition ID / transaction phase / log fragment identifier" for storage. For ease of management, the log stream of each partition is divided into several log fragments of fixed size or time interval. Each time a log fragment is generated, its metadata is registered in the distributed metadata manager. At the same time, when snapshot information is generated, the metadata of the snapshot information records the GTID and PLSN of each partition at that time. Specifically, when a write transaction executes on a log partition, the command log it generates is captured by the log storage adapter before being persisted. This command log includes stored procedure calls and parameters. The adapter appends a unique ID to the log partition, which serves as the partition identifier, and then writes it to a log file or log stream specific to that partition. In this embodiment, the Global Transaction ID (GTID) is used to record the globally unique transaction ID associated with the log; the Partition-Local Sequence Number (PLSN) is used to record the exact order of the log within the current log partition's log stream. Although execution within a log partition is serial, explicitly recording the PLSN helps in more accurate location and verification in extreme cases such as partial log loss due to node failure and manual recovery; the write node information records which physical node initially generated the log; the operation type summary records the operation type and parameters of the log itself, used for subsequent log filtering and selection; and the log fragment complete hash value is used to verify log integrity and is unique.

[0045] For ease of management, the log stream of each partition is divided into log shards of fixed size or at time intervals. Each time a new log shard is generated, its metadata is registered with the metadata manager. When snapshot information is generated, the snapshot information's metadata records the accurate global transaction sequence number and logical sequence number within each log partition at that time. This way, when the recovery plan is executed, it clearly knows where to begin replaying the logs.

[0046] In the above scheme, partition identifiers and multi-dimensional data are attached during the log capture phase and written to a dedicated log stream by partition to achieve physical isolation of log data. Then, log fragments are split and sequence numbers are constructed to provide a reliable location basis for the recovery phase. This ensures that the logs of each partition are strictly ordered internally, providing a reliable data foundation for conflict-free parallel recovery, reducing the complexity of recovery plan execution, and helping to improve the reliability of data recovery.

[0047] In a preferred embodiment, recording the corresponding metadata in several log shards and associating it with the global transaction sequence number and the logical sequence number within the log partition includes: using the log partition start and end index range, the preset transaction association range, the complete hash value of the log shard, and the operation type summary of each log shard as the metadata of the corresponding log shard; and associating the global transaction sequence number and the logical sequence number within the log partition based on the preset transaction association range and the log partition start and end index range.

[0048] For example, the metadata of a log shard includes: the start and end index range of the log partition for each log shard, the preset transaction association range, the complete hash value of the log shard, and a summary of the operation type. Specifically, the metadata manager no longer simply stores the independent metadata of each log shard, but instead constructs a global transaction-log shard index mapping table. This index table can quickly query a given GTID to find its corresponding log shard and its specific storage location. It can also query a given timestamp or GTID range to quickly retrieve all involved log shards and their partitions, as well as record the latest GTID and PLSN of each partition for quickly locating the recovery starting point. In the metadata of each log shard, the start and end index range of the log partition and the preset transaction association range are set to be associated with the index of the associated global transaction sequence number and the logical sequence number within the log partition, so as to facilitate subsequent quick queries and location.

[0049] In the above scheme, the start and end index range, transaction association range, complete hash value, and operation type digest are used as metadata for log shards. Based on the transaction association range and start and end index, the global transaction sequence number and the logical sequence number within the log partition are associated to construct a rich global and local view. As a result, this metadata can be used for fast retrieval and intelligent decision-making. Global transaction tracking is achieved through the global transaction sequence number, and the order within the log partition is guaranteed through the logical sequence number within the log partition. The operation type digest is used for subsequent log pruning, which helps to improve the efficiency and accuracy of recovery plan generation, thereby improving the reliability of data recovery.

[0050] In a preferred embodiment, generating a corresponding recovery plan for each log partition based on the pre-received recovery instructions and the metadata includes: obtaining the target recovery time node and initial snapshot information of each log partition based on the pre-received recovery instructions, wherein the initial snapshot information includes a timestamp and is composed of snapshot information of each log shard in each log partition; querying initial snapshot information with timestamps less than or equal to the target recovery time node as target snapshot information for each log partition; querying the starting global transaction sequence number or the starting logical sequence number within the starting log partition corresponding to each log partition based on the target snapshot information; obtaining the metadata that needs to be replayed between the starting global transaction sequence number and the target recovery time node in each log partition, or obtaining the metadata that needs to be replayed between the starting logical sequence number within the starting log partition and the target recovery time node in each log partition; constructing an initial list of log shards that need to be replayed for each log partition based on the metadata; and generating a recovery plan for each log partition based on the target snapshot information and the initial list of log shards that need to be replayed for each log partition.

[0051] For example, the recovery controller queries the metadata manager based on the target time point to obtain the latest snapshot of each partition and its recorded starting GTID or starting PLSN. Then, using the GTID log shard index and the PLSN index within the partition established by the metadata manager, the recovery controller quickly retrieves all log shard metadata for each partition from the starting point to the target recovery time point. The returned list is already sorted by PLSN, eliminating the need for cross-partition sorting. After completing the above steps, the recovery controller builds a recovery plan for each partition. Specifically, first, upon receiving a recovery instruction, the target recovery time point is read from the recovery instruction, and the snapshot information of log shards in each log partition is integrated into initial snapshot information, which includes a timestamp. This timestamp is used to compare with the target recovery time point. The cloud data manager is queried to select the latest available snapshot information with a timestamp less than or equal to the target recovery time point as the target snapshot information. This target snapshot information can be a full log chain or an incremental snapshot chain. Then, the metadata manager is queried starting from the starting global transaction sequence number or the starting logical sequence number within the log partition of the target snapshot information to retrieve the metadata for each log partition from the starting global transaction sequence number. The initial list of log shards to be replayed from the starting global transaction sequence number to the target recovery time node is used as the initial list of log shards to be replayed. This list includes the metadata to be replayed. Alternatively, the initial list of log shards to be replayed from the logical sequence number within the starting log partition to the target recovery time node is also used as the initial list of log shards to be replayed, including the metadata to be replayed. Typically, during implementation, only one of the starting global transaction sequence number or the logical sequence number within the starting log partition is used; it is not necessary to use both simultaneously for list retrieval. To clarify the implementation, both scenarios are explained simply in this embodiment. After constructing the initial list of log shards to be replayed, a recovery plan for each log partition is generated based on the target snapshot information and the initial list of log shards to be replayed for each log partition. The recovery plan includes: a plan identifier, a target partition identifier, a recovery mode, a snapshot information group, a log replay sequence array, and a termination condition object.

[0052] In the above scheme, the snapshot timestamp is used for positioning and the starting global transaction sequence number or the starting logical sequence number within the log partition is used for querying. Combined with the target recovery time point, the log shard list of each partition is retrieved independently, which effectively avoids performing a global sorting operation. Since the logs within each partition are naturally ordered, only the metadata index needs to be queried to obtain the log list sorted by the logical sequence number within the log partition. This eliminates the risk of sorting errors caused by clock asynchrony or network latency, reduces the computational complexity of recovery plan generation, and helps improve the reliability of data recovery.

[0053] A preferred embodiment generates a recovery plan for each log partition based on the target snapshot information corresponding to each log partition and the initial list of log shards to be replayed corresponding to each log partition. The plan includes: obtaining the complete hash value of each log shard in the initial list of log shards to be replayed; obtaining the verification hash value of each log shard in the pre-distributed pre-check task based on a pre-distributed pre-check task; if the complete hash value of each log shard matches the verification hash value, then marking each log shard as verified; if the complete hash value of any log shard does not match the verification hash value, then marking the log shard as a corrupted log shard, obtaining several replica log shards corresponding to the corrupted log shard, and sequentially distributing pre-check tasks to the several replica log shards to perform hash value checks. The verification process continues until the complete hash value of any of the replica log shards matches the verification hash value, thus obtaining the target replica log shard; the corrupted log shard is updated to the target replica log shard, and the target replica log shard is marked as verified; when all the log shards in the initial log shard list that need to be replayed are marked as verified, a verification log shard list that needs to be replayed is obtained; based on the pre-acquired recovery configuration file and the operation type summary of each log shard, the operation type of the verification log shard list that needs to be replayed is verified, and log shards that do not meet the preset verification requirements are removed, thus obtaining the target log shard list that needs to be replayed; based on the target snapshot information corresponding to each log partition and the target log shard list that needs to be replayed corresponding to each log partition, a recovery plan corresponding to each log partition is constructed.

[0054] For example, during the pre-inspection optimization phase, the recovery controller batch-fetches the integrity hash values ​​of each shard and distributes pre-inspection tasks to the parallel recovery engine. Worker threads read remote log shard data in a streaming manner and calculate hash values ​​in real time. If a match is found, it is marked as verified; if not, other replicas are queried and the verification is repeated until a valid replica is found. After all shards pass the integrity verification, the recovery controller uses the operation type summary based on the recovery configuration file to filter out shards that do not contain user data modification instructions, thereby generating a target log replay list. After completing the above steps, the recovery controller builds a recovery plan for each partition. The recovery plan includes: plan identifier, target partition identifier, recovery mode, snapshot information group, log replay sequence array, and termination condition object.

[0055] Since the initial list of log shards requiring replay may contain a large number of redundant log shards, a pre-check optimization is performed in this embodiment to make the generated recovery plan more reliable. Specifically, firstly, the recovery controller initiates a batch request to the metadata manager. This batch request retrieves the integrity metadata of all log shards in the initial list of log shards requiring replay. This integrity metadata includes either a complete hash value or a pre-calculated and stored checksum, both of which can be used for integrity verification. Then, the recovery controller sends a pre-check task to the parallel recovery engine. Each pre-check task includes the storage path of one or more log shards and their corresponding integrity metadata. Next, the parallel recovery engine executes the pre-check task in its worker threads. In each worker thread, the specified log shard data stream is reread from the distributed storage system, and the checksum or complete hash value corresponding to the reread log shard is calculated in real time. In this embodiment, the complete hash value is used as an example. Then, the real-time calculated complete hash value is compared with the pre-set checksum hash value. If the comparison is successful... If the results are consistent, it means that the real-time calculated complete hash value matches the verification hash value, and the log fragment is marked as verified. If the comparison results are inconsistent, it means that the real-time calculated complete hash value does not match the verification hash value, indicating that the log fragment is corrupted. However, the corrupted log fragment is not discarded directly. Instead, the following improvement steps are performed: First, the recovery controller quickly queries the metadata manager to obtain the storage location of other replicas of the corrupted log fragment. It is worth mentioning that distributed file systems usually provide multiple replicas to ensure high availability, and the metadata manager also records the location information of all replicas. Then, a new pre-inspection task is generated for the obtained replica fragments, and the updated pre-inspection task is executed to perform hash value matching verification on the replica log fragments. Once a correct replica log fragment is matched, it is used as the target replica log fragment, and the corrupted log fragment is updated to the target replica log fragment. If all replicas cannot be matched correctly, the corrupted log fragment is marked as unrecoverable, and the recovery of the partition can be stopped and an alarm is issued.

[0056] Once all log shards in the initial list of log shards requiring replay are marked as verified, a new list of log shards requiring replay is obtained. After obtaining this list, the recovery controller performs pruning decisions based on a specified recovery profile, which is the file carried in the recovery instructions. This recovery profile includes two modes: the most commonly used mode, PROFILE_DATA_ONLY, which only concerns the final state of user data; and PROFILE_FULL_REPLAY, used for auditing or problem reproduction, which requires replaying all operations, including internal management operations. Taking PROFILE_DATA_ONLY mode as an example, the pruning decision is as follows: the recovery controller checks the "Operation Type Summary" metadata of the shards. If a shard's summary shows that it only contains non-data change operations, then that shard will be safely removed from the recovery plan. Non-data change operations include those where HAS_USER_WRITE is false in the summary, but HAS_INDEX_REBUILD or HAS_AUDIT_LOG is true. After the pruning decision, the subsequent recovery plan only contains log fragments that actually contain user data modification instructions, which greatly reduces the amount of data that the subsequent recovery engine needs to download and process.

[0057] Finally, the complete pruning decision verification log list that needs to be replayed is used as the target log list that needs to be replayed; and based on the target snapshot information and the target log list that needs to be replayed for each log partition, a recovery plan is generated for each log partition. The recovery plan includes: plan identifier, target partition identifier, recovery mode, snapshot information group, log replay sequence array, and termination condition object.

[0058] The above scheme introduces integrity pre-check verification of remote log fragments to detect damaged fragments in advance and dynamically replace them with replicas, realizing the transformation from passive failure to proactive repair. Semantic pruning optimization is also introduced, and irrelevant log fragments are filtered using operation type digests based on the recovery configuration file, reducing the amount of data to be transmitted and processed, improving recovery efficiency and success rate, and helping to improve the reliability of data recovery.

[0059] In a preferred embodiment, before executing the step of constructing a recovery plan for each log partition based on the target snapshot information and the target replay log list corresponding to each log partition, the method further includes: obtaining the start and end index range or a preset transaction association range of the log partition corresponding to each log partition based on the target replay log list corresponding to each log partition; performing continuity verification on each log shard in the target replay log list corresponding to each log partition based on the start and end index range or the preset transaction association range of the log partition corresponding to each log partition; if all log shards meet the preset continuity verification requirements, then executing the step of constructing a recovery plan for each log partition based on the target snapshot information and the target replay log list corresponding to each log partition; if any log shard does not meet the preset continuity verification requirements, then suspending the step of constructing a recovery plan for each log partition based on the target snapshot information and the target replay log list corresponding to each log partition.

[0060] For example, the recovery controller performs PLSN continuity verification on the log lists of each partition to ensure that the end_plsn of the previous partition and the start_plsn of the next partition are accurately connected. If a break is found, recovery is aborted and an alarm is issued. In this embodiment, end_plsn represents the end sequence number and start_plsn represents the start sequence number. Specifically, for each log partition corresponding to the target log list to be replayed, the recovery controller needs to check the log partition start and end index range and the preset transaction association range of its metadata. When performing continuity verification, either the log partition start and end index range or the preset transaction association range can be used. The recovery controller verifies that the end_plsn of the previous shard and the start_plsn of the next shard are accurately aligned, or that the end_plsn of the previous shard and the start_plsn of the next shard differ by 1. In this embodiment, the preset continuity verification requirement is that the end_plsn of the previous shard and the start_plsn of the next shard are accurately aligned, or that the end_plsn of the previous shard and the start_plsn of the next shard differ by 1. If the log sequences of all partitions are continuous, i.e., the continuity verification is passed, the recovery controller solidifies the recovery plan and sends it to the parallel recovery engine; if any discontinuity exists, the recovery process will be terminated and an alarm will be issued, indicating which partition and which LSN range has a data gap.

[0061] In the above scheme, the log fragments of each partition are verified for continuity before the final solidification recovery plan is executed to ensure that the log chain within each log partition is complete and unbroken. This step serves as the final verification of the correctness of parallel recovery, which can effectively detect data gaps caused by metadata errors or unrecoverable damage, prevent data inconsistency caused by missing logs, and ensure that the recovery process within each log partition still strictly follows the original order of transactions when log partitions are recovered independently in parallel, thus maintaining global data consistency and helping to improve the reliability of data recovery.

[0062] A preferred embodiment involves starting multiple recovery threads in parallel, loading pre-acquired snapshot information of corresponding log partitions into each recovery thread based on the recovery plan, determining log shards to be replayed according to the recovery plan and the snapshot information, and replaying the log shards to be replayed to the corresponding log partitions to perform parallel recovery of the in-memory database. This includes: starting multiple recovery threads in parallel and assigning a unique recovery plan to each recovery thread; loading snapshot information of corresponding log partitions into each recovery thread based on the recovery plan, and re-establishing the initial state of the corresponding log partitions to obtain the initialized log partitions corresponding to each log partition; replaying the log shards to the corresponding initialized log partitions in the order of the log shards in the target log list to be replayed, to obtain the recovery log partitions corresponding to each log partition; outputting a recovery thread completion report whenever all log shards have completed the replay operation; if all recovery threads output recovery thread completion reports, calculating the recovery hash value of the recovery log partitions, and performing a consistency check between the recovery hash value and a preset check value; and completing the parallel recovery of the in-memory database after the check passes.

[0063] For example, the recovery controller starts a corresponding number of recovery threads in parallel based on the total number of log partitions. Each thread obtains its own recovery plan. The recovery thread first parses the snapshot information in the recovery plan and loads the data blocks of its own log partition in the snapshot file in sequence, thereby reconstructing the initial state of the partition in memory. Subsequently, the recovery thread downloads the log fragments one by one in the order of the log replay sequence. The recovery thread replays each command log in the fragment serially, applying the data changes to the memory partition. When all the logs that need to be replayed in the log partition have been processed, the recovery thread sends a completion report to the controller. The recovery controller waits for all threads to complete before triggering a global consistency check. For example, it instructs each partition to calculate the hash value of the current data and compare it with the check value pre-stored during backup, or it executes business logic checks. After the check passes, the controller announces that the database cluster has been successfully recovered and resumes providing services. Specifically, through the above steps, the recovery plan is constructed. The data structure of the recovery plan includes: plan identifier, target partition identifier, recovery mode, snapshot information group, log replay sequence array, and termination condition object, etc. Then, the parallel recovery engine is controlled to execute the recovery plan according to the pre-designed requirements, which include: task independence and self-containment, data loading security, resource acquisition optimization, and precise execution termination. To meet the requirements of task independence and self-containment, recovery threads need to be started in parallel and recovery plans need to be allocated independently to each thread. To meet the requirements of data loading security, after the recovery thread reads any data block from distributed storage, the hash value of the data needs to be calculated and compared with the hash value recorded in the recovery plan. If the verification fails, the recovery is terminated and an error is alerted. To meet the requirements of resource acquisition optimization, the recovery thread needs to be scheduled to execute on the computing node closest to the data source, or the nearest storage replica should be selected first when pulling data. To meet the requirements of precise execution termination, when replaying logs one by one, the timestamp of the current log record or the target recovery time node needs to be continuously checked. Once the termination condition is met, the replay stops, the remaining records of the log are discarded, and the task is reported as complete.

[0064] The recovery plan execution process begins with the recovery controller starting a corresponding number of recovery worker threads from the parallel recovery engine based on the total number of partitions in the cluster. Each recovery worker thread is assigned a unique recovery plan corresponding to each log partition, and each recovery thread performs recovery independently. Then, during the recovery process, the data portion belonging to the corresponding recovery thread's log partition is read from the base snapshot file in distributed storage, and the initial state of the log partition is reconstructed in the memory of the new node, resulting in the initialized log partition. Next, within the initialized log partition, according to the target log shard replay list in the recovery plan, log shards are downloaded sequentially from distributed storage to obtain the recovery log partition. When a recovery thread completes the replay of all logs in its partition, it reports its completion status to the recovery controller, outputting a recovery thread completion report. After all recovery threads have reported completion, it signifies that the entire cluster data has been restored to the target time point. Then, the hash value of the recovery log partition in the restored cluster is calculated and compared with a pre-stored checksum. If the checksum passes, the controller declares the database cluster recovery successful and resumes providing services.

[0065] In the above scheme, an independent recovery thread is allocated to each partition, and snapshot loading, log fragment verification and replay are executed sequentially within the thread to achieve large-scale parallel recovery; all threads run independently without sharing or synchronization, making full use of multi-core CPU resources; after recovery, a global hash verification is performed to ensure that all partition data is accurate, and finally the service is automatically brought online, realizing the automation of the parallel recovery process, reducing manual intervention and simplifying the recovery process, and improving the reliability of data recovery.

[0066] Based on the above method embodiments, corresponding apparatus embodiments are provided; see [link to apparatus embodiments]. Figure 2 , Figure 2 This is a schematic diagram of the module structure of a distributed log-based in-memory database parallel recovery system according to a certain embodiment of the present invention. Figure 2 As shown in the figure, this embodiment of the invention also provides a parallel recovery system for a distributed log memory database, including: a data acquisition module 201, a recovery plan generation module 202, and a parallel recovery module 203; the data acquisition module 201 is used to physically isolate and store the command logs generated in the memory database according to log partitions, and record the metadata of each log partition; the recovery plan generation module 202 is used to generate a corresponding recovery plan for each log partition based on the pre-received recovery instructions and the metadata; the parallel recovery module 203 is used to start multiple recovery threads in parallel, load the pre-acquired snapshot information of the corresponding log partition in each recovery thread based on the recovery plan, determine the log fragments that need to be replayed according to the recovery plan and the snapshot information, and replay the log fragments that need to be replayed to the corresponding log partitions to perform parallel recovery of the memory database.

[0067] This invention proposes a distributed log-based parallel recovery system for in-memory databases. Command logs are physically isolated and stored according to log partitions, and a corresponding recovery plan is generated independently for each log partition. This allows multiple threads to be started in parallel during the recovery process. Each recovery thread only handles the loading of snapshot information and log fragment replay for its corresponding log partition, thus completing the parallel recovery of the in-memory database. Therefore, by starting an independent recovery thread for each log partition and executing a corresponding independent recovery plan within that thread, the parallel recovery process can be automated on a large scale, reducing manual intervention, simplifying the recovery process, and improving data recovery reliability.

[0068] Furthermore, the data acquisition module 201 is used to physically isolate and store the command logs generated in the memory database according to log partitions, and record the metadata of each log partition, including: a command log acquisition unit 301, a data preprocessing unit 302, a data segmentation unit 303, a sequence number construction unit 304, and a metadata recording unit 305; the command log acquisition unit 301 is used to capture the command logs generated during the execution of a preset transaction; the data preprocessing unit 302 is used to set partition identifiers for the command logs and embed multi-dimensional data, and write the multi-dimensional data into the log file or log stream of the corresponding log partition according to the partition identifiers; the data segmentation unit 303 is used to segment the log file or the log stream into several log fragments based on preset segmentation rules; the sequence number construction unit 304 is used to acquire the corresponding metadata in several log fragments and associate them with the global transaction sequence number and the logical sequence number within the log partition; the metadata recording unit 305 is used to record the metadata of each log partition according to the global transaction sequence number and the logical sequence number within the log partition, based on preset partitioning rules.

[0069] Furthermore, the recovery plan generation module 202 is used to generate corresponding recovery plans for each log partition based on the pre-received recovery instructions and the metadata, including: an initial snapshot information acquisition unit 401, a target snapshot information query unit 402, a sequence number query unit 403, a log shard list construction unit 404, and a recovery plan acquisition unit 405; the initial snapshot information acquisition unit 401 is used to acquire the target recovery time node and the initial snapshot information of each log partition based on the pre-received recovery instructions, wherein the initial snapshot information includes a timestamp, and the initial snapshot information is composed of the snapshot information of each log shard in each log partition; the target snapshot information query unit 402 is used to query the initial snapshot information with a timestamp less than or equal to the target recovery time node as the target for each log partition. The snapshot information; the sequence number query unit 403 is used to query the starting global transaction sequence number or the logical sequence number within the starting log partition corresponding to each log partition based on the target snapshot information; the log shard list construction unit 404 is used to obtain the metadata that needs to be replayed between the starting global transaction sequence number and the target recovery time node in each log partition, or obtain the metadata that needs to be replayed between the logical sequence number within the starting log partition and the target recovery time node in each log partition, and construct the initial log shard list that needs to be replayed for each log partition based on the metadata; the recovery plan acquisition unit 405 is used to generate the recovery plan corresponding to each log partition based on the target snapshot information and the initial log shard list that needs to be replayed for each log partition.

[0070] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement the parallel recovery method and system for a distributed log memory database provided by any of the above-described method embodiments of the present invention.

[0071] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0072] Based on the above-described embodiment of a parallel recovery method and system for a distributed log memory database, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a parallel recovery method and system for a distributed log memory database according to any embodiment of the present invention.

[0073] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0074] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0075] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0076] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute the distributed log in-memory database parallel recovery method and system described in any of the above-described method embodiments of the present invention.

[0077] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0078] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A parallel recovery method for a distributed log-based in-memory database, characterized in that, include: The command logs generated in the in-memory database are physically isolated and stored according to log partitions, and the metadata of each log partition is recorded. Based on the pre-received recovery instructions and the metadata, a corresponding recovery plan is generated for each log partition; Multiple recovery threads are started in parallel, and the snapshot information of the corresponding log partition is pre-acquired in each recovery thread based on the recovery plan. The log fragments that need to be replayed are determined according to the recovery plan and the snapshot information, and the log fragments that need to be replayed are replayed to the corresponding log partitions to perform parallel recovery of the in-memory database.

2. The method for parallel recovery of a distributed log-based in-memory database as described in claim 1, characterized in that, The command logs generated in the in-memory database are physically isolated and stored according to log partitions, and the metadata of each log partition is recorded, including: Capture command logs generated during the execution of a predefined transaction; The command log is assigned a partition identifier and multi-dimensional data is embedded in it. The multi-dimensional data is then written into the log file or log stream of the corresponding log partition according to the partition identifier. Based on preset splitting rules, the log file or the log stream is split into several log fragments; Obtain the corresponding metadata from several of the aforementioned log shards, and associate them with the global transaction sequence number and the logical sequence number within the log partition; Based on the global transaction sequence number and the logical sequence number within the log partition, the metadata of each log partition is recorded according to preset partitioning rules.

3. The method for parallel recovery of a distributed log-based in-memory database as described in claim 2, characterized in that, The record includes metadata corresponding to several log shards, and associates them with global transaction sequence numbers and logical sequence numbers within log partitions, including: The log partition start and end index range, preset transaction association range, complete hash value of log fragment, and operation type summary of each log fragment are used as the metadata of the corresponding log fragment. Based on the preset transaction association range and the log partition start and end index range, associate the global transaction sequence number and the logical sequence number within the log partition.

4. The method for parallel recovery of a distributed log-based in-memory database as described in claim 3, characterized in that, The process of generating corresponding recovery plans for each log partition based on the pre-received recovery instructions and the metadata includes: Based on the pre-received recovery instructions, the target recovery time node and the initial snapshot information of each log partition are obtained. The initial snapshot information includes a timestamp and is composed of the snapshot information of each log fragment in each log partition. The initial snapshot information whose timestamp is less than or equal to the target recovery time node is used as the target snapshot information for each log partition; Based on the target snapshot information, query the starting global transaction sequence number or the starting logical sequence number within the log partition corresponding to each log partition. Obtain the metadata that needs to be replayed between the starting global transaction sequence number and the target recovery time node in each log partition, or obtain the metadata that needs to be replayed between the logical sequence number within the starting log partition and the target recovery time node in each log partition, and construct an initial list of log shards that need to be replayed for each log partition based on the metadata. Based on the target snapshot information corresponding to each log partition and the initial list of log shards to be replayed corresponding to each log partition, a recovery plan is generated for each log partition.

5. The method for parallel recovery of a distributed log-based in-memory database as described in claim 4, characterized in that, Based on the target snapshot information corresponding to each log partition and the initial list of log shards to be replayed corresponding to each log partition, a recovery plan is generated for each log partition, including: Obtain the complete hash value of each log fragment in the initial list of log fragments to be replayed; Based on the pre-distributed pre-inspection task, obtain the verification hash value of each log fragment in the pre-inspection task; If the complete hash value of each log shard matches the verification hash value, then each log shard is marked as verified. If the complete hash value of any log shard does not match the verification hash value, the log shard is marked as a corrupted log shard. Several replica log shards corresponding to the corrupted log shard are obtained, and pre-inspection tasks are distributed to several replica log shards in sequence. The hash value verification process is performed until the complete hash value of any replica log shard matches the verification hash value, and the target replica log shard is obtained. Update the corrupted log shard to the target replica log shard, and mark the target replica log shard as verified; When all log fragments in the initial list of log fragments to be replayed are marked as verified, the list of log fragments to be replayed after verification is obtained. Based on the pre-acquired recovery configuration file and the operation type summary of each log segment, the operation type of the log list to be replayed is checked, and log segments that do not meet the preset check requirements are removed to obtain the target log list to be replayed. Based on the target snapshot information corresponding to each log partition and the target log list that needs to be replayed corresponding to each log partition, a recovery plan is constructed for each log partition.

6. The method for parallel recovery of a distributed log-based in-memory database as described in claim 5, characterized in that, Before executing the step of constructing a recovery plan for each log partition based on the target snapshot information and the target log list to be replayed for each log partition, the following steps are also included: Based on the target replay log list corresponding to each log partition, obtain the start and end index range of the log partition or the preset transaction association range corresponding to each log partition. Based on the start and end index range of the log partition or the preset transaction association range corresponding to each log partition, the continuity of each log shard is verified in the target replay log list corresponding to each log partition. If the log shards all meet the preset continuity verification requirements, the recovery plan steps corresponding to each log partition are constructed based on the target snapshot information and the target replay log list corresponding to each log partition. If any of the log shards fails to meet the preset continuity verification requirements, the steps for constructing a recovery plan for each log partition based on the target snapshot information and the target log list to be replayed for each log partition will be suspended.

7. The method for parallel recovery of a distributed log-based in-memory database as described in claim 5, characterized in that, Multiple recovery threads are started in parallel, and based on the recovery plan, the pre-acquired snapshot information of the corresponding log partition is loaded in each recovery thread. The log fragments to be replayed are determined according to the recovery plan and the snapshot information, and the log fragments to be replayed are replayed to the corresponding log partitions to perform parallel recovery of the in-memory database, including: Multiple recovery threads are started in parallel, and a unique recovery plan is assigned to each of the recovery threads; Based on the recovery plan, the snapshot information of the corresponding log partition is loaded in each of the recovery threads, and the initial state of the corresponding log partition is re-established to obtain the initial log partition corresponding to each log partition. Based on the order of the log fragments in the target log list to be replayed, the log fragments are replayed to the corresponding initial log partitions in the order to obtain the recovery log partitions corresponding to each log partition; Once all log shards have completed the replay operation, a recovery thread completion report is output. If all recovery threads output a recovery thread completion report, the recovery hash value of the recovery log partition is calculated, and the recovery hash value is checked for consistency with a preset check value. After the check passes, the parallel recovery of the in-memory database is completed.

8. A parallel recovery system for a distributed log-based in-memory database, characterized in that, The method for parallel recovery of a memory database with a distributed log as described in any one of claims 1-7 includes: Data acquisition module, recovery plan generation module, and parallel recovery module; The data acquisition module is used to physically isolate and store the command logs generated in the memory database according to log partitions, and record the metadata of each log partition. The recovery plan generation module is used to generate corresponding recovery plans for each log partition based on the pre-received recovery instructions and the metadata. The parallel recovery module is used to start multiple recovery threads in parallel, load the pre-acquired snapshot information of the corresponding log partition in each recovery thread based on the recovery plan, determine the log fragments that need to be replayed according to the recovery plan and the snapshot information, and replay the log fragments that need to be replayed to the corresponding log partition, so as to perform parallel recovery of the in-memory database.

9. A parallel recovery system for a distributed log-based in-memory database as described in claim 8, characterized in that, The data acquisition module is used to physically isolate and store the command logs generated in the in-memory database according to log partitions, and record the metadata of each log partition, including: The system includes a command log acquisition unit, a data preprocessing unit, a data segmentation unit, a sequence number construction unit, and a metadata recording unit. The command log acquisition unit is used to capture the command log generated during the execution of a preset transaction; The data preprocessing unit is used to set partition identifiers for the command log and embed multi-dimensional data, and write the multi-dimensional data into the log file or log stream of the corresponding log partition according to the partition identifiers. The data cutting unit is used to cut the log file or the log stream into several log fragments based on a preset cutting rule; The sequence number construction unit is used to obtain the corresponding metadata in several log shards and associate the global transaction sequence number with the logical sequence number within the log partition; The metadata recording unit is used to record the metadata of each log partition based on the global transaction sequence number and the logical sequence number within the log partition, according to preset partitioning rules.

10. A distributed log-based in-memory database parallel recovery system as described in claim 9, characterized in that, The recovery plan generation module is used to generate corresponding recovery plans for each log partition based on the pre-received recovery instructions and the metadata, including: The system includes an initial snapshot information acquisition unit, a target snapshot information query unit, a serial number query unit, a log fragment list construction unit, and a recovery plan acquisition unit. The initial snapshot information acquisition unit is used to acquire the target recovery time node and the initial snapshot information of each log partition based on the pre-received recovery instruction. The initial snapshot information includes a timestamp and is composed of the snapshot information of each log fragment in each log partition. The target snapshot information query unit is used to query the initial snapshot information whose timestamp is less than or equal to the target recovery time node as the target snapshot information for each log partition; The sequence number query unit is used to query the starting global transaction sequence number or the starting log partition logical sequence number corresponding to each log partition based on the target snapshot information. The log shard list construction unit is used to obtain the metadata that needs to be replayed between the starting global transaction sequence number and the target recovery time node in each log partition, or to obtain the metadata that needs to be replayed between the logical sequence number in the starting log partition and the target recovery time node in each log partition, and construct an initial log shard list that needs to be replayed for each log partition based on the metadata. The recovery plan acquisition unit is used to generate a recovery plan for each log partition based on the target snapshot information corresponding to each log partition and the initial list of log shards to be replayed corresponding to each log partition.

Images