Method, system, and storage medium for processing a database

Abstract

This application discloses a method, system, and storage medium for processing a database, belonging to the field of cloud computing. The method includes: receiving a first log of a first transaction sent by a first computing node, the first log recording m processing operations; if the sum of the number of processing operations in a transaction of first snapshot data and the number of processing operations in n first transactions is less than or equal to a threshold, storing the information of the first snapshot data and the transactions of the first snapshot data in a first correspondence relationship, the first snapshot data recording the first database after the transactions based on the first snapshot data have been processed, and the n first transactions being the transactions between the transactions of the first snapshot data and the transactions of the last saved first snapshot data; and retrieving the first data in the first database before the recovery time based on the recovery time, the first log, and the first correspondence relationship. This application can improve the efficiency of database recovery.

Description

Technical Field

[0001] This application relates to the field of cloud computing, and in particular to a method, system, and storage medium for processing databases. Background Technology

[0002] A database system includes compute nodes and storage devices. The memory of the compute nodes contains the database, and the compute nodes can process the database. For example, a compute node can modify data in the database, save data to the database, or delete data from the database.

[0003] In related technologies, a compute node receives a transaction, which includes at least one processing operation. Based on this operation, the compute node processes the database, causing changes to the data in the database. Simultaneously, the compute node saves a log to a storage device, recording the at least one processing operation. The compute node's memory can temporarily store the database, while the storage device has the characteristic of persistent data storage. Therefore, when the data in the database changes, the compute node persistently stores the data in the database to the storage device. In related technologies, there is a need to recover data from a database at any point in time; however, in some cases, data recovery takes a long time, resulting in low recovery efficiency. Summary of the Invention

[0004] This application provides a method, system, and storage medium for processing databases to improve recovery efficiency. The technical solution is as follows:

[0005] In a first aspect, this application provides a method for processing a database. The method is applied to a database system, which includes a storage device and a first computing node. The first computing node receives a first transaction, which includes m processing operations for processing a first database, where m is an integer greater than or equal to 1. In the method, the storage device receives a first log of the first transaction sent by the first computing node. The first log records the m processing operations. If the sum of the number of processing operations in the first snapshot data transaction and the number of processing operations in the n first transactions is less than or equal to a threshold, the storage device stores the first snapshot data and the transaction information in a first correspondence relationship. The first snapshot data records the first database after the transaction based on the first snapshot data has been processed. The n first transactions are the intervals between the transactions in the first snapshot data and the transactions in the last saved first snapshot data, where n is an integer greater than or equal to 1. The storage device retrieves the first data in the first database prior to the recovery time based on the recovery time, the first log, and the first correspondence relationship.

[0006] Regarding the interval between two consecutive saving of the first snapshot data in the first correspondence, the total number of processing operations performed by the first computing node within this interval is less than or equal to a threshold value, denoted by A. Therefore, when using the first correspondence to restore the first database, the number of processing operations required is guaranteed not to exceed A. This ensures that the time required to restore the first data in the first database prior to the restoration time is less than or equal to the total time required to perform A processing operations, thereby shortening the time required to restore the database and improving the efficiency of database restoration.

[0007] In one possible implementation, the storage device obtains second snapshot data and transaction information of the second snapshot data based on the recovery time and a first correspondence, wherein the second snapshot data is the snapshot data in the first correspondence whose completion time is before the recovery time and is the closest to the recovery time. Based on the transaction information of the second snapshot data, the storage device obtains the first logs of z first transactions, where z is an integer greater than or equal to 1, and the completion times of the z first transactions are between the completion time and the recovery time of the transactions in the second snapshot data. Based on the second snapshot data and the first logs of the z first transactions, the storage device obtains the first data in the first database prior to the recovery time.

[0008] Since the completion time of the z first transactions is between the completion time and recovery time of the transactions in the second snapshot data, the number of processing operations obtained based on the first log records of the z first transactions does not exceed the number threshold A, thereby shortening the time to recover the first database.

[0009] In another possible implementation, the storage device receives a storage request from a first computing node. The storage request includes second data in a first database based on the completed first transaction and information about the first transaction. The storage device saves the second data in the first database and generates snapshot data, which is used to record the second data in the first database. The storage device obtains the number of processing operations included in x first transactions between the first transaction and the transaction that last saved the first snapshot data, where x is an integer greater than or equal to 1. If the cumulative value between the obtained number and m does not exceed a number threshold and the difference between the cumulative value and the number threshold does not exceed a difference threshold, the storage device determines the snapshot data as the first snapshot data and the first transaction as the transaction of the first snapshot data.

[0010] If the accumulated value is less than or equal to the number threshold A, the information of the first snapshot data and the transactions associated with the first snapshot data are stored in the first correspondence. Thus, based on the recovery time and the first correspondence, the time required for the storage device to recover the first database prior to the recovery time is less than or equal to the total time for processing A processing operations, thereby shortening the time required to recover the database and improving the efficiency of database recovery.

[0011] Furthermore, since the difference between the accumulated value and the number threshold A does not exceed the difference threshold, the storage device saves the first snapshot data, reducing the number of times the first snapshot data is saved and saving storage resources.

[0012] In another possible implementation, the database system further includes a second computing node. This second computing node receives a second transaction, which includes y processing operations for processing the second database, where y is an integer greater than or equal to 1. The storage device receives a second log of the second transaction sent by the second computing node. This second log records the y processing operations. If the sum of the number of processing operations in the third snapshot data transaction and the number of processing operations in k second transactions is less than or equal to a threshold, the storage device stores the information of the third snapshot data and its transactions in a second correspondence. The third snapshot data records the second database after processing based on the transactions in the third snapshot data. The k second transactions are the intervals between the transactions in the third snapshot data and the transactions in the last saved third snapshot data, where k is an integer greater than or equal to 1. Based on the recovery time, the second log, and the second correspondence, the storage device retrieves the third data in the second database prior to the recovery time. This ensures that the time required to recover the third data in the second database prior to the recovery time is less than or equal to the total time of executing A processing operations, thereby shortening the time required to recover the database and improving the efficiency of database recovery.

[0013] In another possible implementation, the database system also includes a management node. The first transaction and the second transaction are two sub-transactions included in the distributed transaction. The management node sends the first transaction to the first compute node and the second transaction to the second compute node at a first-time interval. After processing the first database based on the first transaction, the first compute node sends a first feedback message. After processing the second database based on the second transaction, the second compute node sends a second feedback message. Based on the first and second feedback messages, the management node sends acknowledgment messages to both the first and second compute nodes at a second-time interval. These acknowledgment messages indicate that the completion time of both the first and second transactions is the second-time interval. This ensures that the completion time of each sub-transaction within the distributed transaction is consistent, guaranteeing a consistent database state during the recovery of the database on each compute node and preventing data errors.

[0014] In another possible implementation, a first curve is obtained, which describes the relationship between the number of multiple processing operations and the time required to restore the first database. The multiple processing operations include w processing operations included in the first transaction (interval between the third and fourth transactions), and the processing operations included in the fourth transaction. The third and fourth transactions are two consecutive transactions that save the first snapshot data, with the third transaction preceding the fourth transaction. w is an integer greater than or equal to 1. A second curve is obtained, which describes the relationship between the number of multiple processing operations and the total amount of snapshot data saved within the storage period. The storage duration of the first snapshot data in the storage device is equal to the length of the storage period. Based on the first and second curves, a threshold for the number of operations is determined. This ensures that the threshold balances the time required to restore the database with the storage capacity required to store the first snapshot data, thus optimizing the performance of the database system.

[0015] Secondly, this application provides a database system for performing the methods of the first aspect or any possible implementation thereof. Specifically, the database system includes apparatus for performing the methods of the first aspect or any possible implementation thereof.

[0016] Thirdly, this application provides a computing device cluster, the computing device cluster including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method in the first aspect or any possible implementation of the first aspect.

[0017] Fourthly, this application provides a computer program product comprising computer program instructions stored in a computer-readable storage medium, wherein the computer program instructions are loaded by a processor to implement the method in the first aspect or any possible implementation thereof.

[0018] Fifthly, this application provides a computer-readable storage medium for storing computer program instructions, which are loaded by a processor to execute the methods described in the first aspect or any possible implementation thereof. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of a database system provided in an embodiment of this application;

[0020] Figure 2 This is a schematic diagram of another database system structure provided in an embodiment of this application;

[0021] Figure 3 This is a schematic diagram of another database system structure provided in an embodiment of this application;

[0022] Figure 4 This is a flowchart of a method for processing a database provided in an embodiment of this application;

[0023] Figure 5 This is a schematic diagram of another database system structure provided in an embodiment of this application;

[0024] Figure 6 This is a schematic diagram of another database system structure provided in an embodiment of this application;

[0025] Figure 7 This is a flowchart of another method for processing a database provided in an embodiment of this application;

[0026] Figure 8 This is a schematic diagram illustrating a method for storing snapshot data according to an embodiment of this application;

[0027] Figure 9 This is a schematic diagram illustrating a distributed transaction processing method provided in an embodiment of this application;

[0028] Figure 10 This is a flowchart of a method for obtaining a number threshold provided in an embodiment of this application;

[0029] Figure 11 This is a schematic diagram of a first curve provided in an embodiment of this application;

[0030] Figure 12 This is a schematic diagram of a second curve provided in an embodiment of this application;

[0031] Figure 13 This is a schematic diagram illustrating a method for determining a threshold number, as provided in an embodiment of this application.

[0032] Figure 14 This is a schematic diagram of another database system structure provided in an embodiment of this application;

[0033] Figure 15 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0034] Figure 16 This is a schematic diagram of a computing device cluster structure provided in an embodiment of this application;

[0035] Figure 17 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0036] See Figure 1This application provides a database system 100, which includes one or more computing nodes 101 and a storage device 102. Each computing node 101 includes a database.

[0037] For each computing node 101 in the database system 100, for ease of explanation, this computing node 101 is referred to as the first computing node, and the first computing node includes a first database. Optionally, the memory of the first computing node is used to store the first database. The first computing node can receive a first transaction, which includes m processing operations for processing the first database, where m is an integer greater than or equal to 1. The first computing node processes the first database based on the m processing operations included in the first transaction. The first computing node also sends a first log of the first transaction to the storage device 102, which is used to record the m processing operations. The storage device 102 receives the first log and saves it.

[0038] Optionally, the first computing node performs m processing operations on the first database to realize the processing of the first database based on the first transaction.

[0039] In some embodiments, the first database may be a relational database, a non-relational database, a key-value (KV) database, or a document database, etc.

[0040] In some embodiments, the one or more computing nodes 101 may be physical nodes, such as servers or terminal devices. Alternatively, the one or more computing nodes 101 may be virtual nodes, such as virtual machines or containers.

[0041] In some embodiments, storage device 102 may be a storage server, or a system including multiple storage servers, or a distributed storage system, etc.

[0042] In some embodiments, the m processing operations include one or more of the following: reading data, writing data, deleting data, or modifying data, etc. Alternatively, the m processing operations may include a combination operation obtained by combining some or all of the four operations.

[0043] The first computing node processes the first database based on the first transaction, which may cause changes to the data in the first database. The first computing node can continuously receive the first transaction, process the first database based on the first transaction, and continuously save the first log of the first transaction to storage device 102. Therefore, the data in the first database is constantly changing, and the data stored in the first database may be different at different times, that is, the first database may be different at different times.

[0044] The memory of the first computing node is temporary storage, while storage device 102 is a persistent storage device. Therefore, in order to avoid data loss, the first computing node can periodically or irregularly save the data in the first database to storage device 102.

[0045] Optionally, in implementation, the first computing node sends a storage request to the storage device 102, the storage request including data from the first database. The storage device 102 receives the storage request and saves the data from the first database included in the storage request.

[0046] The storage request may include the entire dataset of the first database, or it may include data that has changed in the first database. Regarding the last time data was saved to storage device 102, the changed data refers to data that has changed in the first database since that time.

[0047] Optionally, the storage request may include the full data of the first database, and storage device 102 may update the data in the locally stored first database to the full data of the first database. Alternatively, the storage request may include changed data in the first database, and storage device 102 may update the locally stored first database with the changed data. This ensures that the data in the first database in storage device 102 is consistent with the data in the first database in the memory of the first computing node.

[0048] The first computing node can send storage requests to the storage device 102 periodically or irregularly. For example, the first computing node can send storage requests to the storage device 102 periodically, or send a storage request to the storage device 102 whenever the first database is processed based on H first transactions, where H is an integer greater than or equal to 1.

[0049] Database system 100 can provide services to users. For example, if a user needs to restore data from a previous time in the first database, database system 100 can restore the first data from the first database before that time and provide the first data from the first database to the user.

[0050] For example, a user might need to restore data in the first database prior to 9:00 AM the day before yesterday. Database system 100 can receive a restore request message that includes the restore time of 9:00 AM the day before yesterday. Database system 100 can then restore the first data in the first database prior to 9:00 AM the day before yesterday and provide the user with that first data.

[0051] To enable the recovery of data from the first database at any point in time to meet user needs, the following method is provided:

[0052] (1): When the first computing node discovers that the number of data changes in the first database after the last checkpoint operation has reached the threshold, it sends a checkpoint trigger request to the storage device 102. The checkpoint trigger request includes the first time, which is the current checkpoint time.

[0053] Alternatively, a data item can be data stored in a memory page of the first compute node.

[0054] (2): The storage device 102 receives the checkpoint trigger request, obtains the data in the first database at the first time, takes the first time as a checkpoint time, and stores the data in the first database and the checkpoint time in the correspondence between data and checkpoint time.

[0055] During the time period of executing step (2), the first computing node of storage device 102 may suspend providing services to users, and will continue to provide services to users after executing step (2).

[0056] The operation in step (2) can also be called the checkpoint operation, which is triggered by the first computing node in this method.

[0057] After the checkpoint time, if the number of data changes in the first database reaches the threshold, the first computing node will repeat the above steps (1) and (2).

[0058] (3): The storage device 102 receives a recovery request message, which includes a recovery time, obtains the checkpoint time that is before the recovery time and closest to the recovery time from the correspondence, and obtains the second data in the first database corresponding to the checkpoint time from the correspondence.

[0059] The second data in the first database is the data stored in the first database at that checkpoint time.

[0060] (4): Storage device 102 retrieves the first log of multiple first transactions completed between the checkpoint time and the recovery time from the saved first log.

[0061] (5): Storage device 102 processes the second data in the first database based on the operation set to obtain the first data in the first database before the recovery time, the operation set including the processing operations of the first log records of the plurality of first transactions.

[0062] This involves obtaining the first data in the first database prior to the recovery time, meaning the first database prior to the recovery time has been recovered. Since the execution time required for each processing operation by storage device 102 is the same or approximately the same, the majority of the time required to recover the first database is the total time spent executing each processing operation in this set of operations. Therefore, the more processing operations included in the set of operations, the longer the recovery time for the first database; conversely, the fewer processing operations included in the set of operations, the shorter the recovery time for the first database.

[0063] Although the database system 100 provides the operations of steps (1)-(5) to restore the first data in the first database before the recovery time, which can realize the recovery of the first database before any time, the set of operations may include a large number of processing operations, resulting in a long recovery time for the first database and reducing the efficiency of the recovery of the first database.

[0064] This is mainly due to the following reasons: After the previous checkpoint time, the first compute node receives a large number of first transactions. These first transactions include a very large number of processing operations. However, these very large processing operations may frequently modify a small portion of the data in the first database, resulting in a small number of data changes in the first database after the previous checkpoint time.

[0065] For example, this very large processing operation might frequently modify Q memory pages in the first compute node's memory, where Q is an integer greater than or equal to 1, while the number of data changes in the first database is also Q. This results in a long time after the previous checkpoint before the number of data changes in the first database reaches the threshold, at which point the first compute node triggers a checkpoint operation on storage device 102. In other words, a long time interval exists between the previous checkpoint and the current checkpoint, during which a large number of processing operations included in the first transaction are executed.

[0066] If the recovery time provided by the user is between the previous checkpoint time and the current checkpoint time, this may result in the acquired set of operations including a large number of processing operations, causing the time required to recover the first database to be very long.

[0067] To reduce the time required to restore the first database, embodiments of this application provide, as follows: Figure 4 or Figure 7 The method shown will not be explained in detail here.

[0068] In some embodiments, the database system 100 may include a plurality of computing nodes 101, each computing node 101 including a database and capable of performing the same operations as the first computing node. Optionally, the memory of each computing node 101 is used to store the database.

[0069] For example, see Figure 2 The database system 100 may further include a second computing node. The second computing node stores a second database in its memory and can perform the same operations as the first computing node. For example, the second computing node can receive a second transaction, which includes y processing operations for processing the second database, where y is an integer greater than or equal to 1. The second computing node processes the second database based on the y processing operations included in the second transaction. The second computing node also sends a second log of the second transaction to the storage device 102, which records the y processing operations. The storage device 102 receives and saves the second log.

[0070] Optionally, the second computing node performs y processing operations on the second database to realize the second database based on the second transaction processing.

[0071] In some embodiments, see Figure 3 The database system 100 may also include a management node 103. The first transaction and the second transaction are two sub-transactions of a distributed transaction. The management node 103 can receive a distributed transaction including the first transaction and the second transaction, and simultaneously send the first transaction to the first computing node and the second transaction to the second computing node.

[0072] In some embodiments, the management node 103 may be a physical node, such as a server or terminal device. Alternatively, the management node 103 may be a virtual node, such as a virtual machine or container.

[0073] The following examples will illustrate in detail how to shorten the time required to restore a database, thereby improving the efficiency of database restoration.

[0074] See Figure 4 This application provides a method 400 for processing a database, which can be applied to... Figure 1 , Figure 2 or Figure 3 The database system 100 shown.

[0075] In method 400, a first computing node includes a first database. The first computing node receives a first transaction, which includes m processing operations for processing the first database, where m is an integer greater than or equal to 1. Based on the m processing operations included in the first transaction, the first computing node processes the first database and sends a first log of the first transaction to a storage device. The first log is used to record the m processing operations.

[0076] Step 401: The storage device receives the first log of the first transaction sent by the first computing node.

[0077] In step 401, the storage device saves the first log of the first transaction.

[0078] In this process, the first computing node continuously receives the first transaction, processes the first database based on the first transaction, and sends the first log of the first transaction to the storage device. Therefore, the storage device may continuously receive and save the first log of the first transaction.

[0079] Step 402: If the sum of the number of processing operations included in the first snapshot data transaction and the number of processing operations included in n first transactions is less than or equal to the number threshold, the storage device saves the first snapshot data and the transaction information of the first snapshot data in the first correspondence relationship.

[0080] The first snapshot data is used to record the first database after the transaction based on the first snapshot data is completed. The n first transactions are the transactions between the transactions of the first snapshot data and the transactions of the last saved first snapshot data, where n is an integer greater than or equal to 1.

[0081] The transaction for the first snapshot data is a first transaction used by the first computing node to process the first database. After the first database is processed based on this first transaction, a snapshot can be taken of the processed first database to obtain the first snapshot data used to record the processed first database.

[0082] Step 403: The storage device retrieves the first data in the first database prior to the recovery time based on the recovery time, the first log, and the first correspondence.

[0083] Since the sum of the number of processing operations in the first snapshot data transaction and the number of processing operations in n first transactions is less than or equal to a threshold value (let A represent the threshold value), and the information of the first snapshot data and its transactions is stored in a first correspondence, the number of processing operations required to restore the first database using the first correspondence value does not exceed the threshold value A. Thus, the time required to restore the first data in the first database prior to the restoration time is less than or equal to the total time required to execute A processing operations, thereby shortening the time required to restore the database and improving the efficiency of database restoration.

[0084] For step 401 in method 400, see [link to relevant documentation]. Figure 5 After the first computing node completes processing the first database based on the received first transaction, it can save the first log of the first transaction to the storage device, that is, send the first log of the first transaction to the storage device. The storage device receives the first log of the first transaction and saves it.

[0085] In addition to recording the m processing operations included in the first transaction, the first log may also record information about the first transaction.

[0086] Optionally, the information for the first transaction includes the transaction number of the first transaction. The transaction number of the first transaction may be mapped to its completion time; that is, the completion time of the first transaction can be derived from its transaction number. The completion time of the first transaction refers to the time it takes for the first database to be processed by the first transaction.

[0087] Optionally, the transaction number of the first transaction currently received by the first computing node is obtained by incrementing the transaction number of the previously received first transaction. For example, the first computing node can initialize the transaction number of the first received transaction, assuming it is initialized to 0, i.e., the transaction number of the first transaction is 0. When the first computing node receives the second transaction, it increments the transaction number of the first transaction by 1, resulting in a transaction number of 1 for the second transaction. When it receives the third transaction, it increments the transaction number of the second transaction by 1, resulting in a transaction number of 2 for the third transaction, and so on. Optionally, the first computing node can schedule the received first transactions in order of transaction number and process the first database based on the scheduled first transactions.

[0088] Optionally, the information for the first transaction may also include the completion time of the first transaction.

[0089] The first transaction may be a sub-transaction of a distributed transaction. Alternatively, the first transaction may not be a sub-transaction of a distributed transaction; it may be an independent transaction.

[0090] Distributed transactions refer to transactions that require the collaboration of multiple computing nodes. Therefore, distributed transactions include sub-transactions that need to be completed on multiple computing nodes.

[0091] For example, suppose a distributed transaction is used to implement a transfer, where a amount is deducted from one user's account and added to another user's account. A distributed transaction can include two sub-transactions, which, for ease of explanation, will be referred to as the first transaction and the second transaction. The first transaction deducts the amount from one user's account, and the second transaction adds the amount to another user's account.

[0092] Assume that one user's account is data in the first database of the first computing node, and the other user's account is data in the second database of the second computing node. Thus, the first computing node, based on the first database and a first transaction, deducts an amount from the account of the user; and the second computing node, based on the second database and a second transaction, increases an amount from the account of the user.

[0093] An independent transaction is one that can be completed with just one computing node.

[0094] In some embodiments, for each first transaction processed by the first computing node, the first log of each first transaction can be stored in the storage device in order of completion time. Therefore, the storage device stores a first log sequence, which includes the first logs of each first transaction arranged in order of completion time.

[0095] Regarding step 402 of method 400, the first computing node can periodically or irregularly save data from the first database to the storage device. This can be achieved through the following processes 4021-4026.

[0096] 4021: After the first computing node finishes processing the first database based on the received first transaction, it sends a storage request to the storage device. The storage request includes the second data in the current first database and the information of the first transaction.

[0097] Optionally, the second data may be the full set of data in the first database, meaning the second data is all the data stored in the first database. Alternatively, the second data may be changed data in the first database, meaning the data in the first database has changed since the last time data was saved to the storage device.

[0098] Optionally, the first computing node may periodically send storage requests to the storage device, or send a storage request to the storage device whenever the first database is processed based on H first transactions, where H is an integer greater than or equal to 1.

[0099] 4022: The storage device receives a storage request, saves the second data in the first database and generates snapshot data. The snapshot data is used to record the second data in the first database, and the information of the first transaction included in the storage request is used as the transaction information of the snapshot data.

[0100] Optionally, the snapshot data may include indexes of changed data and unchanged data in the first database, which are used to redirect to the unchanged data in the first database.

[0101] After a storage device generates snapshot data, it can temporarily save the snapshot data and the transaction information of the snapshot data for a period of time. After that period of time, the snapshot data and the transaction information of the snapshot data may be deleted.

[0102] For example, after a storage device generates snapshot data, it may temporarily save the snapshot data and related transaction information for one hour. After one hour, it may delete the snapshot data and related transaction information. One hour is just an example; in actual implementation, it may save for two, three, or four hours, etc. The specific storage duration can be configured based on requirements.

[0103] During this period, the storage device may receive storage requests from the first compute node at different times, so the storage device may temporarily store one or more snapshot data and transaction information of the one or more snapshot data.

[0104] For example, see Figure 5 After receiving a storage request, the storage device saves the second data in the first database from the storage request and generates snapshot data Tree9. Snapshot data Tree9 is used to record the second data in the first database and temporarily saves snapshot data Tree9 and the transaction information of snapshot data Tree9 (not shown in the figure). Snapshot data Tree7 and snapshot data Tree8 are generated before snapshot data Tree9 and are temporarily stored in the storage device.

[0105] 4023: The number of processing operations included in x first transactions between the storage device acquiring the first transaction and the transaction that last saved the first snapshot data, where x is an integer greater than or equal to 1.

[0106] In 4023, the first transaction acquired by the storage device is a transaction of any snapshot data temporarily saved by the storage device.

[0107] The storage device can retrieve information about the transaction that last saved the first snapshot data. Based on the information of the first transaction and the transaction that last saved the first snapshot data, the first logs of x first transactions, representing the interval between the first transaction and the transaction that last saved the first snapshot data, are retrieved from the first logs of the saved first transactions. The number of processing operations recorded in the first logs of these x first transactions is counted to obtain the total number of processing operations included in these x first transactions.

[0108] For example, see Figure 5 Assume the first transaction acquired by the storage device is the transaction of temporarily saved snapshot data Tree8, and the last saved first snapshot data is snapshot data Tree6. Based on the information of the transactions of snapshot data Tree8 and snapshot data Tree6, retrieve the first logs of x first transactions that occurred between the transactions of snapshot data Tree8 and snapshot data Tree6 from the first logs of the already saved first transactions. Count the number of processing operations recorded in the first logs of these x first transactions to obtain the total number of processing operations included in these x first transactions.

[0109] Optionally, the storage device locates the first log of the transaction that last saved the first snapshot data and the first log of the first transaction in the first log sequence based on the transaction information and the first transaction information of the first transaction. It then retrieves the first logs of x first transactions located between the first log of the transaction that last saved the first snapshot data and the first log of the first transaction. The completion times of these x first transactions are between the completion time of the transaction that last saved the first snapshot data and the completion time of the first transaction.

[0110] Optionally, the x first transactions are transactions that are capable of modifying data in the first database.

[0111] 4024: The storage device obtains the number m of processing operations included in the first transaction, and obtains the cumulative value between the number m and the number of processing operations included in the x first transactions.

[0112] 4025: If the accumulated value does not exceed the number threshold and the difference between the accumulated value and the number threshold does not exceed the difference threshold, the storage device determines the snapshot data as the first snapshot data and the first transaction as the transaction of the first snapshot data.

[0113] At this point, x equals n.

[0114] 4026: The storage device stores the first snapshot data and the transaction information of the first snapshot data in the first correspondence relationship.

[0115] If the difference between the accumulated value and the number threshold does not exceed the difference threshold, the snapshot data is determined as the first snapshot data. This reduces the number of times the first snapshot data is saved, thereby reducing the storage resources required to store the first correspondence.

[0116] For example, the storage device acquires snapshot data Tree8, which includes the number m of processing operations in the transaction. The accumulated value between this number m and the number of processing operations included in the first x transactions is obtained. This accumulated value does not exceed a number threshold, and the difference between this accumulated value and the number threshold does not exceed a difference threshold. Therefore, see... Figure 6 The storage device stores the snapshot data Tree8 and the transaction information of snapshot data Tree8 in the first correspondence.

[0117] The storage duration of the first snapshot data in the first correspondence can be equal to the length of a storage cycle. That is, when the storage duration of the first snapshot data on the storage device reaches the length of a storage cycle, the first snapshot data and the transaction information of the first snapshot data are deleted from the first correspondence.

[0118] For example, see Figure 5 When the information of snapshot data Tree0 and its transaction in the first correspondence reaches the length of a storage cycle in the storage device, the information of snapshot data Tree0 and its transaction in the first correspondence is deleted from the first correspondence.

[0119] Optionally, the storage period can be one month, two months, one quarter, six months, or one year, etc. The length of the storage period can be configured based on requirements.

[0120] From the process described above (4021-4026), the first snapshot data saved to the first correspondence is the snapshot data temporarily saved in the storage device, rather than the newly generated snapshot data. This can reduce the resources consumed in generating the first snapshot data (such as computing power resources and / or input / output (IO) resources).

[0121] Furthermore, considering the processes described in 4021-4026 above, the storage device does not require the first compute node to trigger the checkpointing operation. Instead, the storage device itself triggers the checkpointing operation to save the first snapshot data and its transaction information in the first correspondence. This way, the first compute node remains unaware of the checkpointing operation, thus avoiding any interruption of service to users and preventing disruption to service operation.

[0122] For step 403 in method 400, the storage device receives a recovery request message, which includes a recovery time, and then retrieves the first data in the first database prior to the recovery time according to the following operations 4031-4033.

[0123] 4031: The storage device obtains information about the second snapshot data and the transaction of the second snapshot data based on the recovery time and the first correspondence, wherein the completion time of the transaction of the second snapshot data is before the recovery time and is closest to the recovery time in the first correspondence.

[0124] In step 4031, based on the transaction information of each snapshot data in the first correspondence, the completion time of the transaction for each snapshot data is obtained. Based on the completion time of each snapshot data, the information of the transaction whose completion time is before the recovery time and is closest to the recovery time is obtained. The snapshot data corresponding to the obtained transaction is used as the second snapshot data, thus obtaining the transaction information of the second snapshot data and the second snapshot data.

[0125] Optionally, the transaction information in the snapshot data may include the completion time of the transaction, which can be obtained from the transaction information in the snapshot data. Alternatively,

[0126] Optionally, the transaction information in the snapshot data may include the transaction number of the transaction, which has a mapping relationship with the completion time of the transaction, and the completion time of the transaction is obtained based on the transaction number.

[0127] 4032: The storage device obtains the first logs of z first transactions based on the transaction information of the second snapshot data, where z is an integer greater than or equal to 1, and the completion time of the z first transactions is between the completion time and recovery time of the transactions in the second snapshot data.

[0128] In some embodiments, for each first transaction included in the first log sequence in the storage device, the first log of the first transaction includes information about the first transaction, so the completion time of each first transaction in the first log sequence can be obtained based on the information of each first transaction in the first log sequence.

[0129] Therefore, in 4032, based on the completion time and recovery time of the transaction in the second snapshot data and the completion time of each first transaction in the first log sequence, the first log of the z first transactions whose completion time is between the completion time and recovery time of the transaction in the second snapshot data is obtained from the first log sequence.

[0130] 4033: The storage device retrieves the first data in the first database prior to the recovery time based on the second snapshot data and the first log of the z first transactions.

[0131] In 4033, the storage device can derive the data in the first database recorded by the second snapshot data based on the second snapshot data. An operation set is derived based on the first logs of z first transactions, and this operation set includes the processing operations included in the z first transactions. Based on the processing operations included in the operation set, the data in the first database is processed to obtain the first data in the first database before the recovery time.

[0132] The second snapshot data includes indexes of changed and unchanged data in the first database. Based on the indexes of unchanged data, the unchanged data can be derived. Based on the changed and unchanged data, the data in the first database recorded in the second snapshot data can be derived.

[0133] In this embodiment, the storage device acquires the first snapshot data within an interval of n first transactions between the transaction that acquired the first snapshot data and the transaction that previously saved the first snapshot data. The accumulated value between the number of processing operations in the transaction that acquired the first snapshot data and the number of processing operations in the n first transactions is used. If this accumulated value is less than or equal to a number threshold A and the difference between the accumulated value and the number threshold does not exceed a difference threshold, the information of the first snapshot data and the transactions associated with the first snapshot data is stored in a first correspondence. Thus, based on the recovery time and the first correspondence, the storage device acquires the second snapshot data and the transaction information of the second snapshot data. Based on the transaction information of the second snapshot data, it acquires the first logs of z first transactions. Based on the first logs of the z first transactions, it derives an operation set, ensuring that the number of processing operations in this operation set does not exceed the number threshold A. Therefore, the time required to recover the first data in the first database before the recovery time is less than or equal to the total time of executing A processing operations, thereby shortening the time required to recover the database and improving the efficiency of database recovery.

[0134] See Figure 7 This application provides a method 700 for processing a database, which can be applied to... Figure 1 , Figure 2 or Figure 3 The database system 100 shown.

[0135] In method 700, the database system 100 includes multiple computing nodes; for example, the database system 100 also includes a second computing node. The second computing node includes a second database. The second computing node receives a second transaction, which includes y processing operations for processing the second database, where y is an integer greater than or equal to 1. Based on the y processing operations included in the second transaction, the second computing node processes the second database and sends a second log of the second transaction to a storage device. The second log records the y processing operations.

[0136] Step 701: The storage device receives the second log of the second transaction sent by the second computing node.

[0137] In step 701, the storage device saves the second log of the second transaction.

[0138] The second computing node continuously receives the second transaction, processes the second database based on the second transaction, and sends the second transaction's second log to the storage device. Therefore, the storage device may continuously receive and save the second transaction's second log.

[0139] In some embodiments, for each second transaction processed by the second computing node, the second logs of each second transaction can be stored in the storage device in order of their completion time. Therefore, the storage device maintains a second log sequence, which includes the second logs of each second transaction arranged in order of completion time.

[0140] Step 702: If the sum of the number of processing operations included in the transaction of the third snapshot data and the number of processing operations included in the k second transactions is less than or equal to the number threshold, the storage device saves the information of the third snapshot data and the transaction of the third snapshot data in the second correspondence.

[0141] The third snapshot data is used to record the second database after the transaction based on the third snapshot data is completed. The k second transactions are the transactions between the transactions of the third snapshot data and the transactions of the last saved third snapshot data, where k is an integer greater than or equal to 1.

[0142] The transaction for the third snapshot data is a third transaction used by the second computing node to process the second database. After the second database is processed based on this third transaction, a snapshot can be taken of the processed second database to obtain the third snapshot data used to record the processed second database.

[0143] The detailed implementation process of step 702 can be found in the relevant content of step 402 in the above method 400, and will not be described in detail here.

[0144] Step 703: The storage device retrieves the third data in the second database prior to the recovery time based on the recovery time, the second log, and the second correspondence.

[0145] The detailed implementation process of step 703 can be found in the relevant content of step 403 in the above method 400, and will not be described in detail here.

[0146] See Figure 8For each computing node in the database system (such as the first computing node and the second computing node), the same number threshold is used for storage devices. This means that the higher the activity level of the services running on the computing nodes, the higher the frequency of transactions received by the computing nodes, the higher the frequency of transaction-based database processing, and the higher the frequency of the computing nodes saving the correspondence between snapshot data and transaction information. Conversely, the lower the activity level of the services running on the computing nodes, the lower the frequency of transactions received by the computing nodes, the lower the frequency of transaction-based database processing, and the lower the frequency of the computing nodes saving the correspondence between snapshot data and transaction information.

[0147] In this way, when the storage device restores the database of each compute node, the number of processing operations included in the operation set obtained for each compute node does not exceed the number threshold, thus ensuring that the time required to obtain the database of each compute node before the recovery time is not significantly different.

[0148] The recovery performance of a database system depends on the computing node that takes the longest time to recover the database. In other words, the database system also suffers from the "weakest link" effect, as described in the "law of the weakest link". However, the method in this application makes the time required to obtain the database from each computing node before the recovery time not much different, thus overcoming the weakest link effect and improving the recovery performance of the database system.

[0149] In some embodiments, the first transaction and the second transaction may be two sub-transactions included in a distributed transaction. When obtaining the first data in the first database and the third data in the second database before the recovery time, the operation set corresponding to the first computing point needs to include the processing operations in the first transaction of the distributed transaction, and the operation set corresponding to the second computing node needs to include the processing operations in the second transaction of the distributed transaction. This ensures that the recovered state of the first database and the second database is consistent and that no data errors occur.

[0150] For example, taking distributed transactions for fund transfers as an example, a transfer involves deducting money from one user's account and adding money to another user's account. The first transaction of the distributed transaction is used to deduct money from the first user's account, and the second transaction is used to add money to the second user's account.

[0151] If the completion time of the first transaction and the completion time of the second transaction are inconsistent, for example, assuming the completion time of the first transaction is before the recovery time and the completion time of the second transaction is after the recovery time, when restoring the first data in the first database, the transactions whose completion time is before the recovery time include the first transaction, and the set of operations obtained includes the processing operations in the first transaction, then in the first data of the restored first database, the amount in a user's account has been reduced.

[0152] However, when restoring the third data in the second database, the second transaction was not included in the transactions whose completion time preceded the restoration time, and the set of operations retrieved did not include the processing operations from the second transaction. Consequently, in the restored third data in the second database, the amount in another user's database was not increased. This resulted in an inconsistency between the restored first and second databases, leading to data errors.

[0153] To solve this problem, it is necessary to ensure that the completion times of all sub-transactions within the same distributed transaction are consistent. This ensures that when retrieving transactions that occurred before the recovery time, all sub-transactions within the distributed transaction can be retrieved. See [link / reference needed] for details. Figure 9 The database system also includes a management node, which receives distributed transactions, including a first transaction and a second transaction. The first transaction is sent to the first computing node and the second transaction is sent to the second computing node at the first moment.

[0154] The first computing node receives the first transaction, processes the first database based on the first transaction, and sends the first feedback message to the management node after processing the first database based on the first transaction.

[0155] The second computing node receives the second transaction, processes the second database based on the second transaction, and sends a second feedback message to the management node after processing the second database based on the second transaction.

[0156] The management node receives a first feedback message and a second feedback message. Based on these messages, it determines that the first computing node has completed processing the first database based on the first transaction, and the second computing node has completed processing the second database based on the second transaction. At a second time point, it sends confirmation messages to both the first and second computing nodes. These confirmation messages indicate that the completion time of both the first and second transactions is the second time point. Optionally, the confirmation message may include the second time point.

[0157] The first computing node receives the confirmation message and sets the completion time of the first transaction to the second time. The second computing node receives the confirmation message and sets the completion time of the second transaction to the second time. This ensures that the information for both the first and second transactions includes the same completion time.

[0158] In this embodiment, since the sum of the number of processing operations in the third snapshot data transaction and the number of processing operations in the k second transactions is less than or equal to a threshold A, the information of the third snapshot data and the transactions of the third snapshot data is stored in the second correspondence. Therefore, when using the second correspondence to restore the second database, the number of processing operations required does not exceed the threshold A. Thus, the time required to restore the third data in the second database before the restoration time is less than or equal to the total time of performing A processing operations, thereby shortening the time required to restore the database and improving the efficiency of database restoration.

[0159] See Figure 10 Regarding the aforementioned number threshold, this application provides a method 1000 for obtaining the number threshold, which includes the following process.

[0160] Step 1001: Obtain the first curve, which describes the relationship between the number of multiple processing operations and the time required to restore the first database.

[0161] The multiple processing operations include the processing operations included in the first transaction (within w intervals between the third and fourth transactions) and the processing operations included in the fourth transaction. The third and fourth transactions are two consecutive transactions that save the first snapshot data, with the third transaction preceding the fourth transaction, and w being an integer greater than or equal to 1.

[0162] The number of these multiple processing operations is the threshold used by the storage device to store the first snapshot data to the first correspondence. For example... Figure 11 As shown, multiple thresholds can be configured. The smaller the threshold, the more frequently the storage device stores the first snapshot data to the first correspondence, and the shorter the time required to restore the database. Conversely, the larger the threshold, the more frequently the storage device stores the first snapshot data to the first correspondence, and the longer the time required to restore the database.

[0163] For each threshold, the first snapshot data and the transaction information of the first snapshot data can be stored in the first correspondence according to the process described in 4021-4026 above, so that the first correspondence corresponding to each threshold can be obtained.

[0164] For each threshold, based on the first correspondence corresponding to that threshold, the saved first log, and the recovery time, the first data in the first database before the recovery time is obtained according to the process described in 4031-4033 above, and the time length corresponding to that threshold is obtained. Then, based on the time length corresponding to each threshold, a first curve is fitted.

[0165] For example, see Figure 11The configured thresholds include 5, 10, 20, 30, 60, 120, 240, 360, 720, and 1440. Then, the time length corresponding to threshold 5, threshold 10, ..., and threshold 1440 are obtained. Based on the time lengths corresponding to threshold 5, threshold 10, ..., and threshold 1440, a time frame is fitted. Figure 11 The first curve shown.

[0166] Step 1002: Obtain the second curve, which describes the relationship between the number of multiple processing operations and the total amount of snapshot data saved within the storage period. The storage duration of the first snapshot data in the storage device is equal to the length of the storage period.

[0167] In step 1001, the first correspondence relationship corresponding to each threshold is obtained. The data volume of the first correspondence relationship corresponding to each threshold is counted to obtain the total data volume of snapshot data saved within the storage period corresponding to each threshold, that is, the total data volume corresponding to each threshold. Based on the total data volume corresponding to each threshold, a second curve is fitted.

[0168] For example, see Figure 12 Obtain the total data volume corresponding to threshold 5, threshold 10, ..., threshold 1440. Based on the total data volume corresponding to threshold 5, threshold 10, ..., threshold 1440, fit... Figure 12 The second curve shown.

[0169] Step 1003: Determine the number threshold based on the first curve and the second curve.

[0170] In step 1003, the intersection point between the first curve and the second curve is obtained, and the threshold value near this intersection point is used as the count threshold. For example, see... Figure 13 The intersection of the first curve and the second curve is obtained, and the threshold of 60 located near the intersection is used as the number threshold.

[0171] This ensures that the number threshold takes into account both the time required to restore the first database and the storage capacity required to store the first snapshot data, thus optimizing the performance of the database system.

[0172] See Figure 14 This application provides a database system 1400, which can be used for... Figure 1 , Figure 2 or Figure 3 The database system 100 shown, or the database system 1400, can be... Figure 4 Method 400 or shown Figure 7The database system used in the method 700 shown. The database system 1400 includes a storage device 1401 and a first computing node 1402;

[0173] The first computing node 1402 is used to receive the first transaction, which includes m processing operations for processing the first database, where m is an integer greater than or equal to 1, and to send the first log of the first transaction to the storage device 1401. The first log is used to record the m processing operations.

[0174] Storage device 1401 is used to store the information of the first snapshot data and the transactions of the first snapshot data in a first correspondence relationship when the sum of the number of processing operations included in the transaction of the first snapshot data and the number of processing operations included in n first transactions is less than or equal to a number threshold. The first snapshot data is used to record the first database after the transaction based on the first snapshot data is completed. The n first transactions are the transactions between the transactions of the first snapshot data and the transactions of the last saved first snapshot data, where n is an integer greater than or equal to 1.

[0175] Storage device 1401 is also used to retrieve first data in a first database prior to the recovery time, based on the recovery time, the first log, and the first correspondence.

[0176] Optionally, the first computing node 1402 receives the first transaction and sends the first log of the first transaction to the storage device 1401. For details on this process, please refer to [link to relevant documentation]. Figure 4 The relevant content in step 401 of method 400 shown will not be described in detail here.

[0177] Optionally, for details on how storage device 1401 stores the first snapshot data and the transaction information of the first snapshot data in the first correspondence, see [link to detailed implementation]. Figure 4 The relevant content in step 402 of method 400 shown will not be described in detail here.

[0178] Optionally, for details on how storage device 1401 retrieves the first data in the first database prior to the recovery time based on the recovery time, the first log, and the first correspondence, please refer to [link to documentation]. Figure 4 The relevant content in step 403 of method 400 shown will not be described in detail here.

[0179] Optionally, storage device 1401 is used for:

[0180] Based on the recovery time and the first correspondence, the second snapshot data and the transaction information of the second snapshot data are obtained. The second snapshot data is the snapshot data of the transaction in the first correspondence whose completion time is before the recovery time and is closest to the recovery time.

[0181] Based on the transaction information of the second snapshot data, obtain the first logs of z first transactions, where z is an integer greater than or equal to 1, and the completion time of the z first transactions is between the completion time and recovery time of the transactions in the second snapshot data;

[0182] Based on the second snapshot data and the first log of the z first transactions, retrieve the first data in the first database before the recovery time.

[0183] Optionally, the storage device 1401 obtains the second snapshot data and the transaction information of the second snapshot data based on the recovery time and the first correspondence. For details on this process, please refer to [link to relevant documentation]. Figure 4 The relevant content in step 4031 of method 400 shown will not be described in detail here.

[0184] Optionally, for details on how storage device 1401 obtains the first logs of z first transactions based on transaction information from the second snapshot data, please refer to [link to relevant documentation]. Figure 4 The relevant content in step 4032 of method 400 shown will not be described in detail here.

[0185] Optionally, storage device 1401 obtains the first data in the first database before the recovery time based on the second snapshot data and the first logs of the z first transactions. For a detailed implementation process, see [link to documentation]. Figure 4 The relevant content in step 4033 of method 400 shown will not be described in detail here.

[0186] Optionally, storage device 1401 is also used for:

[0187] Receive a storage request sent by the first computing node. The storage request includes second data in the first database based on the first transaction and information of the first transaction.

[0188] Save the second data in the first database and generate snapshot data. The snapshot data is used to record the second data in the first database.

[0189] The number of processing operations included in x first transactions between the first transaction that retrieves the first transaction and the transaction that last saved the first snapshot data, where x is an integer greater than or equal to 1;

[0190] If the cumulative value between the number of items obtained and m does not exceed the number threshold and the difference between the cumulative value and the number threshold does not exceed the difference threshold, the snapshot data is determined as the first snapshot data, and the first transaction is determined as the transaction of the first snapshot data.

[0191] Optionally, for details on how storage device 1401 receives storage requests sent by the first computing node, please refer to [link to relevant documentation]. Figure 4 The relevant content in step 4022 of method 400 shown will not be described in detail here.

[0192] Optionally, for details on how storage device 1401 saves the second data in the first database and generates snapshot data, please refer to [link to relevant documentation]. Figure 4 The relevant content in step 4022 of method 400 shown will not be described in detail here.

[0193] Optionally, for details regarding the number of processing operations included in the x-th first transaction between the storage device 1401 acquiring the first transaction and the transaction that last saved the first snapshot data, please refer to [link to relevant documentation]. Figure 4 The relevant content in step 4023 of method 400 shown will not be described in detail here.

[0194] Optionally, storage device 1401 determines the snapshot data as the first snapshot data. For details on the implementation process of determining the first transaction as the first snapshot data, please refer to [link to relevant documentation]. Figure 4 The relevant content in steps 4024-4025 of method 400 shown will not be described in detail here.

[0195] Optionally, the database system 1400 further includes a second computing node 1403;

[0196] The second computing node 1403 is used to receive the second transaction, which includes y processing operations for processing the second database, where y is an integer greater than or equal to 1, and to send the second log of the second transaction to the storage device 1401. The second log is used to record the y processing operations.

[0197] Storage device 1401 is further configured to store the information of the third snapshot data and the transactions of the third snapshot data in a second correspondence when the sum of the number of processing operations included in the transactions of the third snapshot data and the number of processing operations included in the k second transactions is less than or equal to a number threshold. The third snapshot data is used to record the second database after the transactions based on the third snapshot data have been processed. The k second transactions are the transactions between the transactions of the third snapshot data and the transactions of the last saved third snapshot data, where k is an integer greater than or equal to 1.

[0198] Storage device 1401 is also used to retrieve third data in a second database prior to the recovery time, based on the recovery time, the second log, and the second correspondence.

[0199] Optionally, the second computing node 1403 receives the second transaction and sends the second log of the second transaction to the storage device 1401. For details on this process, please refer to [link to relevant documentation]. Figure 7 The relevant content in step 701 of method 700 shown will not be described in detail here.

[0200] Optionally, for details on how storage device 1401 stores the third snapshot data and the transaction information of the third snapshot data in the second correspondence, see [link to detailed implementation]. Figure 7 The relevant content in step 702 of method 700 shown will not be described in detail here.

[0201] Optionally, for details on how storage device 1401 retrieves third data from the second database prior to the recovery time based on the recovery time, the second log, and the second correspondence, please refer to [link to documentation]. Figure 7 The relevant content in step 703 of method 700 shown will not be described in detail here.

[0202] Optionally, the database system 1400 further includes a management node 1404, and the first transaction and the second transaction are two sub-transactions included in the distributed transaction;

[0203] Management node 1404 is used to send the first transaction to the first computing node and the second transaction to the second computing node at the first moment;

[0204] The first computing node 1402 is used to send a first feedback message after the first database has been processed based on the first transaction;

[0205] The second computing node 1403 is used to send a second feedback message after the second database has been processed based on the second transaction;

[0206] The management node 1401 is also used to send an acknowledgment message to the first computing node 1402 and the second computing node 1403 at a second time based on the first feedback message and the second feedback message. The acknowledgment message is used to indicate that the completion time of the first transaction and the completion time of the second transaction are both at the second time.

[0207] Optionally, storage device 1401 is also used for:

[0208] Obtain the first curve, which describes the relationship between the number of multiple processing operations and the time length for restoring the first database. The multiple processing operations include the processing operations included in the first transaction, which are spaced w between the third and fourth transactions, as well as the processing operations included in the fourth transaction. The third and fourth transactions are transactions that save the first snapshot data in two consecutive transactions. The third transaction is earlier than the fourth transaction, and w is an integer greater than or equal to 1.

[0209] Obtain the second curve, which describes the relationship between the number of multiple processing operations and the total amount of snapshot data saved within the storage period. The storage duration of the first snapshot data in the storage device is equal to the length of the storage period.

[0210] The number threshold is determined based on the first curve and the second curve.

[0211] Optionally, for details on how storage device 1401 acquires the first curve, please refer to [link to documentation]. Figure 10 The relevant content in step 1001 of method 1000 shown will not be described in detail here.

[0212] Optionally, for details on how storage device 1401 acquires the second curve, please refer to [link to documentation]. Figure 10 The relevant content in step 1002 of method 1000 shown will not be described in detail here.

[0213] Optionally, the storage device 1401 determines the number threshold based on the first curve and the second curve. For details, please refer to [link to relevant documentation]. Figure 10 The relevant content in step 1003 of method 1000 shown will not be described in detail here.

[0214] In this embodiment, since the sum of the number of processing operations included in the first snapshot data transaction and the number of processing operations included in n first transactions is less than or equal to a number threshold, the storage device stores the first snapshot data and the transaction information of the first snapshot data in a first correspondence. Therefore, for the interval between two consecutive storages of the first snapshot data in the first correspondence, the total number of processing operations performed within this interval is less than or equal to the number threshold, denoted by A. Thus, when the storage device uses the first correspondence to restore the first database, the number of processing operations required does not exceed the number threshold A. Consequently, the time required to restore the first data in the first database prior to the restoration time is less than or equal to the total time required to perform A processing operations, thereby shortening the time required to restore the database and improving the efficiency of database restoration.

[0215] See Figure 15 This application provides a computing device 1500. For example, the computing device 1500 may be... Figure 1 , Figure 2 or Figure 3 The devices in the database system 100 shown (such as compute nodes or storage devices), or may be Figure 4 Method 400 shown Figure 7 Method 700 or shown Figure 10 Devices in the database system shown in method 1000 (such as a first computing node, a second computing node, a management node, or a storage device).

[0216] like Figure 15As shown, the computing device 1500 includes: a bus 1502, a processor 1504, a first memory 1506, a communication interface 1508, and a second memory 1510. The processor 1504, the first memory 1506, the communication interface 1508, and the second memory 1510 communicate with each other via the bus 1502. The computing device 1500 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 1500.

[0217] Bus 1502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 15 The bus 1502 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1502 may include a path for transmitting information between various components of the computing device 1500 (e.g., processor 1504, first memory 1506, communication interface 1508, second memory 1510).

[0218] Processor 1504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0219] The first memory 1506 may include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0220] The second memory 1510 may include volatile memory, such as random access memory (RAM).

[0221] See Figure 15 The first memory 1506 stores executable first program code, and the processor 1504 executes the executable first program code to implement... Figure 14The system 1400 shown illustrates the function of storage device 1401. Second memory 1510 stores executable second program code, which processor 1504 executes to implement... Figure 14 The system 1400 shown illustrates the functions of the first computing node 1402, the second computing node 1403, and the management node 1404. This enables the processor 1504 to implement the methods provided in any of the above embodiments. That is, the first memory 1506 and the second memory 1510 store instructions for executing the methods provided in any of the above embodiments. Alternatively,

[0222] The communication interface 1508 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 1500 and other devices or communication networks.

[0223] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0224] like Figure 16 As shown, the computing device cluster includes at least one computing device 1500. The memories (first memory 1506 and / or second memory 1510) of one or more computing devices 1500 in the computing device cluster may store the same instructions for performing the computing device methods provided in any of the above embodiments.

[0225] In some possible implementations, the memories (first memory 1506 and / or second memory 1510) of one or more computing devices 1500 in the computing device cluster may also each store partial instructions for executing the computing device methods described above. In other words, a combination of one or more computing devices 1500 can jointly execute instructions for performing the methods provided in any of the above embodiments.

[0226] The memories (first memory 1506 and / or second memory 1510) in different computing devices 1500 within the computing device cluster can store different instructions, each used to execute, such as... Figure 14 The database system 1400 shown contains some of its functions. That is, the instructions stored in the memory (first memory 1506 and / or second memory 1510) in different computing devices 1500 can implement the functions of one or more devices among storage device 1401, first computing node 1402, second computing node 1403 and management node 1404.

[0227] In some possible implementations, one or more computing devices in the computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 17 One possible implementation is shown. For example... Figure 17 As shown, the two computing devices 1500A and 1500B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0228] In this type of possible implementation, the first memory 1506 in the computing device 1500A stores the execution of, for example Figure 14 The instructions for the function of the storage device 1401 in the illustrated embodiment are shown. Meanwhile, the second memory 1510 in the computing device 1500B stores instructions for performing functions such as... Figure 14 The instructions for the functions of the first computing node 1402, the second computing node 1403, and the management node 1404 in the illustrated embodiment.

[0229] Figure 17 The connection method between the computing device clusters shown can take into account the needs of the database processing method provided in this application (e.g., saving the first log of the first transaction to the storage device, etc.), and therefore consider that the functions implemented by the first computing node 1402, the second computing node 1403 and the management node 1404 are executed by the computing device 1500B.

[0230] It should be understood that Figure 17 The functions of computing device 1500A shown can also be performed by multiple computing devices 1500. Similarly, the functions of computing device 1500B can also be performed by multiple computing devices 1500.

[0231] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 17 The connection method of the computing device cluster. The difference is that the memories (first memory 1506 and / or second memory 1510) in one or more computing devices 1500 in the computing device cluster can store the same instructions for executing the computing device method provided in any of the above embodiments.

[0232] In some possible implementations, the memories (first memory 1506 and / or second memory 1510) of one or more computing devices 1500 in the computing device cluster may also each store partial instructions for executing the methods provided in any of the above embodiments. In other words, a combination of one or more computing devices 1500 can jointly execute instructions for performing the methods provided in any of the above embodiments.

[0233] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the methods provided in any of the above embodiments.

[0234] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium capable of being stored by a computing device, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing device to perform the method provided in any of the above embodiments.

[0235] Those skilled in the art will recognize that the method steps and units described in the embodiments disclosed in this application can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0236] In the embodiments provided in this application, it should be understood that the disclosed system architecture, apparatus, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, or may be electrical, mechanical, or other forms of connection.

[0237] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0238] Furthermore, the modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or in software.

[0239] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0240] In this application, the terms "first" and "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first" and "second," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first" and "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of the various examples, a first access point can be referred to as a second access point, and similarly, a second access point can be referred to as a first access point. Both a first access point and a second access point can be access points, and in some cases, they can be separate and distinct access points.

[0241] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0242] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing a database, characterized in that, The method is applied to a database system, which includes a storage device and a first computing node. The first computing node is used to receive a first transaction, which includes m processing operations for processing a first database, where m is an integer greater than or equal to 1. The method includes: The storage device receives the first log of the first transaction sent by the first computing node, and the first log is used to record the m processing operations; If the sum of the number of processing operations included in the transaction of the first snapshot data and the number of processing operations included in n first transactions is less than or equal to a threshold, the storage device stores the information of the first snapshot data and the transaction of the first snapshot data in a first correspondence relationship. The first snapshot data is used to record the first database after the transaction based on the first snapshot data has been processed. The n first transactions are the transactions between the transaction of the first snapshot data and the transaction of the last saved first snapshot data, where n is an integer greater than or equal to 1. The storage device retrieves the first data in the first database prior to the recovery time based on the recovery time, the first log, and the first correspondence.

2. The method as described in claim 1, characterized in that, The storage device retrieves first data from the first database prior to the recovery time based on the recovery time, the first log, and the first correspondence, including: The storage device obtains second snapshot data and transaction information of the second snapshot data based on the recovery time and the first correspondence, wherein the second snapshot data is the snapshot data whose completion time in the first correspondence is before the recovery time and is closest to the recovery time; Based on the transaction information of the second snapshot data, the storage device obtains the first logs of z first transactions, where z is an integer greater than or equal to 1, and the completion time of the z first transactions is between the completion time of the transactions in the second snapshot data and the recovery time. The storage device obtains the first data in the first database prior to the recovery time based on the second snapshot data and the first logs of the z first transactions.

3. The method as described in claim 1 or 2, characterized in that, The method further includes: The storage device receives a storage request sent by the first computing node, the storage request including second data in the first database based on the first transaction and information of the first transaction; The storage device saves the second data in the first database and generates snapshot data, the snapshot data being used to record the second data in the first database; The storage device obtains the number of processing operations included in x first transactions between the first transaction and the transaction that last saved the first snapshot data, where x is an integer greater than or equal to 1; If the accumulated value between the acquired number and m does not exceed the number threshold and the difference between the accumulated value and the number threshold does not exceed the difference threshold, the storage device determines the snapshot data as the first snapshot data and the first transaction as the transaction of the first snapshot data.

4. The method according to any one of claims 1-3, characterized in that, The database system further includes a second computing node, which is used to receive a second transaction. The second transaction includes y processing operations for processing the second database, where y is an integer greater than or equal to 1. The method further includes: The storage device receives a second log of the second transaction sent by the second computing node, the second log being used to record the y processing operations; If the sum of the number of processing operations in the transaction of the third snapshot data and the number of processing operations in the k second transactions is less than or equal to a threshold, the storage device stores the information of the third snapshot data and the transaction of the third snapshot data in a second correspondence. The third snapshot data is used to record the second database after the transaction based on the third snapshot data has been processed. The k second transactions are the transactions between the transaction of the third snapshot data and the transaction of the last saved third snapshot data, where k is an integer greater than or equal to 1. The storage device retrieves third data from the second database prior to the recovery time based on the recovery time, the second log, and the second correspondence.

5. The method as described in claim 4, characterized in that, The database system further includes a management node, the first transaction and the second transaction are two sub-transactions included in a distributed transaction, and the method further includes: The management node sends the first transaction to the first computing node and the second transaction to the second computing node at the first moment; After the first computing node has completed processing the first database based on the first transaction, it sends a first feedback message. After the second computing node has completed processing the second database based on the second transaction, it sends a second feedback message. Based on the first feedback message and the second feedback message, the management node sends a confirmation message to the first computing node and the second computing node at a second time. The confirmation message is used to indicate that the completion time of the first transaction and the completion time of the second transaction are both at the second time.

6. The method according to any one of claims 1-5, characterized in that, The method further includes: The storage device acquires a first curve, which describes the relationship between the number of multiple processing operations and the time length for restoring the first database. The multiple processing operations include the processing operations included in the first transaction, which are spaced w times between the third and fourth transactions, and the processing operations included in the fourth transaction. The third and fourth transactions are transactions that save the first snapshot data in two consecutive transactions. The third transaction is earlier than the fourth transaction, and w is an integer greater than or equal to 1. The storage device acquires a second curve, which describes the relationship between the number of the plurality of processing operations and the total amount of snapshot data saved within the storage cycle, wherein the storage duration of the first snapshot data in the storage device is equal to the length of the storage cycle. The storage device determines the number threshold based on the first curve and the second curve.

7. A database system, characterized in that, The database system includes a storage device and a first computing node; The first computing node is configured to receive a first transaction, the first transaction including m processing operations for processing a first database, where m is an integer greater than or equal to 1, and send a first log of the first transaction to the storage device, the first log being used to record the m processing operations. The storage device is configured to store the information of the first snapshot data and the transactions of the first snapshot data in a first correspondence relationship when the sum of the number of processing operations included in the transaction of the first snapshot data and the number of processing operations included in n first transactions is less than or equal to a number threshold. The first snapshot data is used to record the first database after the transaction based on the first snapshot data has been processed. The n first transactions are the transactions between the transactions of the first snapshot data and the transactions of the last saved first snapshot data, where n is an integer greater than or equal to 1. The storage device is further configured to obtain first data in the first database prior to the recovery time based on the recovery time, the first log, and the first correspondence.

8. The database system as described in claim 7, characterized in that, The storage device is used for: Based on the recovery time and the first correspondence, information about the second snapshot data and the transaction of the second snapshot data is obtained, wherein the second snapshot data is the snapshot data whose completion time in the first correspondence is before the recovery time and is closest to the recovery time; Based on the transaction information of the second snapshot data, obtain the first logs of z first transactions, where z is an integer greater than or equal to 1, and the completion time of the z first transactions is between the completion time of the transactions in the second snapshot data and the recovery time; Based on the second snapshot data and the first logs of the z first transactions, obtain the first data in the first database prior to the recovery time.

9. The database system as described in claim 7 or 8, characterized in that, The storage device is also used for: Receive a storage request sent by the first computing node, the storage request including second data in the first database based on the first transaction after the first transaction is completed and information of the first transaction; Save the second data in the first database and generate snapshot data, the snapshot data being used to record the second data in the first database; The number of processing operations included in x first transactions between the first transaction and the transaction that last saved the first snapshot data, where x is an integer greater than or equal to 1; If the sum of the obtained number and m does not exceed the number threshold and the difference between the sum and the number threshold does not exceed the difference threshold, the snapshot data is determined as the first snapshot data, and the first transaction is determined as the transaction of the first snapshot data.

10. The database system according to any one of claims 7-9, characterized in that, The database system also includes a second computing node; The second computing node is configured to receive a second transaction, the second transaction including y processing operations for processing the second database, where y is an integer greater than or equal to 1, and send a second log of the second transaction to the storage device, the second log being used to record the y processing operations; The storage device is further configured to store the information of the third snapshot data and the transactions of the third snapshot data in a second correspondence when the sum of the number of processing operations included in the transaction of the third snapshot data and the number of processing operations included in the k second transactions is less than or equal to a number threshold. The third snapshot data is used to record the second database after the transaction based on the third snapshot data has been processed. The k second transactions are the transactions between the transactions of the third snapshot data and the transactions of the last saved third snapshot data, where k is an integer greater than or equal to 1. The storage device is further configured to obtain third data in the second database prior to the recovery time based on the recovery time, the second log, and the second correspondence.

11. The database system as described in claim 10, characterized in that, The database system also includes a management node, and the first transaction and the second transaction are two sub-transactions included in a distributed transaction; The management node is used to send the first transaction to the first computing node and the second transaction to the second computing node at a first moment; The first computing node is configured to send a first feedback message after processing the first database based on the first transaction; The second computing node is configured to send a second feedback message after the second database has been processed based on the second transaction; The management node is further configured to send a confirmation message to the first computing node and the second computing node at a second time based on the first feedback message and the second feedback message, wherein the confirmation message is used to indicate that the completion time of the first transaction and the completion time of the second transaction are both at the second time.

12. The database system according to any one of claims 7-11, characterized in that, The storage device is also used for: Obtain a first curve, which describes the relationship between the number of multiple processing operations and the time length for restoring the first database. The multiple processing operations include the processing operations included in the first transaction, which are spaced w times between the third and fourth transactions, and the processing operations included in the fourth transaction. The third and fourth transactions are transactions that save the first snapshot data in two consecutive transactions. The third transaction is earlier than the fourth transaction, and w is an integer greater than or equal to 1. Obtain a second curve, which describes the relationship between the number of the plurality of processing operations and the total amount of snapshot data saved within the storage cycle, wherein the storage duration of the first snapshot data in the storage device is equal to the length of the storage cycle; The number threshold is determined based on the first curve and the second curve.

13. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1-6.

15. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster performs the method as described in any one of claims 1-6.

Images