Increasing OLTP throughput by improving performance of logging using persistent memory storage

Abstract

Before modifying a persistent ORL (ORL), a database management system (DBMS) persists redo of a transaction and confirms that the transaction has been committed. Later, the redo is appended to the ORL. The DBMS stores a first redo of the first transaction into the first PRB and a second redo of the second transaction into the second PRB. Later, both redo are appended to the ORL. The DBMS stores redo of the first transaction in a volatile SRB (SLB) of the database session, respectively. The redo is stored in a volatile shared buffer shared by the database session. A redo of the second transaction is stored in the volatile shared buffer but not in the SLB. During re-silvering and recovery, the DBMS retrieves a redo from the fast persistent storage device, and then attaches the redo to the ORL in the slow persistent storage device. After re-silvering, redo from the ORL will be applied to the persistent database block during recovery.

Description

Technical Field

[0001] The present disclosure relates to accelerated write-ahead logging (WAL). Presented herein are techniques for persistent redo buffering before appending data to an online redo log (ORL). Background Art

[0002] A common approach to transaction logging in databases is to use write-ahead logging (WAL). WAL ensures the atomicity and durability components of the ACID (atomic, consistent, isolated, durable) transaction properties required in database systems by ensuring that no changes are made to disk before the corresponding redo is written. WAL facilitates high-performance database systems by providing in-place updates and eliminating the need to flush modified data pages to disk at each transaction commit.

[0003] However, the WAL of the prior art introduces delays during transaction commits because the commit process must wait for the log writer to complete, which is called log file synchronization. The online transaction processing (OLTP) throughput in database systems is greatly affected by the logging performance of redo. Specifically, a transaction should wait for its redo to be persisted when the transaction commits, and this delay reduces the throughput.

[0004] Persistent memory (PMEM) can be written at speeds close to dynamic random access memory (DRAM), but unlike DRAM, PMEM has the advantage of persistence. Replacing the disk or flash storage on which the redo logs are stored with PMEM helps reduce the impact of log file synchronization, but this solution has some disadvantages. With PMEM storage, writes are fast, and OLTP database sessions require correspondingly shorter sleep times in the foreground waiting for log file synchronization.

[0005] It is difficult to accurately program such short sleep times because the prior art lacks the fine granularity required for short sleeps. The cost of oversleeping caused by the insufficient granularity of the sleep time solution is high relative to the short duration of the required sleep. In addition, there is a task switch delay for the committing OLTP process to wake up from sleep and become runnable by the operating system. The task switch delay increases with the load on the system, making the sleep duration difficult to control. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In the figure:

[0007] Figure 1 is a block diagram depicting an example database management system (DBMS) that uses a persistent redo buffer to accelerate write-ahead logging (WAL) before appending data to an online redo log (ORL);

[0008] Figure 2 is a flow chart depicting an example computer process that a DBMS may execute to speed up WAL using buffering in fast persistent storage before appending redo records to an ORL;

[0009] Figure 3 is a block diagram depicting example dependency tracking data and associated structures;

[0010] Figure 4 is a flow chart depicting an example process for an early flush performed by a database session when executing a transaction that modifies a data block in a data block buffer;

[0011] Figure 5 is a flow chart depicting an example process that a database session performs to populate a session dependency array when a data block is modified during a transaction;

[0012] Figure 6 is a flow chart depicting an example process for commit flushing performed by a database session executing a transaction upon commit;

[0013] Figure 7 is a flow chart depicting an example computer process that a DBMS may perform during recovery;

[0014] Figure 8 is a flow chart depicting an example computer process that a DBMS may execute for tolerant re-silvering;

[0015] Fig. 9 is a flow chart depicting an example computer process that a DBMS may execute for non-uniform buffering;

[0016] Fig.10 is a flow chart depicting an example computer process that a DBMS may execute to timely switch an uncommitted transaction that was previously using a session redo buffer to instead use only the shared buffer for the remainder of the transaction;

[0017] Fig.11 is a flow chart depicting an example computer process that a database session may perform to persist a first transaction that a second transaction depends on;

[0018] Fig.12 is a flow chart depicting an example computer process that a DBMS may execute to dynamically adjust buffering for a fluctuating transaction mix;

[0019] Fig.13 is a block diagram illustrating a computer system upon which embodiments of the present invention may be implemented;

[0020] Fig.14 is a block diagram illustrating a basic software system that may be employed to control the operation of a computing system. DETAILED DESCRIPTION

[0021] In the following description, for the purpose of explanation, many specific details are set forth to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other cases, well-known structures and devices are shown in block diagram form to avoid unnecessary obscurity of the present invention.

[0022] General Overview

[0023] This paper is an accelerated write-ahead logging (WAL) by persisting redo records in persistent memory (PMEM) before appending data to the online redo log (ORL). The proposed technique utilizes PMEM to eliminate transaction commit delays in most cases, thereby maximizing online transaction processing (OLTP) throughput. Log file synchronization delays and the sleep they would require are avoided, and commit delays may only be incurred by the time it takes to write redo to PMEM.

[0024] This method uses PMEM as a secondary storage device to persist redo separately into a set of "persistent redo buffers" (PRBs) in PMEM that supplements the redo log. A set of one or more redo records may be referred to herein as redo. The generation of redo for a database session continues to use the existing logging model, such that a database session adds its own redo one by one to a shared system-wide dynamic random access memory (DRAM)-based log buffer as it is generated, the contents of which are periodically collected by a system process (such as a log writer) that persists these redos to a globally ordered redo log. In addition, a database session accumulates its redo in a session-private session redo buffer (SRB) based on DRAM. The SRB itself does not maintain a global ordering of interleaved redo for concurrent transactions generated by other database sessions.

[0025] When the transaction of the database session is ready to commit, the redo accumulated in the SRB (including redo records called commit records) can be synchronously persisted to the PRB in PMEM by the database session itself. In an embodiment, the PRB can reside in a remote PMEM. Persistent redo requires remote direct memory access (RDMA) to PMEM. This access is very fast, and because this work is not done by the central processing unit (CPU), the process of the database session does not sleep during RDMA (from the operating system perspective). When RDMA is completed, the redo written by the database session is persistent, so that the committed transaction is continuous and the database session can continue processing freely immediately. Finally, the log writer can also persist a copy of the same redo (including commit records) in the redo log during log file synchronization, but the database session does not need to wait for log file synchronization because the redo of the database session has been persisted in the PRB.

[0026] Architecturally, the redo log does not change. This feature can be an important factor in a complex database management system (DBMS) with many mature components that rely on the redo log.

[0027] When the database recovers from a crash, the redo in the PRB is used to resilver the redo log to ensure that the redo log is repaired to a consistent state for recovery. Resilvering takes the redo found in the PRB that has not yet been persisted to the redo log and writes the redo to the redo log. In this way, the complete log (including redo that would otherwise have been lost from the redo log) can be presented to recovery (and any other log clients that also expect a complete log).

[0028] In this approach, PMEM-based PRBs are used as an adjunct to logging for accelerated commits (which avoid log file synchronization), which reduces transaction commit latency and increases DBMS throughput. The result is speedup and increased reliability of the DBMS.

[0029] 3.0 Example Database Management System (DBMS)

[0030] Figure 1 1 is a block diagram depicting an example database management system (DBMS) 100 that accelerates write-ahead logging (WAL) by using a persistent redo buffer before appending data to an online redo log (ORL). The DBMS 100 is hosted on one or more computers, each of which may be a rack-mounted server such as a blade, mainframe, virtual machine, or other computing device.

[0031] DBMS100 operates one or more databases, such as a relational database, which contains a relational schema, a database dictionary, and relational tables composed of rows and columns stored in files in row-major or columnar (i.e., column-major) format, which may be composed of database blocks each containing one or more rows.

[0032] The DBMS 100 stores the redo generated for the transaction in an online redo log (ORL) 150, which the DBMS 100 uses for the WAL. By design, durably writing the changed redo records to the ORL 150 is faster than writing the changes to the database. For example, the database may reside in a slower storage tier than the slow persistent storage 113 containing the ORL 150. Alternatively, the database may reside in the slow persistent storage 113, and the DBMS 100 may prioritize writes to the ORL 150 over writes to the database.

[0033] In an embodiment, the slow persistent storage 113 is a flash solid state drive (SSD) and the database resides in a disk drive or a network attached storage (NAS). In an embodiment, the slow persistent storage 113 is a disk drive and the database resides in a NAS.

[0034] The database data in DBMS 100 is stored in data blocks in persistent storage. Database data is changed by first reading the data blocks from persistent storage 113 into a memory buffer of the DBMS, modifying the data blocks in the buffer, and then rewriting the modified data blocks to persistent storage 113. The buffer is referred to herein as a data block buffer. The copies of the data blocks stored in the data block buffer may be referred to as buffered data blocks. Redo is generated when a buffered data block is modified.

[0035] Since ORL 150 resides in slow persistent storage 113, ACID transactions are persistent if the changes made by a transaction to a data block are recorded in the redo stored in ORL 150, even if the changed data block has not yet been durably written to the database. If a transaction is confirmed as committed when all of its changes have been recorded in the redo stored in ORL 150 (which may be before the changes made by the transaction have been durably written to the database), the latency of the transaction is reduced.

[0036] 3.0 Storage Media Hierarchy

[0037] DBMS 100 has a hierarchy of storage tiers 111-113 of different speeds and capacities. Speed ​​and capacity are somewhat mutually exclusive such that higher speed means less capacity and vice versa. Volatile memory 111 has the fastest speed and the smallest capacity. Slow persistent storage 113 has the largest capacity and the smallest speed.

[0038] The speed and capacity of fast persistent storage 112 is between the speed and capacity of storage layer 111 and device 113. In a preferred embodiment, fast persistent storage 112 is a byte addressable persistent memory (PMEM) and slow persistent storage 113 is a flash solid state drive (SSD) or a disk drive. In an embodiment, fast persistent storage 112 is an SSD and slow persistent storage 113 is a disk drive.

[0039] 1.2 Volatile Buffering of Redo Records

[0040] Volatile memory 111 cannot provide ACID durability, but log buffer 134 in volatile memory 111 can provide further speedups in ways somewhat similar to or complementary to those of ORL 150. For example, corresponding redo records for changes T1A and T1B of transaction T1 and change T2A of transaction T2 can be stored in log buffer 134 at three different corresponding times and then flushed together in a single write from log buffer 134 to ORL 150. Thus, three writes to ORL 150 that would otherwise be required for three separate redo records are replaced with only one write, which increases throughput.

[0041] Each of changes T1A, T1B, T2A, T1-Begin, and T1-Commit is a redo record. Each of changes T1A, T1B, and T2A specifies the result of a create read update delete (CRUD) operation on a single table row or database block. For example, change T1A may contain a row identifier (ROWID) or a logical block address (LBA), and may contain a byte offset or a field / column identifier, and may contain a new value to be written. Changes T1-Begin and T1-Commit mark transaction T1 as a sequence of changes.

[0042] 1.3 Transaction Lifecycle

[0043] The life cycle of a transaction is a sequence of three phases. For example, the first phase of transaction T1 is uncommitted (also referred to herein as not ready to commit), which requires the generation of changes T1-start, T1A, and T1B and volatile buffering in log buffer 134. The second phase is committed (also referred to herein as ready to commit), which requires the generation and volatile buffering of T1-commit. The third phase is committed, which requires the persistence of all changes of the transaction, regardless of whether the changes of the transaction are written to the database.

[0044] As explained later herein, the staged lifecycle of a transaction may be a logical pipeline of stages, such that different subsequences of changes in a transaction may be in different respective stages of the pipeline (and different stages of the lifecycle) at the same time for various reasons. For example, at the same time: a) change T1A may have been applied to the database, b) ORL 150 may contain change T1B, and c) log buffer 134 may contain change T1-commit. In this case, the transaction itself remains in the second stage (committing) because change T1-commit has not yet been persisted.

[0045] 1.4 SRB for Concurrent Database Sessions

[0046] This article is based on two collaborative accelerations of concurrent multiple transactions and fast persistent storage 112. The log buffer 134 is a shared buffer. Since the log buffer 134 is potentially shared by all transactions that have not yet been fully committed (i.e., not committed or in the process of committing), the log buffer 134 is susceptible to contention by concurrent transactions. The consistency of the log buffer 134 may require thread safety, which uses mutual exclusion to serialize competing threads. Serialization may cause some transactions to stagnate (stall) or starve (starve), and may cause priority inversion when a high-priority transaction waits for (one or more) low-priority transactions to complete appending (one or more) redo records to the log buffer 134.

[0047] Without sacrificing thread safety, serialization is avoided by using SRBs 131-133 in volatile memory 111. Each of the database sessions 121-123 executing uncommitted transactions temporarily obtains and uses a single SRB for the transaction to accumulate redo generated for executing the uncommitted transaction in the single SRB. For example, each of the database sessions 121-123 may correspond to a respective client that sends DML statements to DBMS 100 through the respective database session 121-123. The database sessions 121-123 may concurrently: a) execute respective DML statements within respective transactions, and b) operate the respective SRBs 131-133 exclusively without contention to store redo generated for these transactions.

[0048] For various reasons, using SRB 131 is faster than using only log buffer 134 for storing redo. First, there is no contention waiting between multiple database sessions. Second, as explained later in this document, based on PRBs 141-143 in fast persistent storage 112, unlike log buffer 134, SRB does not have producers and consumers that may cause contention.

[0049] For example, a database session is a producer that may compete to append changes to the log buffer 134, while the controller of the log buffer 134 is a consumer that may compete to flush the log buffer 134 to the ORL 150 at the same time. Producer / consumer contention may require synchronization overhead for thread safety. In an embodiment, the producer or consumer is a thread. In an embodiment, the controller of the log buffer 134 or ORL 150 is a consumer and a thread.

[0050] Third, SRB 131 reduces the serialization overhead of thread safety (with mutual exclusion (mutual) locks, critical sections, or memory fences / barriers) because the use of SRB 131 avoids multi-session contention and producer / consumer contention. The use of SRBs 131-133 avoids producer / consumer contention by combining the responsibilities of producers and consumers into the same thread, which is inherently thread-safe. For example, as a producer, database session 123 appends redo for transaction T1 onto SRB 133, and when transaction T1 commits, database session 123 flushes the redo from SRB 133 to PRB 143.

[0051] 1.5PRB

[0052] The SRBs 131-133 are independently flushed by the database sessions 121-123 to the corresponding PRBs 141-143. The SRBs are flushed in this manner to commit a transaction and are flushed at other times as described herein.

[0053] In an embodiment, when SRB 131 is emptied by flushing, SRB 131 is disassociated from database session 121 and returned to the unused SRB pool. When a database session needs an SRB to store generated redo, the database session can obtain an unused SRB from the pool.

[0054] When an SRB is used by a database session to store redo records generated by transactions executed by the database session, the SRB is called the current redo buffer of the database session. The SRB can be used by multiple database sessions, one at a time, to store redo generated for transactions executed by the database session.

[0055] In an embodiment, database session 121 may "steal" an empty SRB from another database session. For example, database session 121 may need an SRB because its previous SRB was empty and was stolen. In an embodiment, database session 121 may steal a non-empty SRB from another database session, as long as database session 121 flushes the stolen SRB to the corresponding PRB before appending changes to the stolen SRB.

[0056] If a database session requires an SRB and the pool is empty and / or there are no empty SRBs to steal, the database session 121 waits until an SRB becomes available. In this way, there may be more concurrent database sessions than SRBs and more concurrent transactions than SRBs. In an embodiment, if a database session waits a threshold duration without obtaining or stealing a required SRB, the database session temporarily resorts to storing redo only in the log buffer 134, as discussed later herein, in which case the transaction should not be confirmed as committed until after the log buffer 134 is flushed to the ORL 150, as discussed below.

[0057] In an embodiment, no matter whether SRB 131 is allocated to database session 121 or reallocated to another database session, or is not allocated in the pool, there is always a fixed one-to-one association between SRBs 131-133 and PRBs 141-143. For example, no matter which database session SRB 131 is allocated to, SRB 131 is always flushed to PRB 141, and SRBs 132-133 are never flushed to PRB 141.

[0058] The result of the reallocation of SRBs between database sessions is as follows. PRB 141 may contain redo for multiple transactions of the same or different database sessions. Likewise, redo records for different transactions of database session 121 may reside in the same or different corresponding PRBs in PRBs 141-143. In any case, redo records for the same transaction are never divided between multiple PRBs 141-143. In an embodiment, database session 121 selects an SRB to steal or obtain from the pool based on which SRB is associated with the PRB with the most unused capacity or with at least a threshold amount of unused capacity.

[0059] 1.6 Log file synchronization

[0060] In the embodiments discussed below, flushing of log buffer 134 is asynchronous (ie, performed by a separate consumer), and log buffer 134 is decoupled from ORL 150. In embodiments, the capacity of ORL 150 is not actually limited, such as only limited by the capacity of slow persistent storage 113.

[0061] In an embodiment, the capacity of SRBs 131-134 and PRBs 141-143 is limited. For example, the size of each of SRBs 131-133 may be 64 kilobytes, and the size of each of PRBs 141-143 may be eight times larger, such as half a megabyte. In an embodiment, SRBs 131-134 reside in the volatile system global area (SGA) of DBMS 100.

[0062] Overflow of any of the SRBs 131 - 133 or PRBs 141 - 143 does not halt transaction execution of the database session 121 - 123 , as the database session will instead discontinue use of any of these buffers and resort to using only the log buffer 134 .

[0063] When all of the changes in the transaction have been stored in one of the PRBs 141-143 or the ORL 150, the transaction is confirmed as committed. In an embodiment, the COMMIT statement of the transaction can specify WAIT (i.e., synchronously) or NOWAIT (i.e., asynchronously). In an embodiment, WAIT flushes the log buffer 134 synchronously, while NOWAIT flushes the log buffer 134 asynchronously.

[0064] In the prior art, WAIT is much slower than NOWAIT because WAIT does not acknowledge the commit until all (including the commit entry) in the transaction's redo records have been appended to the contended ORL 150 in the slow persistent storage 113. With the innovation herein, WAIT is almost as fast as NOWAIT because transaction persistence is instead provided by an uncontended buffer in the fast persistent storage 112. In an embodiment, WAIT flushes the log buffer 134 asynchronously after the commit of a transaction synchronously flushes the SRB of the transaction.

[0065] Synchronous flushing of SRBs is faster than asynchronous flushing of shared buffers 133 for several reasons, including no producer / consumer contention, no serialization overhead, and reduced or eliminated input / output waits (IOWAITs), as discussed earlier herein, because fast persistent storage 112 is faster than slow persistent storage 113. For example, PMEM never incurs IOWAITs. PMEM can be part of user space in virtual memory, which speeds up by avoiding operating system calls accessing PMEM in a manner that would require a temporary, time-consuming switch from the operating system's user mode to kernel mode.

[0066] 1.7 Redundant Volatile Buffer

[0067] In an embodiment, a copy of some or all redo records is stored in both the SRB and the log buffer 134. For example, database session 123 stores change T1-commit in both the SRB 133 and the log buffer 134.

[0068] In an embodiment, storing change T1-commit in SRB 133 causes a synchronous flush of SRB 133, but does not cause a synchronous flush of log buffer 134. SRBs may contain redo for only one transaction at a time, while log buffer 134 may contain redo records for multiple transactions and multiple database sessions at a time.

[0069] During normal OLTP operation of DBMS 100, PRBs 141-143 operate as data sinks and are never flushed. The normal fate of redo records in the PRBs is to expire in the PRBs and never be flushed from the PRBs to ORL 150. The dashed arrow from PRB 143 to ORL 150 indicates that PRB 143 is only flushed during recovery after a crash of DBMS 100, which is not a normal operation. Recovery and tracking the expiration of redo change records in PRBs 141-143 are also discussed later in this article.

[0070] When a redo record in a PRB is flushed from log buffer 134 to ORL 150, the redo record expires. The PRB can be cyclic so that appending can overwrite expired redo and the PRB cannot overflow due to expired redo. In an embodiment, when log buffer 134 is finally completely flushed, all redo in any one of PRBs 141-143 expires.

[0071] 1.8 Global Sorting

[0072] The various speedups discussed earlier herein through uncontention buffers 131-133 and 141-143 are achieved by sacrificing serialization and global ordering of redo records that would otherwise be provided by ORL 150. In an embodiment, DBMS 100 typically assigns a monotonically increasing timestamp or system change number (SCN) to each redo record. Global ordering of redo records in log buffer 134 requires that, for any given redo record stored in ORL 150, any redo record with an earlier SCN is stored in ORL 150. Furthermore, redo records are stored or otherwise accessible in SCN order.

[0073] 2.0 Write-Ahead Logging (WAL) Process

[0074] Figure 2 is a flowchart depicting an example computer process that DBMS 100 may execute to accelerate write-ahead logging (WAL) using buffering in fast persistent storage 112 before appending redo records to ORL 150 in an embodiment. Figure 1 discuss Figure 2 .

[0075] Steps 201-202 are accelerated because they can occur without appending to ORL 150. Appending redo for transaction T1 to SRB 133 causes step 201. Redo records are appended to log buffer 134 before or after they are appended to SRB 133, depending on the embodiment.

[0076] Step 201 flushes SRB 133 to PRB 143, which is caused by the change T1-commit being appended to SRB 133. In an embodiment, fast persistent storage 112 is PMEM, and flushing SRB 133 is accelerated and implemented by a single memory operation such as memcpy() or block transfer (BLT). In an embodiment, appending change T1-commit to log buffer 134 requires requesting log buffer 134 to be flushed asynchronously in the background, and the asynchronous flush is postponed until step 203.

[0077] Step 202 confirms that transaction T1 is committed because step 201 persisted the change T1-commit in PRB 143 , even though the change T1-commit has not yet been flushed from log buffer 134 to ORL 150 .

[0078] Step 201 demonstrates persisting a transaction. Even though step 202 confirms that transaction T1 is committed, some or all of the redo records for transaction T1 may still remain in log buffer 134 after step 202. Eventually, the controller of log buffer 134 asynchronously flushes log buffer 134, including some or all of the redo records for transaction T1 (including change T1-commit in step 203). Even though step 203 flushes transaction T1 from log buffer 134 to ORL 150, some or all of the redo records for transaction T1 may still remain in PRB 143 after step 203 and after the recorded redo expires, as discussed later herein.

[0079] Thus, redo records may be appended to ORL 150 and still remain in any of PRBs 141-143. Steps 204-205 demonstrate an optional scenario in which redo records appended to ORL 150 may still remain in either SRB or PRB 141-143. Steps 203-204 may be combined into a single flush of log buffer 134, and within the single flush, steps 203-204 append (one or more) different corresponding redo records to ORL 150.

[0080] Specifically, step 203 appends the redo record(s) of transaction T1 as discussed above, and step 204 appends the redo record(s) of uncommitted transaction T2. ​​Transactions T1-T2 are generated by database sessions 122-123, respectively. For example, Figure 1 As shown in , log buffer 134 contains redo records for both transactions T1 - T2 , and a single flush of log buffer 134 flushes redo records for both transactions T1 - T2 .

[0081] In various cases, step 205 retains some redo records of uncommitted transaction T2 in SRB 132 and / or some other unexpired redo records of uncommitted transaction T2 in PRB 142, even though any of these redo records have been appended to ORL 150 by step 204. Thus, ORL 150 may contain redo records for transactions that are not ready to commit because log buffer 134 may have been flushed by different transactions of different database sessions per step 203.

[0082] 3.0 Reworked dependencies

[0083] An important guarantee provided by the global ordering of redo in ORL 150 is that redo dependencies are inherently resolved. To understand why this is important, it is necessary to understand what redo record dependencies are, how redo dependencies are resolved, and why resolving redo dependencies is important.

[0084] As the term "redo dependency" is used herein, a dependency exists for a second redo record on a first redo record when both the first redo record and the second redo record record consecutive changes to the same data block, and the second redo record records the later of the consecutive changes. For example, redo record A with SCN 1000 records a change to a data block, and redo record B with SCN 1003 records a subsequent and consecutive change to the same data block. Redo record B has a redo dependency on redo record A. A redo record that is dependent on (i.e., has a redo dependency on) another redo record has an SCN that is later than the other redo record.

[0085] A redo dependency may be referred to herein as a dependency. With respect to a dependency and a second redo record having a dependency on a first record, the second redo record is referred to as a dependent record and the first redo record is referred to as an antecedent record. For example, dependent record B has a dependency on an antecedent record A.

[0086] A dependency between a pair of redo records is resolved when the predecessor redo record is persisted in a manner that makes it available for recovery later. For example, when a database session stores redo record A in PRB 143 or ORL 150, the dependency that redo record B has on redo record A is resolved. When the term "redo" or its equivalent is the direct object in a sentence with the term "persist", persisting in this article refers to storing the redo record in a persistent session buffer or ORL or other persistent storage in a manner that makes the redo record available for recovery. For example, persisting redo record A means storing record A in PRB or ORL 150.

[0087] It is critical to the persistence and recovery of a database that before a redo record is persisted, if a particular redo record has dependencies, the dependencies are resolved before the redo record is persisted. Specifically, recovering a database requires applying the redo record to a past state of the database, such as a backup of the database. In order to recover a data block, the set of redo records that need to be applied to the data block is the set of redo records for changes made to the data block that occurred after the past state was created. In order to recover a data block using a set of redo records, recovery of the data block requires applying the set of redo records in the order of their corresponding SCNs (in other words, in the order in which the corresponding changes were made to the data block). Therefore, before applying the corresponding dependent record, the predecessor record must be applied to the data block. If the dependent redo record is persisted but the corresponding predecessor record is not, the predecessor record is not applicable to the data block before the dependent record during recovery, and the data block may not be effectively recovered.

[0088] For example, a database is being restored from a backup. The database includes a data block modified by changes recorded by redo record A and redo record B. The changes to the data block occurred after the backup was created. In order to correctly restore the data block, redo record A must be applied to the data block before redo record B. If the dependency that redo record B has on redo record A is not resolved, redo record A is not persisted and cannot be applied to provide valid recovery of the data block.

[0089] It is common in DBMS that separate uncommitted transactions may modify the same data block, and therefore, the predecessor and dependent records of the data block may be stored in separate SRBs. In other cases where there is no mechanism to prevent it, it is also possible that the SRB holding the dependent record is flushed to the PRB while the SRB holding the predecessor record is not flushed to the PRB. Therefore, the dependent redo record may be persisted without the predecessor record being persisted. A failure may completely prevent the predecessor record from being persisted. As a result, the predecessor record is not available to recover the data block, and the data block may not be effectively recovered.

[0090] Described herein are mechanisms to prevent persistence of dependent records when the corresponding predecessor records have not yet been persisted in the PRB or ORL. These mechanisms are referred to herein as dependency persistence enforcement ("DPE").

[0091] 3.1DPE Mechanism

[0092] Several types of DPE mechanisms are described. These mechanisms are referred to herein as Early Flush and Commit Flush. Both mechanisms detect when a dependent redo record is about to be created, and in response take steps to ensure that the dependent redo record is not persisted before the predecessor record. Such steps may include a database session determining that another database session's SRB will hold a predecessor record of a redo record to be created by this database session, and in response to this determination, "requisitioning" that SRB and flushing the redo therein to the PRB, thereby resolving the dependency. An important way in which an Early Flush differs from a Commit Flush is when a database session attempts to flush the SRB of another database session. An Early Flush attempts to flush the SRB of another database session before changes are made to data blocks and redo records are generated that will depend on redo records in the SRB of another database session. A Commit Flush attempts to similarly flush the SRB of another database session at commit time.

[0093] Figure 3 Dependency tracking data and associated structures are described. Dependency tracking data and associated structures are used for early flushing, and some are used for commit flushing. Commit flushing also uses other forms of dependency tracking data and data structures, which are described later.

[0094] refer to Figure 3 , which depicts a buffered data block 310. The buffered data block 310 includes a dependency SRB reference 340 and a dependency SCN 350. When a database session changes a buffered data block, the database session generates a dependency SRB reference. A dependency SRB reference is a reference to the following SRB that the database session uses to store redo records that record changes to the data block. Figure 3 As depicted in , the dependency SRB reference of data block 310 points to SRB 131. The database session also stores the SCN associated with the change in the data block, which is also associated with the redo record generated for the change. With respect to the dependency SRB reference and its corresponding dependency, the SCN is referred to as the dependency SCN.

[0095] The dependency SRB reference and the dependency SCN are stored in the data block buffer header. The data block buffer header stores metadata about the buffered data block. The buffer header can be stored in an in-memory structure separate from the data block, such as in a buffer header array.

[0096] When another database session is processing another change to the buffered data block, the dependency SRB reference in the buffered data block informs the database session that the redo record to be generated for the change will have a dependency on the redo record in the SRB pointed to by the dependency SRB reference.

[0097] For example, in response to modifying data block 310, database session A stores redo record A in SRB 131. The database session creates a dependency SRB reference pointing to SRB 131. The SCN associated with the modification is 1000 and is stored in the data block. The corresponding dependency SCN of the dependency SRB reference is 1000.

[0098] Another database session B, using SRB 132, initiates a change to data block 310. Before making the change and generating redo record B for the change, database session B checks for the dependency SRB reference in data block 310. After detecting the dependency SRB reference in data block 310, database session B determines that redo record B will be a dependent redo record that has a dependency on some redo record in SRB 131 that has a corresponding dependency SCN 1000. This record happens to be record A. However, the database session has not yet identified record A as a predecessor record that creates a dependency, and such an identification is not necessary. Instead, having determined that this dependency will be created on the redo record in SRB 131, the database session attempts measures to ensure that the redo record in SRB 131 is persisted before the dependent redo record B is created in SRB 132, thereby resolving the dependency of redo record B on record A.

[0099] Among these measures is comparing the "global ORL SCN" to the dependency SCN. Associated with ORL 150 is a global ORL SCN 330, which is the highest SCN of any redo record persisted in ORL 150. Due to the global ordering of ORL 150, redo records with SCNs equal to or less than the ORL SCN have been persisted in ORL 150. Therefore, if the ORL SCN is at least equal to the dependency SCN, the corresponding redo record has been persisted in ORL 150.

[0100] In the current example, the ORL SCN is compared to the dependency SCN 1000. If the ORL SCN is at least equal to 1000, then the predecessor redo record that created the dependency must have been flushed to PRB 141.

[0101] On the other hand, if the ORL SCN is less than the dependency SCN, then the ORL SCN does not indicate that the predecessor redo record that created the dependency has been persisted. In this case, check the highest flushed SCN of the referenced SRB.

[0102] Highest flushed SCNs 311-313 are associated with each of SRBs 131-133, respectively. The highest flushed SCN specifies the highest SCN of any redo record flushed from an SRB to a PRB. Redo records stored in an SRB with an SCN less than or equal to the corresponding highest flushed SCN have been persisted to the PRB.

[0103] In the current example, the highest flushed SCN of SRB 131 is compared to the dependency SCN 1000. If the highest flushed SCN is at least equal to 1000, then the predecessor redo record that created the dependency must have been flushed to PRB 141.

[0104] On the other hand, if the highest flushed SCN is less than the dependency SCN, then the highest flushed SCN does not indicate that the predecessor redo record that generated the dependency has been persisted. In this case, an attempt will be made to requisition the SRB referenced by the dependency SRB reference to flush that SRB. As used herein, the term "requisition" means that a database session obtains control of an SRB that is being used by another database session to flush the redo records generated by the other database session to the corresponding PRB of that SRB.

[0105] As will be explained in more detail, the SRB that the database session seeks to requisition may itself include dependent redo records that may have unresolved dependencies. In this case, rather than attempting to resolve this unresolved dependency by requisitioning another SRB (which may result in the need to requisition even more SRBs), the database session defers resolving the dependency. Additionally, if the value of the dependency SCN is greater than the current value of the "highest ORL dependency" associated with the database session SRB, the database session sets the high ORL dependency to the value of the dependency SCN.

[0106] In this paper, the highest ORL dependency is also referred to as the highest ORL SCN. Figure 3As shown in , each of the SRBs 131-133 is associated with a highest ORL dependency, shown as a corresponding highest ORL SCN 321-323. For a particular SRB, the highest ORL dependency specifies the highest dependency SCN of any dependency of a redo record stored or stored in that SRB. As will be explained later, during subsequent DRE processing, the highest ORL dependency may be increased to another, even higher dependency SCN.

[0107] When the ORL SCN of ORL 150 is at least equal to the highest ORL dependency of an SRB, then any dependency that ever existed for any redo that was in or in the SRB has been resolved. The term "highest ORL dependency" includes the term "ORL" because, in some cases, the SRB is not flushed until the ORL SCN is at least equal to the highest ORL dependency.

[0108] In the current example, database session B does not enlist SRB 131 and flush it. Database session B sets the highest ORL dependency of SRB 132 to dependency SCN 1000. Later, database session B encounters a dependency SRB reference with dependency SCN 1088 in another data block. Again, because the referenced SRB cannot be enlisted and flushed, database session B increases the highest ORL dependency of session buffer 132 to 1088.

[0109] 4.0 Early flushing process

[0110] Figure 4 The process of early flushing performed by a database session when executing a transaction that modifies a data block in a data block buffer is depicted. The process is performed in response to the database session detecting a dependency SRB reference in a data block (see step 401), thereby detecting a dependency. The dependency SRB reference is detected before generating a redo record of the change to the data block and changing the data block.

[0111] At step 402, the database session determines whether the ORL SCN indicates that the dependency has been resolved. This determination is made by comparing the corresponding dependency SCN to the ORL SCN. If the ORL SCN is at least equal to the dependency SCN, then the dependency has been resolved. Otherwise, the ORL SCN does not indicate that the dependency has been resolved, and the database session next checks the highest flushed SCN of the SRB referenced by the dependency SRB of the data block (the "referenced SRB").

[0112] At step 403, the database session determines whether the highest flushed SCN of the referenced SRB is at least equal to the dependency SCN. This determination is made by comparing the dependency SCN with the highest flushed SCN. If the highest flushed SCN is at least equal to the dependency SCN, then the dependency has been resolved.

[0113] Otherwise, the database session determines whether the database session should enlist the referenced SRB at step 404. The database session enlists the referenced SRB only if the corresponding highest ORL dependency of the referenced session buffer indicates that there are no unresolved dependencies for any redo records in the referenced SRB.

[0114] The highest ORL dependency of a referenced SRB can be used to determine whether there are any resolved dependencies for the referenced SRB in two ways. First, the ORL SCN is compared to the highest ORL dependency of the referenced SRB to determine whether the ORL SCN is at least equal to the highest ORL dependency. If so, there are no unresolved dependencies for the referenced SRB. Second, the highest ORL dependency may not specify an SCN because the ORL SCN was never set or was reset when the referenced SRB was previously flushed to the persistent buffer.

[0115] At step 405, when the highest ORL dependency of the referenced SRB does not indicate that all dependencies of the referenced redo buffer are resolved, the database session abandons flushing the referenced SRB. Conversely, if the highest ORL dependency is less than the dependency SCN, the database session sets the highest ORL dependency of the current SRB of the database session to the dependency SCN. Next, the database session initiates an operation to be performed to modify the data block at step 407, as described later.

[0116] At step 406, if the highest ORL dependency of the referenced SRB does indicate that there is no unresolved dependency for the referenced SRB, the database session enlists the referenced SRB and flushes the redo records therein to the corresponding PRB. After flushing the redo records, the highest ORL dependency of the referenced session buffer may be reset.

[0117] At step 407, the database session generates a redo record for the change to the data block. The database session then stores the redo record in the current SRB of the database session.

[0118] At step 408, the database adds or modifies a dependency SRB reference in the data block. The dependency SRB reference points to the current SRB of the database session. The database session also adds an SCN for changes to the data block.

[0119] At step 409 , the database changes the data block in the data block buffer.

[0120] 5.0 Submission Flushing

[0121] Under commit flush, when a database session executing a transaction modifies a data block and stores redo in the database session's current SRB, the database session tracks any unresolved dependencies originating from other SRBs. Unresolved dependencies are tracked in a session dependency array. Each element in the array corresponds to an SRB. While executing a transaction, if at least one dependency SRB reference corresponding to an SRB is encountered in a data block, the element in the session dependency array corresponding to that SRB is set to the highest dependency SCN encountered during the transaction for that SRB. In addition, the highest dependency SCN stored in the session dependency array is tracked; this SCN is referred to as the highest SRB dependency SCN with respect to the session dependency array and the database session's current SRB.

[0122] At commit time, the current SRB for the database session is not flushed until it is determined that all dependencies tracked by the session dependency array have been resolved. Although some of the dependencies may be unresolved when the session dependency array is set to record the dependencies, these dependencies may have been resolved at commit time.

[0123] Other dependencies in the dependency chain may not have been resolved at the time of commit. Try flushing the SRB to resolve those dependencies by enlisting it.

[0124] Figure 5 and 6 Describes the implementation of commit flushing. Figure 5 Describes the process that a database session performs to populate the session dependency array as it modifies blocks of data during a transaction. Figure 6 Describes the process that is performed to ensure that the dependencies represented by the session dependency array are resolved before a database session flushes its current SRB.

[0125] 6.1 Populate the session dependencies array

[0126] refer to Figure 5 , at step 501, the database session executing the transaction detects a dependency SRB reference in the data block being modified.

[0127] At step 502, the database session determines whether the dependency represented by the dependency SRB reference has been resolved. As described above with respect to early flushing, the database session determines whether the ORL SCN indicates that the dependency is resolved based on the corresponding dependency SCN recorded in the data block. If the ORL SCN is less than the dependency SCN, the database session next checks the highest flushed SCN of the referenced SRB. If the highest flushed SCN is less than the dependency SCN, the database session determines that the dependency is not resolved.

[0128] At step 503, in response to determining that the dependency is not resolved, the database session updates the session dependency array element corresponding to the referenced SRB to the dependency SCN. If the dependency SCN is greater than the existing SCN value stored in the element, the value is overwritten with the dependency SCN. In an embodiment, the array index of the element in the session dependency array corresponds to and uniquely identifies the SRB. In this way, the index of the element serves as a reference to the SRB.

[0129] At step 504, the highest SRB dependency SCN is updated. The highest SRB dependency SCN is updated to the maximum value of its current value and the dependency SCN.

[0130] The database session generates redo records for changes to the data blocks at step 505. The database session then stores the redo records in the current SRB of the database session.

[0131] At step 506, the database adds or modifies a dependency SRB reference in the data block. The dependency SRB reference points to the current SRB of the database session. The database session also adds an SCN for changes to the data block.

[0132] At step 507 , the database changes the data block in the data block buffer.

[0133] 6.0 Submission

[0134] Figure 6 A database session executing a transaction is depicted performing a process for commit flushing upon commit. The process is performed before committing the transaction. The database session may begin committing the transaction in response to receiving a request to commit the transaction from a user through the database session.

[0135] Figure 6The process begins at step 611. At step 611, the database session determines whether there may be any remaining unresolved dependencies recorded in the session dependency array. This determination can be made by comparing the ORL SCN to the highest SRB dependency SCN. If the ORL SCN is less than the highest SRB dependency SCN, there are unresolved dependencies. If all dependencies are resolved, then in step 617, the database session flushes the SRBs to the corresponding PRBs.

[0136] Otherwise, there may be unresolved dependencies recorded by the session dependency array. In response, each element in the session dependency array that records the dependency is checked. As shown, the control flow loop includes steps 611-616. In each iteration of the loop, steps 612-616 process a corresponding different dependency on a corresponding different referenced SRB. For example, if the session dependency array of the database session indicates dependencies on two referenced SRBs, the loop has two iterations.

[0137] At step 612, the database session determines whether the highest flushed SCN or ORL SCN of the SRB referenced by the element indicates that the corresponding dependency has been resolved. If the highest flushed SCN or ORL SCN of the referenced SRB is at least equal to the dependency SCN of the element, then the dependency has been resolved. If not, the database session next determines whether the referenced SRB can be flushed immediately.

[0138] At step 613, the database session compares the corresponding highest SRB dependency SCN of the referenced SRB with the ORL SCN. If the ORL SCN is at least equal to the highest SRB dependency SCN, the next iteration of the loop begins at step 611.

[0139] Otherwise, the ORL SCN is less than the highest SRB dependency SCN, and step 614 determines that there may be unresolved dependencies for the referenced SRB. If the database session determines in step 614 that the referenced SRB is ready to be flushed, then at step 615, the database session enlists the referenced SRB and flushes the redo therein to the corresponding PRB.

[0140] Conversely, if the database session determines in step 614 that the referenced SRB cannot be flushed immediately, then step 616 forgoes immediately flushing the referenced SRB. Instead, step 616 waits for the dependency to be persisted by one of the following means: a) the database session of the referenced SRB flushes the referenced SRB, or b) any database session or thread of DBMS 100 flushes the log buffer 134 to ORL 150.

[0141] In an embodiment, subsequent iterations of the loop occur while the current iteration is waiting in step 616. In any case, step 617 occurs only after no iterations are waiting in step 616. At step 617, the database session flushes its current SRB to the corresponding PRB.

[0142] 7.0 Re-Silvering and Restoration Process

[0143] Figure 7 is a flowchart depicting an example computer process that DBMS 100 may perform during recovery in an embodiment. Figure 1 discuss Figure 7 .

[0144] Figure 7 The process recovers redo records that are persisted in PRBs 141-143 but not persisted in ORL 150. In this article, during recovery, buffered redo records are redo records in any one of PRBs 141-143. Only persisted redo records are processed during recovery. Volatile buffers 131-134 are not used during recovery.

[0145] Resilvering is a recovery action that: a) detects when buffered redo records in PRBs 141-143 are not contained in ORL 150, b) detects what the global ordering of these buffered redo records is, and c) appends these buffered redo records to ORL 150. Here, detection (a) and (b) are based on analyzing the SCN.

[0146] When recovery begins, PRB 141 may be empty or contain redo records for the same or different transactions of the same or different database sessions before the crash. Some or all redo records in PRB 141 may be out of date. Figure 7 The recovery process detects and ignores those PRBs 141-143 that are empty.

[0147] For the buffered redo records in PRBs 141-143, a general pattern at the beginning of recovery may be that the buffered redo records with lower SCNs have expired, but the buffered redo records with higher SCNs have not expired. Step 701 detects and ignores expired buffered redo records. As discussed earlier herein, during normal operation, DBMS 100 retains the SCN of the last redo record appended to ORL 150 in volatile memory 111 and slow persistent storage 113, which is the ORL SCN.

[0148] As explained earlier herein, ORL 150 contains globally ordered redo records, but PRBs 141-143 instead contain locally ordered but non-globally ordered redo records within each transaction. Since global ordering can interleave redo records for multiple concurrent transactions (such as shown in log buffer 134), PRB 141 should not be processed as a whole without considering PRBs 142-143, as resilvering may require interleaving redo records for PRBs 141-143.

[0149] For global ordering during resilvering, two redo records that should be appended consecutively to ORL 150 may be retrieved together from PRB 143 or separately from corresponding PRBs 142-143, depending on the example, and Figure 7 The steps of the process are the same in either case.

[0150] Steps 701-702 re-establish the global ordering of redo records. In step 701, the metadata in PRBs 141-143 is analyzed to detect what is the lowest SCN number of the unexpired redo records in PRBs 141-143, and what is the highest SCN number of the expired redo records in PRBs 141-143.

[0151] Any SCN that is equal to or greater than the ORL SCN corresponds to an unexpired redo record, as discussed above. Any SCN that is less than the ORL SCN corresponds to an expired redo record. Expired redo records are ignored and not processed. If the PRB contains only expired redo records, then Figure 7 The process treats the buffer as empty and ignores the buffer. If all PRBs are empty, no re-silvering is required and steps 702-703 do not occur. In this case, step 704 can occur.

[0152] Step 702 maintains a corresponding pointer or offset into each non-empty PRB in PRBs 141-143. Initially, each pointer points to the oldest (i.e., lowest SCN) unexpired redo record in the buffer. In each iteration of iterating the redo records, step 702 identifies which pointer points to the unexpired redo record with the lowest SCN.

[0153] Reestablishing the global ordering of redo records may require iterating through the unexpired redo records from the lowest SCN to the highest SCN.Which PRB in PRBs 141-143 contains the redo records for the current, previous, or next iteration, each iteration checks a corresponding different redo record.

[0154] When step 702 starts, there is at least one non-empty PRB, and all SCNs in the SCN range of unexpired redo records are identified by step 701. Step 702 is repeated in each iteration.

[0155] Step 702 retrieves the redo record from any one of PRBs 141-143 that contains an unexpired redo record for the current iteration and copies the redo record to a buffer in volatile memory 111. Step 702 may access multiple PRBs and have many iterations, but step 702 only fills one volatile buffer. Iterations of step 702 continue until no un-iterated un-expired redo records remain. Based on this iterative global SCN sort, all unexpired redo records from all PRBs 141-143 are copied to the volatile buffer.

[0156] Step 703 flushes the volatile buffer, which appends all unexpired redo records to the ORL 150. In an embodiment, step 703 also retains the SCN of the redo record as the ORL SCN.

[0157] During recovery, ORL 150 operates as a first-in, first-out (FIFO) queue. Step 703 appends newer redo records to the tail of ORL 150, and step 704 removes less new redo records from the head of ORL 150.

[0158] Step 704 applies its dequeued redo records to the persistent database blocks previously written by the transaction. In an embodiment, step 704 does not occur until steps 702-703 are completed. In an embodiment with pipeline parallelism, steps 703-704 may occur concurrently. For example, if ORL 150 is not empty when DBMS 100 crashes, step 704 may begin applying redo records before step 703 begins.

[0159] When step 703 is complete, resilvering is complete. In an embodiment, when resilvering is complete, recovery is complete even if step 704 has not emptied the ORL 150, or even if step 704 has not yet begun. When recovery is complete, the DBMS 100 becomes online and serves OLTP.

[0160] As discussed earlier herein, during normal OLTP operation of DBMS 100, log buffer 134 and ORL 150 have the same original global ordering. However, log buffer 134 is not used during the resilvering steps 702-703 to reestablish a new global ordering in ORL 150. The resilvered global ordering need not be the same as the original global ordering in log buffer 134 when DBMS 100 crashed.

[0161] For example, interleaved redo records of concurrent independent transactions (i.e., transactions that are not dependent on each other, as discussed earlier and later in this document) may have a different ordering in the resilvered global ordering than the original global ordering. For example, as discussed earlier in this document, SCNs do not uniquely identify redo records, and two redo records of different corresponding database blocks and different corresponding independent transactions do not conflict, and if two redo records share an SCN, the two redo records may be reordered.

[0162] In an embodiment, even without a shared SCN, non-conflicting redo records of different corresponding database blocks and different corresponding independent transactions can be reordered. In any case, resilvering does not reorder redo records that have been appended to ORL 150 when DBMS 100 crashes.

[0163] 8.0 Fault Tolerant Re-Silvering Process

[0164] In an embodiment, the volatile memory 111 and the fast persistent storage 112 are not in the same computer. In other words, the fast persistent storage 112 may be remote, and when the fast persistent storage 112 is PMEM, then the fast persistent storage 112 may be accessed via remote direct memory access (RDMA). If the volatile memory 111 and the fast persistent storage 112 are separated only by a rack backplane or a network switch (such as for Ethernet or Fibre Channel), then even if the slow persistent storage 113 is a local disk or local flash, accessing the fast persistent storage 112 is still faster than the slow persistent storage 113. For example, local flash may require IOWAIT, which PMEM RDMA mostly or completely avoids.

[0165] Since RDMA is not limited to local devices, RDMA can access multiple remote PMEMs in different network elements. For example, RDMA to two PMEMs may still be faster than accessing flash once. Therefore, redundant remote PMEMs are encouraged for high availability (HA) through fault tolerance, assuming that PMEM devices may eventually fail. In an embodiment, one or both PMEMs are remote or local.

[0166] In an embodiment, each PMEM in a pair of PMEMs contains a copy of each of PRBs 141-143. Flushing SRB 131 to PRB 141 may require flushing to both PMEMs simultaneously and redundantly. Recovery of DBMS 100 may find that the two PMEMs are inconsistent in state / content with respect to PRBs 141-143 because one of the PMEMs in the pair of PMEMs failed.

[0167] For example, congestion or crash of a switch or backplane may crash DBMS 100 while flushing SRB 131 redundantly to a PMEM pair such that the flush succeeds for only one PMEM in the pair. In another example, one of the redundant flushing pair fails due to a network switch or the PMEM device itself, but DBMS 100 does not crash. In this case, DBMS 100 may discontinue use of the unavailable PMEM device and continue normal OLTP operations using only the surviving PMEMs. In any case, DBMS 100 may eventually crash when using two or only one PMEM, and detecting which PMEM should be used as the system of record for re-silvering during recovery may require some analysis as follows.

[0168] Figure 8 is a flow chart depicting an example computer process that DBMS 100 may perform for fault-tolerant resilvering in an embodiment. Figure 7-Figure 8 Some or all of the process steps may be complementary and interleaved. Figure 1 discuss Figure 8 .

[0169] When DBMS 100 discovers that a previously unavailable PMEM device has become available, the SCN in the PMEM itself should not be trusted because the PMEM may have been out of service before DBMS 100 crashed (and in some cases, before DBMS 100 was restored from backup). Figure 8 The procedure correctly selected PMEM as the recording system for re-silvering.

[0170] In this example, the mirrored fast persistent storage device 112 is a remote PMEM. In other embodiments, Figure 8 The process is effective for local mirrored fast persistent storage 112 and / or non-PMEM (such as flash memory). For example, PRBs 141-143 can be mirrored in two local solid-state drives (SSDs), and slow persistent storage 113 can be a disk. Various media combinations of persistent storage 112-113 are discussed earlier in this document.

[0171] Figure 8 The process requires two or three computers, and at least one of these computers hosts DBMS100, and the hosting computer performs the recovery of DBMS100 during the recovery. Figure 8 Step 801 retrieves a first mirror incarnation number from a first fast persistent storage device 112 in a first computer, such as by RDMA as discussed earlier herein. Step 802 retrieves a second mirror incarnation number from a second fast persistent storage device 112 in a second computer.

[0172] Step 803 detects that the second mirror incarnation number is higher than the first mirror incarnation number, which causes step 803 to select the second fast persistent storage device 112 as the recording system for re-silvering. In this case, re-silvering uses only the second fast persistent storage device 112. Figure 5 As discussed, the resilvering step 502 entails retrieving unexpired redo records from the second fast persistent storage device 112, such as via RDMA.

[0173] At the end of recovery, DBMS 100 can reinitialize the first fast persistent storage device 112 that is not the recording system, so that the two fast persistent storage devices 112 store the same mirror incarnation number, and each mirror of each PRB in PRB 141-143 is empty. In this case, DBMS 100 can become online and use the two fast persistent storage devices 112 to serve OLTP.

[0174] When DBMS 100 transfers one of the fast persistent stores 112 into or out of service, the mirror incarnation number is incremented by one and is persisted in any (one or more) fast persistent stores 112 in service. DBMS 100 never changes the mirror incarnation number of a fast persistent store 112 that is out of service.

[0175] In an embodiment, when the log buffer 134 is flushed, the mirror incarnation number is also increased. In an embodiment, the DBMS 100 may have more than two fast persistent storage devices 112. In this case, even if multiple fast persistent storage devices 112 fail, the DBMS 100 may continue normal OLTP operations.

[0176] 9.0 Non-uniform Buffering Process

[0177] As discussed later in this document, DBMS 100 can characterize and distinguish categories of transactions so that OLTP transactions are prioritized over large maintenance transactions. Prioritization may require non-uniform buffering so that OLTP transactions should have priority access to SRBs. For example, as discussed earlier in this document, there may be more concurrent transactions than SRBs. As discussed later in this document, intelligent heuristics for allocating SRBs to characterized transactions can maximize the throughput of DBMS 100.

[0178] Whether buffers 131-133 and 141-143 provide speed-up or speed-down depends on the mix of OLTP and non-OLTP transactions. For non-OLTP transactions, DBMS 100 should not use buffers 131-133 and 141-143 and should instead resort to traditional write-ahead logging (WAL) based on log buffer 134 alone.

[0179] The mix of transactions may fluctuate, such that DBMS 100 should sometimes use buffers 131-133 and 141-143 and other times not use them. In an embodiment based on monitoring the mix of transactions, DBMS 100 occasionally switches between using and not using buffers 131-133 and 141-143 for all transactions.

[0180] In an embodiment, DBMS 100 instead dynamically adjusts the fluctuation ratio of: a) transactions using buffers 131-133 and 141-143 to b) transactions not using buffers 131-133 and 141-143. In an embodiment, DBMS 100 timely switches uncommitted transactions previously using buffers 131 and 141 to instead use only log buffer 134 for the remainder of the transaction.

[0181] Figure 9-10 These various adaptive and dynamic behaviors are demonstrated in . Fig. 9 is a flowchart depicting an example computer process that DBMS 100 may execute for non-uniform buffering in an embodiment. Figure 1 discuss Fig. 9 .

[0182] Steps 901-902 occur concurrently. Depending on the embodiment discussed later herein, step 903 may or may not occur concurrently with steps 901-902.

[0183] Step 901 appends redo records for a first set of transactions to SRBs 131-133. Ideally, the first set of transactions are OLTP transactions. Step 902 redundantly appends the same redo records to log buffer 134. As discussed earlier in this document, log buffer 134 is also referred to as a shared buffer.

[0184] Step 903 appends the redo records of the second set of transactions to log buffer 134 but not to SRBs 131 - 133. Ideally, the second set of transactions are non-OLTP. Fig. 9 The process does not buffer the first and second sets of transactions in the same manner. The non-uniform buffering of redo of transactions is novel.

[0185] 10.0 Transaction Differentiation Process

[0186] Fig.10 is a flow chart depicting an example computer process that DBMS 100 may execute to, in an embodiment, appropriately switch an uncommitted transaction that was previously using buffers 131 and 141 to instead use only log buffer 134 for the remainder of the transaction. Figure 9-10 The process steps can be complementary and interleaved. Figure 1 and Fig. 9 discuss Fig.10 .

[0187] As mentioned earlier in this article Fig. 9 As explained, the first set of transactions uses buffers 131-134 and 141-143, and the second set of transactions instead uses only log buffer 134. In this example, the first and second sets of transactions execute concurrently.

[0188] In step 1001, database session 121 generates / starts the first transaction in the first set of transactions that uses buffers 131-134 and 141-143. Specifically, database session 121 and the first transaction use buffers 131, 134, and 141.

[0189] In this example, non-OLTP transactions are long or large transactions. To detect non-OLTP transactions, step 1002 has a size threshold of the count of redo records that have been generated by uncommitted transactions and / or a life threshold of the duration that the transaction has not been committed. In an embodiment, the size threshold is instead a byte count in these redo records.

[0190] Before the first transaction is ready to commit, step 1002 detects that the first transaction exceeds any threshold. Step 1002 detects that the first transaction is non-OLTP. In this case, keeping the first transaction in the first group of transactions using buffers 131-134 and 141-143 may slow down DBMS 100 through priority inversion because too much redo for OLTP transactions is appended only to the slow and contended log buffer 134 and ORL 150.

[0191] For the subsequent redo records of the first transaction generated after the first transaction is detected as non-OLTP in step 1002, the first transaction is reallocated from the first group of transactions to the second group of transactions in step 1003. Therefore, the subsequent redo records of the first transaction (eventually including the commit redo record of the first transaction) will be appended to the log buffer 134 but not to the SRB 131.

[0192] In step 1004, database session 121 generates a second transaction. The first and second transactions are separately characterized as OLTP or non-OLTP. For example, even if the first transaction is ultimately characterized as non-OLTP, and even if the first and second transactions are generated by the same database session 121, step 1005 may initially assume that the second transaction is not (e.g., not yet) non-OLTP. Step 1005 adds the second transaction to the first set of transactions that use buffers 131-134 and 141-143.

[0193] 11.0 Mixed Flushing Process

[0194] Fig.11 is a flowchart depicting an example computer process that a database session 123 may perform to persist a transaction T2 on which a transaction T1 depends, in an embodiment. Figure 1 discuss Fig.11 .

[0195] As mentioned earlier in this article Fig. 9 As explained, the first set of transactions uses buffers 131-134 and 141-143, and the second set of transactions instead uses only log buffer 134. Step 1101 detects that the first transaction ready to commit depends on the second transaction. Step 1101 occurs when transaction T1 of database session 123 becomes ready to commit. In this example, step 1101 detects that transaction T1 depends on a transaction of database session 121.

[0196] Step 1102 detects whether the redo record has never been appended to any SRB. Step 1102 effectively detects whether the second transaction is in the first group of transactions that use buffers 131-134 and 141-143, or in the second group of transactions that instead use only log buffer 134. A "yes" determination in step 1102 means that the second transaction is in the second group of transactions that use only log buffer 134.

[0197] As explained earlier in this document, each SRB has a session dependency array that contains identifiers of other SRBs to track dependencies on other SRBs. Herein, log buffer 134 also has an identifier. The session dependency array uses the identifier of log buffer 134 to indicate that the basis of the dependency is not in another SRB, but is instead only appended to log buffer 134. Step 1102 is determined to be yes only if step 1102 detects the identifier of log buffer 134 of the redo record of the second transaction that is the basis of the dependency.

[0198] Step 1103 is reached only when steps 1101-1102 cooperatively detect that the first transaction that is ready to commit depends on a second transaction that uses only log buffer 134. In this case, there are redo record(s) of the second transaction generated before some redo record(s) of the first transaction. These redo record(s) of the second transaction should be persisted before those redo record(s) of the first transaction to maintain global ordering.

[0199] and Figure 4 Unlike the process of synchronously flushing SRB to satisfy the dependency, the second transaction does not use SRB. The redo record of the second transaction as the dependency basis may have been appended only to the log buffer 134.

[0200] Step 1103 detects whether the redo record of the second transaction, which is the basis of the dependency, has been flushed from the log buffer 134. If not, the first transaction should not be persisted until step 1104 flushes the redo record of the second transaction from the log buffer 134 to the ORL 150. In an embodiment of step 1104, the database session 123 flushes the log buffer 134 synchronously.

[0201] In an embodiment of step 1104, database session 123 instead requests that log buffer 134 be flushed asynchronously. In this case, database session 123 should not confirm that the first transaction has been committed until database session 123 is finally notified through a callback from DBMS 100 that log buffer 134 is finally flushed asynchronously.

[0202] Each iteration in the process may flush different SRBs to satisfy corresponding dependencies, because the first transaction ready to commit may depend on multiple concurrent transactions.

[0203] As long as those multiple concurrent transactions are in the first group of transactions using buffers 131-134 and 141-143, Fig.11 The process can be iterated in this manner. Those dependencies can be satisfied as discussed earlier in this article. When the iteration encounters a dependency on a transaction in the second set of transactions that instead only uses the log buffer 134, then the dependency is satisfied by steps 1101-1104 instead.

[0204] When step 1104 occurs in an iteration, then the iteration and Fig.11 The process stops because step 1104 satisfies all dependencies, because the log buffer 134 is shared by all transactions and the log buffer 134 is globally ordered.

[0205] 12.0 Adaptive Buffering Process

[0206] As discussed earlier herein, DBMS 100 can monitor the fluctuating transaction mix and can dynamically adjust the ratio of: a) the size of the first set of transactions that use buffers 131-133 and 141-143 to b) the sum of the size of the first set of transactions and the size of the second set of transactions that do not use buffers 131-133 and 141-143. This is the SRBing ratio, which is the fraction or percentage of concurrent transactions that should use buffers 131-133 and 141-143. The SRBing ratio can be expressed as a percentage from zero to one hundred, and also as a fraction such as zero, half, or one.

[0207] Fig.12 is a flow chart depicting an example computer process that DBMS 100 may execute to dynamically adjust buffering of a fluctuating transaction mix in an embodiment. Figure 1 discuss Fig.12 .

[0208] DBMS 100 may impose an SRBing ratio that is optimal for the current mix of OLTP and non-OLTP transactions. For example, if the SRBing ratio is half, new transactions should be added to whichever is smaller of the first or second set of transactions, which may fluctuate based on the arrival of new transactions, completion of older transactions, and transitions by switching transactions from the first set of transactions to the second set of transactions as discussed earlier herein. However, if the mix of transactions changes, the SRBing ratio is no longer optimal and throughput decreases.

[0209] DBMS 100 can more or less continuously monitor throughput as the rate at which data is flushed from log buffer 134 to ORL 150. The amount of data flushed can be measured in bytes or counts of redo records. If log buffer 134 is flushed too frequently or too infrequently, throughput decreases.

[0210] DBMS 100 can track: a) whether throughput is increasing or decreasing; and b) the direction of the trend, i.e. Fig.12 The trend of the SRBing ratio within the most recent recurrence of the process is increasing or decreasing. Based on this tracking, the SRBing ratio can be intelligently adjusted to maximize throughput.

[0211] The first, second and third cycles are Fig.12 The process uses three distinct and overlapping time periods. Fig.12 The repetitions of the process are separated by a first period (eg one minute).

[0212] Step 1201 is speculative. Step 1201 temporarily adjusts the SRBing ratio from the current value to the evaluation value. For example, step 1201 may calculate the evaluation value by increasing or decreasing the current value by an increment (e.g., 10 percent) based on whether the trend direction is increasing or decreasing. Step 1201 operates DBMS 100 with the evaluation ratio in a second period (e.g., six seconds) that is shorter than the first period.

[0213] Step 1202 detects whether the throughput increased during the second period. If step 1202 detects an increase, DBMS 100 employs the evaluation ratio for at least the remainder of the first period. In an embodiment, step 1202 employs the evaluation ratio only if the throughput increased by more than an improvement threshold (such as one percent). In an embodiment, step 1202 employs the evaluation ratio only if consecutive previous occurrences of a consecutive threshold count (e.g., five) of step 1202 consistently detect an improvement.

[0214] Otherwise, step 1202 detects that the throughput has decreased or has not changed. In this case, step 1203 restores the current value before the evaluation value, and the evaluation value is discarded.

[0215] The third period is a multiple of the first period. For example, if the third period is twenty times longer than the first period, then only Fig.12 Every twenty iterations of the process will execute step 1204. Step 1204 is more speculative than step 1201 because step 1204 can speculatively select a landmark ratio that is arithmetically different from the current SRBing ratio by a larger amount.

[0216] In an embodiment, step 1204 selects one or more milestone ratios from ratio values ​​of zero, half, and one. Step 1204 operates DBMS 100 using each selected milestone ratio for a milestone period (e.g., half a minute) that is longer than the second period and shorter than the first period, respectively, and adopts the milestone ratio that provides the best improvement or restores the current value before the milestone ratio based on whether the throughput increases.

[0217] If no milestone ratio is used, the directional trend remains unchanged. If zero or one is used, the directional trend is set to decrease or increase, respectively. If half is used, the directional trend is set based on which of zero or one provides a greater increase or lesser decrease.

[0218] Memory Overview

[0219] Because embodiments of the present invention relate to novel uses of non-volatile random access memory (NVRAM), it is referred to herein as persistent memory (PMEM). As used herein, "non-volatile" refers to a property of memory in which data is retained in the absence of any form of electrical energy (including external or battery backup). Examples of non-volatile memory include e-prom memory, flash memory, and disk memory. Non-volatile memory does not include volatile memory that is backed up by a battery to retain power in the absence of another external power source. For example, volatile memory coupled to a board with an embedded battery backup is not non-volatile memory because the volatile memory does not retain data in the absence of power provided by the battery.

[0220] Byte-addressable memory, such as PMEM and volatile dynamic random access memory (DRAM), is distinct from block-addressable memory. A byte or word is the smallest amount of data that can be addressed, retrieved from memory, or written to byte-addressable memory. In order to manipulate one or more bits in a byte or word, the byte or word containing the bits must be fetched into a register of a processor executing a machine instruction that references the byte or word and manipulated according to that machine instruction or another machine instruction.

[0221] In contrast, the smallest size of a unit of block-addressable memory is a data block. A data block includes multiple bytes and multiple words and cannot be completely stored in a processor's registers. For block-addressable memory, a data block is the smallest amount of data that can be addressed, retrieved from memory, or written to memory. Examples of block-addressable memory include flash memory and disk memory. To manipulate the bits or bytes in a block, the block containing the bits is loaded into a byte-addressable memory by issuing instructions that reference the block to a block-based interface. Therefore, block-addressable memory is not random access memory (RAM).

[0222] The difference between RAM and read-only memory (ROM) is that the data in RAM can be overwritten. As used herein, overwriting data refers to replacing data with new data without first erasing the data in the memory. Therefore, as used herein, RAM refers to byte-addressable memory that can be overwritten.

[0223] Using PMEM

[0224] In order for a process to access a region of PMEM, the region of PMEM is attached to the process. Attaching means mapping the virtual address space of the process to the address space of PMEM. The mapping may be stored, for example, in an MMU (memory management unit). The address space of PMEM may be interleaved across multiple memory modules (e.g., dual in-line memory modules).

[0225] According to an embodiment of the present invention, a process is attached to an area of ​​PMEM using a protocol similar to that used for memory mapped files. An example of such a protocol is DAX (direct access). The mmap() function can be called using a parameter specifying the address space of the PMEM area. Using PMEM is described in Introduction to Programming with Persistent Memory from Intel, published by U. Usharani and Andy Rudoff on August 14, 2017 (the contents of which are incorporated herein by reference) and in Persistent Memory Programming, published by Andy Rudoff in the summer of 2017 (the contents of which are incorporated herein by reference).

[0226] 11.0 Database System Overview

[0227] A database management system (DBMS) manages one or more databases. A DBMS may include one or more database servers. A database includes database data and a database dictionary stored on a persistent storage mechanism (such as a set of hard disks). Database data may be stored in one or more data containers. Each container contains records. The data within each record is organized into one or more fields. In a relational DBMS, data containers are called tables, records are called rows, and fields are called columns. In an object-oriented database, data containers are called object classes, records are called objects, and fields are called attributes. Other database architectures may use other terms.

[0228] A user interacts with the database server of a DBMS by submitting commands to the database server that cause the database server to perform operations on the data stored in the database. A user may be one or more applications that interact with the database server running on a client computer. Multiple users may also be collectively referred to as users in this article.

[0229] Database commands may be in the form of database statements that conform to a database language. The database language used to express database commands is Structured Query Language (SQL). There are many different SQL versions, some standard and some proprietary, and with various extensions. Data Definition Language ("DDL") commands are issued to a database server to create or configure database objects, such as tables, views, or complex data types. SQL / XML is a common extension of SQL used when manipulating XML data in an object-relational database.

[0230] A multi-node database management system consists of interconnected nodes that share access to the same one or more databases. Typically, the nodes are interconnected via a network and share access to shared storage to varying degrees, such as shared access to a set of disk drives and data blocks stored thereon. The varying degrees of shared access between nodes can include no sharing, sharing everything, exclusive access to database partitions by node, or some combination thereof. The nodes in a multi-node database system can be in the form of a group of computers (such as workstations and / or personal computers) interconnected via a network. Alternatively, the nodes can be nodes of a grid consisting of nodes in the form of server blades interconnected to other server blades on a rack.

[0231] Each node in a multi-node database system hosts a database server. A server, such as a database server, is a combination of integrated software components and allocation of computing resources, such as memory, nodes, and processes on nodes for executing the integrated software components on processors, that are dedicated to performing specific functions on behalf of one or more clients.

[0232] Resources from multiple nodes in a multi-node database system can be allocated to software running a particular database server. Each combination of software and resource allocation from a node is a server referred to herein as a "server instance" or "instance." A database server may include multiple database instances, some or all of which run on separate computers, including separate server blades.

[0233] Hardware Overview

[0234] According to one embodiment, the technology described herein is implemented by one or more special-purpose computing devices. The special-purpose computing device can be hard-wired to perform the technology, or can include a digital electronic device (such as one or more application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs)) that is permanently programmed to perform the technology, or can include one or more general-purpose hardware processors that are programmed to perform the technology according to program instructions in firmware, memory, other storage, or combination. Such a special-purpose computing device can also combine customized hard-wired logic, ASICs or FPGAs with customized programming to complete the technology. The special-purpose computing device can be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that incorporates hard-wired and / or program logic to implement the technology.

[0235] For example, Fig.13 1 is a block diagram illustrating a computer system 1300 upon which embodiments of the present invention may be implemented. Computer system 1300 includes a bus 1302 or other communication mechanism for communicating information, and a hardware processor 1304 coupled to bus 1302 for processing information. Hardware processor 1304 may be, for example, a general purpose microprocessor.

[0236] The computer system 1300 also includes a main memory 1306, such as a random access memory (RAM) or other dynamic storage device, coupled to the bus 1302 for storing information and instructions to be executed by the processor 1304. The main memory 1306 may also be used to store temporary variables or other intermediate information during execution of instructions to be executed by the processor 1304. When such instructions are stored in a non-transitory storage medium accessible to the processor 1304, the computer system 1300 is presented as a special-purpose machine customized to perform the operations specified in the instructions.

[0237] Computer system 1300 also includes a read only memory (ROM) 1308 or other static storage device coupled to bus 1302 for storing static information and instructions for processor 1304. A storage device 1310, such as a magnetic or optical disk, is provided and coupled to bus 1302 for storing information and instructions.

[0238] The computer system 1300 may be coupled to a display 1312, such as a cathode ray tube (CRT), via the bus 1302 to display information to a computer user. An input device 1314, including alphanumeric and other keys, is coupled to the bus 1302 to communicate information and command selections to the processor 1304. Another type of user input device is a cursor control 1316, such as a mouse, trackball, or cursor direction keys, for communicating directional information and command selections to the processor 1304 and for controlling cursor movement on the display 1312. This input device typically has two degrees of freedom in two axes, a first axis (e.g., x) and a second axis (e.g., y), which allows the device to be positioned in a specified plane.

[0239] The computer system 1300 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware, and / or program logic that is combined with the computer system to make the computer system 1300 a special purpose machine or programmed as a special purpose machine. According to one embodiment, the techniques herein are performed by the computer system 1300 in response to the processor 1304 executing one or more sequences of one or more instructions contained in the main memory 1306. Such instructions may be read into the main memory 1306 from another storage medium, such as the storage device 1310. Execution of the sequences of instructions contained in the main memory 1306 causes the processor 1304 to perform the process steps described herein. In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions.

[0240] The term "storage medium" as used herein refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media may include non-volatile media and / or volatile media. Non-volatile media include, for example, optical or magnetic disks, such as storage device 1310. Volatile media include dynamic memory, such as main memory 1306. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tapes or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a pattern of holes, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or tape cassettes.

[0241] Storage media are distinct from but can be used in conjunction with transmission media. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 1302. Transmission media can also take the form of acoustic or light waves, such as those generated during radio wave and infrared data communications.

[0242] Various forms of media may be involved in carrying one or more sequences of one or more instructions to the processor 1304 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and send the instructions over a telephone line using a modem. A modem local to the computer system 1300 may receive the data on the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal and appropriate circuitry may place the data on the bus 1302. The bus 1302 carries the data to the main memory 1306, from which the processor 1304 retrieves and executes the instructions. The instructions received by the main memory 1306 may optionally be stored on the storage device 1310 before or after execution by the processor 1304.

[0243] The computer system 1300 also includes a communication interface 1318 coupled to the bus 1302. The communication interface 1318 provides a two-way data communication coupled to a network link 1320 connected to a local network 1322. For example, the communication interface 1318 can be an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem that provides a data communication connection to a corresponding type of telephone line. As another example, the communication interface 1318 can be a local area network (LAN) card that provides a data communication connection to a compatible LAN. A wireless link can also be implemented. In any such implementation, the communication interface 1318 sends and receives electrical, electromagnetic or optical signals that carry digital data streams representing various types of information.

[0244] The network link 1320 typically provides data communication to other data devices through one or more networks. For example, the network link 1320 may provide a connection to a host 1324 or to data equipment operated by an Internet Service Provider (ISP) 1326 through a local network 1322. The ISP 1326 in turn provides data communication services through the global packet data communication network now commonly referred to as the "Internet" 1328. Both the local network 1322 and the Internet 1328 use electrical, electromagnetic or optical signals that carry digital data streams. The signals through the various networks and the signals on the network link 1320 and through the communication interface 1318, which carry the digital data to and from the computer system 1300, are example forms of transmission media.

[0245] Computer system 1300 can send messages and receive data, including program code, through the network(s), network link 1320, and communication interface 1318. In the Internet example, server 1330 can transmit the code for the requested application through Internet 1328, ISP 1326, local network 1322, and communication interface 1318.

[0246] The received code may be executed by processor 1304 as it is received, and / or stored in storage device 1310 or other non-volatile storage for later execution.

[0247] Software Overview

[0248] Fig.14 1 is a block diagram of a basic software system 1400 that may be employed to control the operation of the computing system 1300. The software system 1400 and its components, including their connections, relationships, and functions, are meant to be exemplary only and are not meant to limit implementation of the example embodiment(s). Other software systems suitable for implementing the example embodiment(s) may have different components, including components with different connections, relationships, and functions.

[0249] Software system 1400 is provided for directing the operation of computing system 1300. Software system 1400 includes a kernel or operating system (OS) 1410, which may be stored in system memory (RAM) 1306 and on fixed storage (eg, hard disk or flash memory) 1310.

[0250] The OS 1410 manages low-level aspects of computer operation, including managing the execution of processes, memory allocation, file input and output (I / O), and device I / O. One or more application programs, represented as 1402A, 1402B, 1402C ... 1402N, can be "loaded" (e.g., transferred from fixed storage 1310 to memory 1306) for execution by the system 1400. Applications or other software intended for use on the computer system 1300 can also be stored as a set of downloadable computer-executable instructions, for example, for downloading and installation from an Internet location (e.g., a Web server, application store, or other online service).

[0251] The software system 1400 includes a graphical user interface (GUI) 1415 for receiving user commands and data in a graphical manner (e.g., "clicks" or "touch gestures"). The system 1400 can then act on these inputs according to instructions from the operating system 1410 and / or (one or more) applications 1402. The GUI 1415 is also used to display the results of the operations from the OS 1410 and (one or more) applications 1402, so that the user can provide additional input or terminate the session (e.g., log out).

[0252] The OS 1410 may execute directly on the bare hardware 1420 (e.g., processor(s) 1304) of the computer system 1300. Alternatively, a hypervisor or virtual machine monitor (VMM) 1430 may be inserted between the bare hardware 1420 and the OS 1410. In this configuration, the VMM 1430 acts as a software "buffer" or virtualization layer between the OS 1410 and the bare hardware 1420 of the computer system 1300.

[0253] VMM 1430 instantiates and runs one or more virtual machine instances ("guest machines"). Each guest machine includes a "guest" operating system (such as OS 1410), and one or more applications designed to execute on the guest operating system, such as (one or more) application 1402. VMM 1430 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0254] In some cases, VMM 1430 can allow a guest operating system to run as if it were running directly on bare hardware 1420 of computer system 1400. In these cases, the same version of the guest operating system that is configured to execute directly on bare hardware 1420 can also execute on VMM 1430 without modification or reconfiguration. In other words, in some cases, VMM 1430 can provide complete hardware and CPU virtualization for the guest operating system.

[0255] In other cases, the guest operating system may be specially designed or configured to execute on VMM 1430 to improve efficiency. In these cases, the guest operating system "knows" that it is executing on a virtual machine monitor. In other words, in some cases, VMM 1430 can provide paravirtualization for the guest operating system.

[0256] A computer system process includes an allocation of hardware processor time and an allocation of memory (physical and / or virtual), the memory allocation being used to store instructions executed by the hardware processor, to store data generated by the execution of instructions by the hardware processor, and / or to store hardware processor state (e.g., contents of registers) between allocations of hardware processor time when the computer system process is not running. A computer system process runs under the control of an operating system and may run under the control of other programs executing on the computer system.

[0257] cloud computing

[0258] The term "cloud computing" is generally used herein to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows resources to be rapidly provisioned and released with minimal management effort or service provider interaction.

[0259] A cloud computing environment (sometimes referred to as a cloud environment or just a cloud) can be implemented in a variety of different ways to best meet different needs. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the public. In contrast, a private cloud environment is typically intended for use by or within a single organization only. A community cloud is intended to be shared by several organizations within a community; while a hybrid cloud includes two or more types of clouds (e.g., private, community, or public) that are bound together by data and application portability.

[0260] In general, the cloud computing model enables some of those responsibilities that may have previously been provided by an organization's own information technology department to be delivered instead as a service layer within a cloud environment for consumption by consumers (whether internal or external to the organization, depending on the public / private nature of the cloud). The precise definition of the components or features provided by or within each cloud service layer may vary depending on the specific implementation, but common examples include: Software as a Service (SaaS), in which consumers use software applications running on the cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications. Platform as a Service (PaaS), in which consumers can develop, deploy, and otherwise control their own applications using software programming languages ​​and development tools supported by the PaaS provider, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything under the runtime execution environment). Infrastructure as a Service (IaaS), in which consumers can deploy and run arbitrary software applications, and / or provision processing, storage, networking, and other underlying computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS), where the consumer uses a database server or database management system running on a cloud infrastructure, while the DbaaS provider manages or controls the underlying cloud infrastructure and applications.

[0261] The above basic computer hardware and software and cloud computing environment are presented for the purpose of illustrating the basic underlying computer components that can be used to implement the example embodiment(s). However, the example embodiment(s) are not necessarily limited to any particular computing environment or computing device configuration. Rather, the example embodiment(s) can be implemented in any type of system architecture or processing environment that a person skilled in the art would understand in light of the present disclosure as being capable of supporting the features and functions of the example embodiment(s) presented herein.

[0262] In the foregoing specification, embodiments of the invention have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the specification and drawings are to be regarded as illustrative rather than restrictive. The sole and exclusive indicator of the scope of the invention, and what the applicants intend to be the scope of the invention, is the literal and equivalent range of the set of claims issuing from this application, in the specific form in which such claims issue, including any subsequent corrections.

Claims

1. A computer-implemented method comprising: Without modifying the persistent online redo log: persisting the transaction's change entries, and After persisting the change entry of the transaction, confirming that the transaction has been committed; After the confirmation that the transaction has been committed, the change entry is appended to the durable online redo log.

2. The method of claim 1, wherein said appending said change entries to said persistent online redo log occurs during recovery.

3. The method of claim 1, wherein: said persisting said change entry occurs in a first persistent storage device; The persistent online redo log resides in a second persistent storage device that is slower than the first persistent storage device. The method of claim 3 , wherein the second persistent storage device, which is slower than the first persistent storage device, is solid state.

5. The method of claim 1 , wherein the persisting the change entry comprises using at least one selected from the group consisting of: Persistent storage media for at least two network elements, Remote Direct Memory Access (RDMA), and Byte addressable persistent memory PMEM.

6. The method of claim 1, wherein: At least one selected from the group consisting of: The persisting the change entry does not include using a file system, and The persisting of the changed entries is non-blocking.

7. The method of claim 1, further comprising: appending a second change entry to the persistent online redo log; After the second append, the second changed entry is retained in one selected from the group consisting of a volatile buffer and a persistent buffer.

8. A computer-implemented method comprising: During recovery, retrieving the changed entries from the first persistent storage; After said retrieving said change entry, appending said change entry to an online redo log residing in a second persistent storage device that is slower than said first persistent storage device; During the recovery, the change entries from the online redo log in the second persistent storage are applied to persistent database blocks.

9. The method of claim 8, further comprising detecting during the recovery that a second change entry is contained in the first persistent storage and the online redo log is contained in the second persistent storage.

10. The method of claim 8, further comprising: retrieving a first avatar number from the first persistent storage device in the first computer; retrieving a second avatar number from the first persistent storage device in a second computer; Detecting that the second incarnation number is higher than the first incarnation number; wherein said retrieving said change entry is based on said detecting that said second incarnation number is higher than said first incarnation number.

11. The method of claim 10, wherein the retrieving the first avatar number comprises using remote direct memory access (RDMA).

12. The method of claim 8, wherein said retrieving said change entries during said recovery comprises using RDMA.

13. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computing devices, cause the method of any one of claims 1-12 to be performed.

14. A system comprising: a plurality of computing devices, each computing device of the plurality of computing devices having one or more processors; The plurality of computing devices have access to one or more non-transitory computer-readable media storing instructions that, when executed by the plurality of computing devices, cause performance of the method of any one of claims 1-12.

Images