Horizontally scaling multi-statement interactive transactions with snapshot isolation in a database

Abstract

A shared-nothing database system is provided in which rows of each table are assigned to "slices" and multiple replicas ("replicas") of each slice are stored across persistent storage of multiple nodes. A request to read data from a particular row of a table can be handled by any node that stores a replica of the slice to which the row is assigned. For each slice, a single replica of the slice is designated as the "primary replica." All DML operations are performed by the node that has the primary replica of the slice to which the target row is assigned. Changes are then propagated to the other replicas ("secondary replicas") of the same slice.

Description

[0001] Cross-citation of related applications, declaration of interest

[0002] This application is a continuation of the benefit of part of application sequence No. 17 / 070,277, filed October 14, 2020, pursuant to 35 U.S.SC § 120, the entire contents of which are incorporated herein by reference as if fully set forth herein. The applicant hereby withdraws any disclaimers regarding the scope of the claims in the parent application or any disclaimers in its examination history, and informs the United States Patent and Trademark Office (USPTO) that any statements in this application may be more extensive than any statements in the parent application. Technical Field

[0003] This invention relates to storage systems, and more particularly, to shared-nothing database systems. Background Technology

[0004] Databases running on multiprocessor systems are generally divided into two categories: shared persistent storage databases and shared-nothing databases. Shared persistent storage databases assume that all persistent storage devices in the computer system are visible to all processing nodes. Therefore, the coordinator process in a shared persistent storage database system can assign any work granule to a process on any node, regardless of the location of the persistent storage device containing the data that will be accessed during the execution of the work granule. Shared persistent storage databases can run on both shared-nothing and shared-persistent storage computer systems. To run a shared persistent storage database on a shared-nothing computer system, software support can be added to the operating system, or additional hardware can be provided to allow processes direct access to remote persistent storage devices.

[0005] Shared-nothing (SNO) databases assume that a process can only access data if the data resides in persistent storage on the same node as the process. Therefore, if the data to be processed in a work granule resides in persistent storage on the same node as the process, the coordinator process in a SNO database can only assign the work granule to that process. SNO databases can run on both shared persistent storage and shared-nothing multiprocessing systems. To run a SNO database on a shared persistent storage machine, a mechanism can be provided to logically partition the database and assign ownership of each partition to a specific node.

[0006] Based on the foregoing, it is clearly desirable to provide a shared-nothing database system with fewer restrictions on which nodes can handle the workload. For example, when a task is reading a specific version of a specific data item stored in the database system, it is desirable to provide a shared-nothing database system where multiple nodes can execute the task. The larger the number of nodes capable of executing the same task, the easier it is to balance the workload among the available nodes. Furthermore, it is desirable for nodes performing read operations to be able to read data up to a specified snapshot time. To further improve performance, it is desirable to perform read operations without acquiring locks and without blocking, even when reading data items touched by transactions that have not yet been committed.

[0007] The methods described in this section are possible methods, but not necessarily methods that have been previously conceived or adopted. Therefore, unless otherwise stated, none of the methods described in this section should be considered prior art simply because they are included in this section. Furthermore, it should not be assumed that any method described in this section is fully understood, routine, or conventional simply because it is included in this section. Attached Figure Description

[0008] In the picture:

[0009] Figure 1 This is a block diagram of a distributed database system according to an embodiment, illustrating the relationship between the host, the database, and the tablespace;

[0010] Figure 2 This is a block diagram of a distributed database system according to an embodiment, in which rows of a table are mapped to slices, and multiple replicas are stored for each slice;

[0011] Figure 3 This is a block diagram illustrating the contents of a copy of a slice according to an embodiment;

[0012] Figure 4 This is a block diagram illustrating the chronological entry chain of two rows R1 and R2 according to an embodiment;

[0013] Figure 5 This is a block diagram illustrating a timing entry chain R1 after an entry is added to the end of the timing entry chain according to an embodiment.

[0014] Figure 6 This is a block diagram illustrating the sequence of entries in the chain where delta log entries are applied to row R1 after row heap, according to an embodiment.

[0015] Figure 7This is a block diagram illustrating the circular buffer properties of the incremental log according to an embodiment;

[0016] Figure 8 This is a block diagram illustrating the contents of a row stack entry according to an embodiment;

[0017] Figure 9 The illustration shows inter-host messages sent during the execution of a database command statement according to an embodiment;

[0018] Figure 10 The illustration shows inter-host messages sent during a transaction commit according to an embodiment;

[0019] Figure 11 It is a block diagram of a computer system that can be used as a client or host in a distributed database system employing the techniques described herein;

[0020] Figure 12 This is a block diagram illustrating a distributed database system according to an embodiment, having two engine clusters running on six hosts and a control cluster;

[0021] Figure 13 More detailed illustrations are available from Figure 12 A block diagram of the host computer;

[0022] Figure 14 This is a block diagram illustrating messages sent between the control cluster and the host when a host failure occurs, according to an embodiment.

[0023] Figure 15 This is a flowchart illustrating the steps taken to reconfigure a host cluster when a host fails, according to an embodiment; and

[0024] Figure 16 This is a block diagram illustrating a distributed database system including a control cluster according to an embodiment, wherein all hosts are able to communicate with each other over two different networks. Detailed Implementation

[0025] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0026] General Overview

[0027] A shared-nothing database system is provided in which parallelism and workload balancing are increased by assigning rows of each table to “slices” and storing multiple copies (“replicas”) of each slice in persistent storage across multiple nodes of the shared-nothing database system. When the data of a table is distributed across the nodes of the shared-nothing system in this way, a request to read data from a specific row of the table can be handled by any node that stores a copy of the slice to which that row is assigned.

[0028] In this embodiment, for each slice, a single copy of the slice is designated as the "primary copy". All DML operations (e.g., insert, delete, update, etc.) targeting a specific row of the table are executed by the node with the primary copy of the slice to which that specific row is assigned. The changes made by the DML operations are then propagated from the primary copy to other copies ("secondary copies") of the same slice.

[0029] slice

[0030] As mentioned above, a "slice" is the entity to which rows of a table are assigned. Row-to-slice assignment can be performed in various ways, and the techniques described herein are not limited to any particular row-to-slice assignment technique. For example, a table may have a primary key, and each slice may be assigned to rows whose primary key falls within a specific range. In this embodiment, a table whose primary key is an alphabet may assign its rows to three slices, where the first slice includes rows whose primary key begins with letters in the range AK, the second slice includes rows whose primary key begins with letters in the range LT, and the third slice includes rows whose primary key begins with letters in the range UZ.

[0031] As another example, row-to-slice assignment can be done using a hash function. For instance, a hash function that produces a hash value in the range of 1-3 can be used to assign a row to three slices. The slice to which any given row is assigned is determined by the hash value produced when the hash function is applied to the row's primary key.

[0032] For any given table, the number of slices to which its rows are assigned can vary based on various factors. According to one embodiment, the number of slices is chosen such that no single slice will store more than 1GB of data. Therefore, as a general rule, the more data a table contains, the larger the number of slices to which its rows are assigned.

[0033] When a table does not specify a primary key column, the database system creates a column and populates it with values ​​that can act as primary keys to assign rows of the table to slices. The values ​​in such a system-created primary key column can be, for example, integer values ​​that increment with each new row. This is merely an example of how system-generated primary key values ​​can be created, and the techniques described in this article are not limited to any particular method of generating primary key values.

[0034] dungeon

[0035] A "copy" is a stored replica of a slice. According to one embodiment, each slice has at least two copies. As mentioned above, each slice has one copy designated as the primary copy of that slice, and one or more secondary copies. Requests to read data from a slice can be executed by any node that has a copy of the slice in its persistent storage. However, requests to perform DML operations (e.g., insert, delete, update) on a slice are executed only by the node that has the primary copy of the slice in its persistent storage.

[0036] host

[0037] As used herein, the term "host" refers to the hardware components that constitute a shared-nothing node. For example, a host can be a computer system having one or more processors, local volatile memory, and local persistent storage. The host's volatile memory and persistent storage are "local" because I / O commands issued by the host to the volatile memory and persistent storage do not traverse the inter-host network connection. As will be described in more detail below, a host can interact directly with the volatile memory or persistent storage of another host over an inter-host network connection using Remote Direct Memory Access (RDMA) operations.

[0038] Persistent storage device

[0039] As mentioned above, each host has a local persistent storage device on which the host-hosted replicas are stored. Persistent storage devices can take many forms, including but not limited to disk storage devices, NVRAM, NVDIMM, FLASH / NVMe storage devices, etc. Furthermore, persistent storage devices can include combinations of storage technologies, such as NVRAM and disk storage devices, or NVRAM and FLASH / NVMe. For illustrative purposes, it is assumed that the persistent storage device used by the host is NVRAM. However, the technologies described herein are not limited to any particular persistent storage technology.

[0040] Engine Instance

[0041] As used herein, the term "engine instance" refers to code that executes within a host for storing, manipulating, and retrieving data stored in copies on persistent storage located locally on the host. A single host can execute any number of engine instances. An "engine cluster," also referred to herein as a "database system," comprises one or more engine instances that work together to serve database commands from clients. Engine clusters are described in more detail below.

[0042] In one embodiment, each host executes a different engine instance for each database whose data is hosted by that host. For example, if host H1 is hosting copies of tables in database D1 and copies of tables in database D2, host H1 will execute one engine instance to access the copy belonging to database D1 and a second engine instance to access the copy belonging to database D2.

[0043] For illustrative purposes, an example involving a single database will be given below, where each host is running a single engine instance. However, the techniques described herein are not limited to this embodiment.

[0044] Databases and tablespaces

[0045] A database typically consists of a set of tables and corresponding supporting structures, such as indexes. A database includes one or more tablespaces. In some embodiments, each tablespace is assigned to one or more hosts. The host(s) assigned to the tablespace store copies of the tables residing in that tablespace.

[0046] For example, Figure 1 This is a block diagram illustrating a database system 100 comprising six hosts H1, H2, H3, H4, H5, and H6. In the example shown, database system 100 manages two databases, D1 and D2. Database D1 has two tablespaces, D1T1 and D1T2, and database D2 has three tablespaces, D2T1, D2T2, and D2T3.

[0047] Tablespace D1T1 is assigned to hosts H1, H2, H3, and H4. Therefore, the "footprint" of tablespace D1T1 spans hosts H1-H4, and each of hosts H1, H2, H3, and H4 hosts a "tablespace member" of tablespace D1T1. Similarly, tablespace D1T2 is assigned to hosts H4, H5, and H6. Therefore, each of hosts H4, H5, and H6 hosts a tablespace member of D1T2.

[0048] Tablespace D2T1 is assigned to hosts H1, H2, and H3. This means that a single host can host tablespace members from multiple databases (e.g., H1 hosts tablespace members from database D1's D1T1, and tablespace members from database D2's D2T1). Tablespace D2T2 is assigned to hosts H3 and H4. Tablespace D2T3 is assigned to hosts H5 and H6.

[0049] Based on these assignments, copies of tables belonging to D2T2 will be stored, for example, in persistent storage on hosts H3 and H4. Similarly, copies of tables belonging to tablespace D1T2 will be stored in persistent storage on hosts H4, H5, and H6.

[0050] In some embodiments, the database may be hosted on a subset of available hosts. For example, database D1 might be hosted on hosts H1-H4. The hosts for the database's tablespaces are limited to the hosts of the database itself. Therefore, if database D1 is limited to hosts H1-H4, then tablespaces D1T1 and D1T2 will only be hosted on hosts H1-H4. In these cases, copies of tables residing in tablespace D1T2 cannot be hosted on H5 or H6, such as... Figure 1 As shown in the image.

[0051] Exemplary database system

[0052] refer to Figure 2 It is a block diagram of a database system 200 that includes a cluster of engine instances. Figure 2 The database system 200 shown includes five hosts (200A, 202A, 204A, 206A, 208A). Each host includes local volatile memory (200C, 202C, 204C, 206C, 208C) and local persistent storage (200D, 202D, 204D, 206D, 208D). Each host is executing an engine instance (200B, 202B, 204B, 206B, 208B).

[0053] Engine instances 200B, 202B, 204B, 206B, and 208B manage access to copies of data stored in the database managed by database system 200. In the illustrated embodiment, the database includes a single table T, whose rows are mapped to five slices (S1, S2, S3, S4, and S5). The database stores two copies of slices S1, S2, S4, and S5, and three copies of slice S3. Specifically, the primary copy (S1D1) of slice S1 is hosted at host 200A. The secondary copy (S1D2) of slice S1 is hosted at host 202A. The primary copy (S2D1) of slice S2 is hosted at host 204A. The secondary copy (S2D2) of slice S2 is hosted at host 200A. The primary copy (S3D1) of slice S3 is hosted at host 202A. The secondary copies (S3D2 and S3D3) of slice S3 are hosted on hosts 208A and 204A, respectively. The primary copy (S4D1) of slice S4 is hosted on host 208A. The secondary copy (S4D2) of slice S4 is hosted on host 206A. The primary copy (S5D1) of slice S5 is hosted on host 206A. The secondary copy (S5D2) of slice S5 is hosted on host 204A.

[0054] Because each of hosts 200A-208A operates as a shared-nothing node, engine instances on a host can only directly access copies located on their local persistent storage. As mentioned above, operations to read data from a slice can be performed by any engine instance local to any copy of the slice. Therefore, a request to read data from slice S2 can be directed to engine instance 200B (which has access to the secondary copy of S2) or engine instance 204B (which has access to the primary copy of S2). However, DML operations are only performed on the primary copy of the slice. Therefore, any DML operation that operates on data in slice S2 must be directed to engine instance 204B, because only engine instance 204B has access to the primary copy of slice S2.

[0055] Slice to Engine Instance Mapping

[0056] According to an embodiment, to ensure that database requests are directed to the appropriate engine instance, each host maintains a slice-to-engine instance mapping, which indicates the replicas being hosted by each engine instance. For example, Figure 2 The slice-to-engine instance mapping of system 200 shown may contain the following information:

[0057]

[0058] Slice-to-engine instance mapping is used in conjunction with row-to-slice mapping to identify the host to which any given request should be directed. For example, in response to a request to insert a new row into table T, the engine instance receiving the request determines the primary key of the new row and uses row-to-slice mapping to determine the slice of table T to which the new row belongs. For illustrative purposes, it will be assumed that the new row belongs to slice S2. The host then checks the slice-to-engine instance mapping to determine that the primary copy of slice S2 is hosted at host 204A. If the engine instance receiving the request is engine instance 204B, then engine instance 204B performs the insert into the primary copy S2D1. If the engine instance receiving the request is not engine instance 204B, then the engine instance receiving the request sends the request to engine instance 204B. The process of selecting the appropriate engine instance to coordinate the execution of any given database request is described in more detail below.

[0059] In the example given above, the requested operation is a DML operation (insert). Therefore, only the engine instance that controls the primary copy of the slice in question can perform this operation. However, if the operation is simply reading data from slice S2, then the operation can be performed by engine instance 204B (which has a primary copy of S2) or engine instance 200B (which has a secondary copy of S2).

[0060] Contents of the copy

[0061] As mentioned above, replicas store the rows of a table that have been mapped to the slice represented by the replica. For example, suppose slice S1 is a slice of table T1 whose primary key is a Social Security Number (SSN). Further suppose that all rows of table T1 whose primary key falls within the range 000-00-0000 to 399-99-9999 are mapped to slice S1. In these cases, each replica of S1 will store all rows of table T1 whose primary key falls within the range 000-00-0000 to 399-99-9999.

[0062] exist Figure 2 In the system shown, copies of slice S1 reside on host 200A (which has a primary copy S1D1) and host 202A (which has a secondary copy S1D2). Therefore, both copies S1D1 and S1D2 store all rows of table T1 whose primary key falls within the range of 000-00-0000 to 399-99-9999. However, according to one embodiment, to support snapshot-based data retrieval, the copies store more rows than the current version, and these rows are mapped to the slice represented by the copies.

[0063] refer to Figure 3 This illustration depicts various structures within a replica according to an embodiment for improving performance and supporting version control. Specifically, replica S1D1 includes an incremental log 304 containing incremental log entries 310 and 312, and a line stack 302 that initially does not contain entries. Replica S1D1 also includes a local index 306, which includes indices 320, 322, and 324. Each structure contained in replica S1D1 and how these structures are used to efficiently access a specific version of a line stored in slice S1 will be described in more detail below.

[0064] Incremental log: Temporary storage for row data

[0065] Incremental log 304 and row heap 302 are used together to store versions of rows belonging to slice S1. Incremental log 304 serves as short-term storage for row data, while row heap 302 serves as long-term storage for row data. Changes made to rows are first placed in entries within incremental log 304 and then ultimately “applied” to row heap 302 to create new entries in row heap 302. Therefore, entries in incremental log 304 represent changes to rows in slice S1 that have not yet been reflected in row heap 302. Changes represented in incremental log entries may or may not be committed. Any given row can have data distributed between entries in incremental log 304 and entries in row heap 302.

[0066] For example, suppose slice S1 is initially empty. Transaction TX1 can then insert two rows (R1 and R2) into table T1, where those rows have primary keys falling within the range associated with slice S1. When TX1 is executed, engine instance 200B (the engine instance local to the primary copy S1D1 of slice S1) will generate two log records (one for each newly inserted row). These two log records will be stored in incremental log 304. For illustrative purposes, it will be assumed that incremental log entry 310 is the log entry for data inserted by transaction TX1 as row R1, and incremental log entry 312 is the log entry for data inserted by transaction TX1 as row R2.

[0067] According to one embodiment, the incremental log 304 is implemented as a circular buffer for temporarily storing incremental log entries. The circular nature of the incremental log is... Figure 7 As shown in incremental log 714, when a DML operation is performed on a slice, new log entries are added to the incremental log of the primary replica of that slice. Furthermore, those log entries propagate to the secondary replicas of the slice, where log entries are added to the incremental logs of those secondary replicas.

[0068] Older incremental log entries are "applied" to the row stack to make room for new entries to be added to the incremental log. If the transaction that made the changes reflected in the new row stack entry has committed, the new row stack entry includes the transaction's commit time. Otherwise, the new row stack entry includes the transaction's transaction ID. In some cases, the new row stack entry will contain a "full row version." That is, the new row stack entry will include the values ​​of all columns of the row, because they existed when the changes reflected in the row stack entry were made to that row.

[0069] If the applied incremental log contains values ​​for all columns of the table, the row stack entry created when applying the incremental log entry can simply obtain its column values ​​from the incremental log entry. If the applied incremental log does not contain values ​​for all columns of the table, and the new row stack entry contains a full row version, the full row version must be constructed. To construct a full row version of a row, the engine instance managing the replicas in question "stitches together" (a) the column values ​​from the incremental log entry and (b) the values ​​obtained from the older row stack entry for that row.

[0070] Instead of creating a full row version, incremental log entries can be applied by creating row-heap entries that contain sparse row versions of the row. A sparse row version of a row contains fewer column values ​​than the row itself.

[0071] As will be explained in more detail below, in the case of sparse newline heap entries, the missing values ​​of the row required for a read operation can be obtained by following pointers to previous lineline heap entries for the same row. If that previous lineline heap entry is also sparse and does not contain all the required column values, the process of following pointers to previous lineline heap entries for that row is repeated until all the values ​​required for the read operation are obtained.

[0072] Log recording

[0073] Incremental log entries are stored in log records within incremental log 304. According to one embodiment, log records may store all incremental log entries generated for a slice during the execution of a transaction segment. Therefore, if the same transaction segment inserts both rows R1 and R2 into slice S1, the corresponding incremental log entries 310 and 312 will both be included in the same log record 315. When a log record such as log record 315 includes multiple incremental log entries, the log record is not deallocated until all of its contained incremental log entries have been applied to the row heap 302.

[0074] For illustrative purposes, the examples given below will assume that each log record has a single incremental log entry. However, the techniques described in this article are not limited to the number of incremental log entries that can be contained in a single log record.

[0075] Row heap: Long-term storage for row data

[0076] As explained above, when an incremental log entry is "applied" to the row heap, a row heap entry is created. Once applied, the incremental log entry can be deallocated, allowing the space occupied by the applied incremental log entry in the incremental log to be reused to store new incremental log entries. For any given row, the incremental log entry for that row and the row heap entry for that row are linked together using pointers to form a linked list. Within the linked list, entries are arranged in chronological order to form a chronological entry chain for the row. The chronological entry chain will be described in more detail below.

[0077] refer to Figure 8 It illustrates the contents of a row stack entry 800 according to an embodiment. The row stack entry 800 includes transaction information 806, a reference 808 to a previous version of the same row, an indicator 810 for columns in the stack entry that have values, and the values ​​812 for those columns.

[0078] Transaction information 806 includes information about the transaction that made the changes contained in the row heap entry. For example, before a transaction is committed, transaction information 806 may include the transaction ID of the transaction. The transaction ID can be used to look up the transaction's status in the transaction table. After a transaction is committed and row heap entry 800 is cleaned up, transaction information 806 may include the transaction's commit time.

[0079] Previous row reference 808 contains a reference to another row heap entry. The row heap entry pointed to by the previous row reference is (a) the same table row as row heap entry 800 and (b) the row heap entry that includes the value of the row immediately preceding the change reflected in row heap entry 800. If row heap entry 800 is the first row heap entry of the row in question, then previous row reference 808 will be empty. As will be described below, the links created by previous row references form part of a linked list called a chain of sequential entries called rows.

[0080] The column number information 810 indicates the number of columns in which data is stored in the row stack entry 800. If the row stack entry 800 is a sparse row entry (i.e., the entry contains fewer column values ​​than all the columns in the table), the row stack entry 800 can use a bitmap to indicate the columns for which it contains data. For example, if table T has four columns, and the row stack entry 800 is for a row in table T but only contains values ​​for the first and fourth columns, the column number field 810 can include a bitmap "1001" with the first and fourth bits set to indicate that the values ​​for the first and fourth columns are included in the row stack entry 800.

[0081] Data column field 812 contains the actual values ​​of the columns. If row stack entry 800 is the row stack entry corresponding to the initial insertion of the row, then data column field 812 will contain the values ​​of all columns. If row stack entry 800 corresponds to a DML operation performed later on the inserted row, then data column field 812 may contain the values ​​of all columns (if row stack entry 800 is a full-row version) or the values ​​of a subset of the columns (if row stack entry 800 is a sparse-row version).

[0082] Other metadata field 814 includes other metadata associated with the line stack entry 800. For example, other metadata field 814 may include various flags and checksums indicating that the kernel stack entry has not been corrupted.

[0083] Linear time sequence entry chain

[0084] As explained above, the latest change to a row can be included in an incremental log entry, where the incremental log entry points to an older row stack entry containing that row. However, a given row is not limited to a single incremental log entry and / or a single row stack entry. More precisely, the same row (e.g., R1) can have any number of incremental log entries and any number of row stack entries.

[0085] In one embodiment, entries for a row (incremental log entries and row heap entries) are linked sequentially to form a "sequential entry chain" for that row. The "tail" of the sequential entry chain contains the most recent changes to the row, while the "head" contains the oldest available version of the row.

[0086] A line's time-series entry chain can include zero or more incremental log entries, followed by zero or more line stack entries. For example, as... Figure 4 As shown, the entries in the entry chain of row R1 include:

[0087] • Incremental log entry 332, which contains the changes to R1 committed at time T100.

[0088] • Incremental log entry 334, which contains the changes to R1 committed at time T90.

[0089] • Line heap entry 342, which contains the changes made to R1 at commit time T50, and

[0090] • Line heap entry 346 contains all the values ​​of R1 that were originally inserted at commit time T10.

[0091] In this example, line R1 has a time-series entry chain consisting of two incremental log entries and two line heap entries. The pointers between entries in the time-series entry chain of line R1 are... Figure 4 As shown, incremental log entry 332 (with the most recent change to R1) is located at the tail of the time-series entry chain of R1 and row heap entry 346 (with the oldest available data for R1) is located at the head of the time-series entry chain of R1.

[0092] In comparison, Figure 4 The time-series entry chain of row R2 contains only a single entry (row heap entry 344). Therefore, the hash table entry associated with the primary key of row R2 points directly to row heap entry 344.

[0093] As will be explained below, the time-series entry chain for a row includes data for all available versions of the row that are being reconstructed. Therefore, the system can provide data for a row up to any specified snapshot time, provided that snapshot time is not older than the oldest version of the row stored in the time-series entry chain for that row.

[0094] Hash table access

[0095] In one embodiment, each engine instance maintains a hash table for each replica to access rows in each replica it can access. For example, engine instance 200B maintains hash table 350 ( Figure 3 and 4 This is used to access rows in replica S1D1. According to one embodiment, the hash table entry for a row points to the tail of the sequence of entries for that row.

[0096] To access the time-series entry chain of a row using a hash table, the engine instance applies a hash function to the row's primary key to produce a hash value corresponding to a hash bucket within hash table 350. That hash bucket stores the entry for the row with that primary key (if a version of the row is already stored in a slice). The row's hash table entry includes a pointer to the tail of the time-series entry chain for that row.

[0097] If the row in question has no incremental log entries, then the hash table entry for that row will point to the latest row heap entry for that row. Therefore, if row R2 is stored only in row heap entry 344 and row R2 has no incremental log entries, then the hash table entry for row R2 will directly point to row heap entry 344, as shown below. Figure 4 As shown in the image.

[0098] On the other hand, if a row has one or more incremental log entries, then the hash table entries for that row will point to the latest incremental log entry for that row. Therefore, in Figure 4 In the example shown, the hash table entry for line R1 points to incremental log entry 332 (the tail of the time-series entry chain of R1).

[0099] Hash table 350 can be implemented in any of a variety of ways, and the techniques described herein are not limited to any particular hash table implementation. According to one embodiment, a hash function is applied to the primary key to generate a hash value. A first subset of the bits of the hash value is used to identify the hash bucket the primary key falls into, and a second subset of the bits of the hash value is used as an offset relative to that hash bucket. The hash entry for the primary key is then stored within a specified hash bucket, starting at a specified offset. If two rows of primary keys "collide" (generating the same bucket and offset), any of a variety of collision resolution techniques can be used. The techniques described herein are not limited to any particular collision resolution technique.

[0100] According to an alternative embodiment, the first set of bits (e.g., bits 0-7) is used to identify the bucket, and the second set of bits (e.g., bits 8-15) is compared with each tag in a “hash tag array” stored within the bucket. The hash tag array can be, for example, bits 8-15 of the hash value generated from the primary key associated with each hash bucket entry. A SIMD operation can be used to compare bits 8-15 of the primary key in question with each 8-bit entry in the hash tag array in a single operation. The result of the SIMD operation will indicate which hash entries within the bucket have the same bits 8-15 as the primary key in question. Because this comparison is based on fewer bits than the entire primary key, the comparison may produce false positives. For example, the comparison might indicate that bits 8-15 of the hash values ​​of three hash entries “match” the hash value of the primary key in question. At least two of those matching hash entries are definitely false positives because for any given primary key, the table can have at most one row.

[0101] To ensure that the "matching" hash table entry is indeed for the primary key in question, a pointer within the hash table entry can be traced to the tail of the time-series entry chain of the row associated with that hash table entry. That tail of the time-series entry chain, whether an incremental log entry or a row-heap entry, will include the entire primary key for that row. The primary key of that row can then be compared to the primary key in question to determine whether the primary key of that row actually matches the primary key in question, or whether the match is a false positive.

[0102] Insert a newline into the copy.

[0103] When the engine instance hosting the primary replica of a slice receives a request to insert a new row into the slice, the engine instance generates an incremental entry for the change and stores the incremental entry in the incremental log record within the primary replica of the slice. The engine instance then stores the hash table entry for that row in the appropriate bucket of hash table 350. The hash table entry points to the new incremental log entry.

[0104] In addition to storing the hash table entry and incremental log entry for the new row, the engine instance also propagates the log entry to the hosts that have secondary replicas of the slice (in which the row is inserted). Within each of those hosts, the log entry for the new row is stored in the incremental log of the corresponding secondary replica, and a hash table entry is created pointing to those newly stored incremental log entries. Therefore, on each replica of the slice, the newly inserted row begins with a single-entry time-series entry chain.

[0105] Read data from existing rows.

[0106] As mentioned above, read operations can be performed by any engine instance capable of accessing any copy of the slice containing the desired data. Additionally, read operations can be performed up to any specified snapshot time, provided the snapshot time is not earlier than the oldest version of the row in the row's time-series entry chain. Read operations are performed through the following steps:

[0107] • Determine the slice from which to read data

[0108] • Use the slice-to-engine instance mapping to select a host that can access the slice's replica (either a primary or secondary replica).

[0109] • At the selected host, have the appropriate engine instance perform a hash operation on the primary key of the row from which data is to be read.

[0110] • Use the obtained hash value to locate the appropriate hash bucket within the hash table.

[0111] • Locate the hash table entry for the row in question within that bucket.

[0112] • Use a pointer from the hash table entry to locate the end of the sequence of entries for that row.

[0113] as well as

[0114] • Read the desired data from the entries in the chain of entries belonging to the row.

[0115] In some cases, the hash table entry for a row may not point to an entry that contains all the data required for the operation requested on the row. For example, an operation might request values ​​for columns c1 and c2 from R1, while incremental log entry 332 might only have the value for column c1. As another example, an operation might request the value of c1 from R1 up to a specific snapshot time (e.g., T50). However, the value of c1 contained in incremental log entry 322 could be associated with a commit time T100. Therefore, a read operation would require an older version of c1 than the version contained in incremental log 322.

[0116] The commit time stored in each entry indicates the snapshot to which the data in the entry belongs. If the entry has not yet been cleaned up, it will contain the transaction ID instead of the commit time. In these cases, the engine instance performing the read can use the transaction ID to look up the transaction's status in the transaction table. If the transaction has been committed, the commit time of the transaction specified in the transaction table is the commit time of the entry that includes that transaction ID. If the transaction is still active, the entry is skipped because uncommitted changes cannot be provided to any read operation. If the transaction is uncommitted and inactive, additional work can be requested to determine whether a read operation should see the changes in the entry, which will be described in more detail below.

[0117] When a row's hash table entry points to an entry that does not contain all the data required for a specified read operation, the engine instance traverses the links between entries in the row's time-series entry chain until all the required data is obtained. In the case of R1, this could require proceeding from incremental log entry 332 to incremental log entry 334, to row heap entry 342, and then to row heap entry 346.

[0118] An engine instance does not need to always follow the time-series entry chain of a row all the way to the head of the time-series entry chain to obtain the data required for a read operation. For example, the data required for a read operation can be obtained after only reading a subset of the entries in the time-series entry chain. Therefore, if the request is to read the latest version of c1 of R1, it can be obtained directly from incremental log entry 332 without traversing any additional links in the time-series entry chain of R1.

[0119] In one embodiment, all read operations are performed without acquiring any locks. Therefore, reads do not block write operations or other read operations, and write operations do not block read operations. As long as the entry chain of a row in the replica (primary or secondary) contains data committed up to the snapshot time of the read operation, the engine instance performing the read operation can obtain the required data from the entries in the entry chain of the row in the replica without acquiring locks and without blocking any concurrent writes or reads to the same row.

[0120] Local indexes for primary key columns

[0121] like Figure 3 As shown, replicas can store any number of local indexes 306. Figure 3 In the illustrated embodiment, replica S1D1 includes local indexes 320, 322, and 324. Local index 306 can be implemented in various ways. For illustrative purposes, it will be assumed that local index 306 is implemented using a B-tree structure. However, the techniques described herein are not limited to implementing local index 306 using any particular index structure.

[0122] Local indexes can be built on the primary key of a table. Such indexes are generally unnecessary for point lookups (e.g., looking up a single row based on its primary key), as this is the purpose of hash tables. However, local indexes built on primary keys can be useful, for example, to facilitate range scans. For instance, if the request is for the names of people whose Social Security numbers fall within the ranges 555-55-5555 and 555-55-9999, a B-tree index can be traversed to find the first index entry associated with a primary key equal to or greater than "555-55-5555". The first matching primary key can be obtained from that entry. Starting from that index entry, a linked list can be followed from leaf node to leaf node, obtaining the primary key from each index entry so far, until an index entry with a primary key greater than 555-55-9999 is encountered. The primary key obtained in this way can be used to index a hash table to retrieve data from a sequential chain of entries whose primary keys fall within the specified range.

[0123] According to one embodiment, when a local index is built on the primary key to facilitate range scans rather than including the entire primary key, leaf node index entries can simply include pointers to hash table entries associated with the rows. In this embodiment, the engine instance locates the hash table entry directly from the pointer in the index entry, rather than having to apply a hash function to the primary key to obtain a hash value and then look up the hash table entry in the hash bucket associated with that hash value.

[0124] According to an alternative embodiment, the entries in the leaf nodes of the index built on the primary key include hash values ​​generated from the primary key of the row corresponding to the index entry. Based on the hash values ​​obtained from the index entries, a range-based scan operation can locate the appropriate bucket in the hash table. The hash table entries in that bucket can then be examined to identify any hash table entries with primary keys falling within the range specified for the scan. The pointers in those entries (if any such entries exist) point to the tail of the chain of sequential entries whose primary key (a) hashes to that hash table bucket and (b) falls within the range specified for the scan.

[0125] Deferred index maintenance for deletion

[0126] As mentioned above, a local index can be created on the primary key to facilitate range scans based on the primary key. Under normal circumstances, this local index must be updated in response to DML operations affecting rows in the index associated with the slice. For example, if a new row with primary key PK7 is inserted, the entry for that primary key PK7 must (a) be added to the hash table and (b) inserted into the appropriate position within the index. Similarly, if a row with primary key PK9 is deleted, (a) the hash table entry associated with primary key PK9 is deleted, and (b) the index entry associated with primary key PK9 is deleted.

[0127] According to one embodiment, to improve the performance of DML operations that delete rows, the deletion of the corresponding index entry is postponed. Therefore, deleting a row associated with PK9 will result in the deletion of the hash table entry for PK9, but not the deletion of the index entry for PK9.

[0128] Because the hash table entries have been deleted, the system will not respond to read requests containing deleted data, even if the entries for the deleted rows are still retained in the index. For example, suppose that after deleting the row associated with PK9, the system receives a request to read data from rows whose primary keys fall within the range PK1-PK10. In response to this request, the system can use the local index built on the primary keys to determine the slice to which the range PK1-PK10 maps to rows associated with primary keys PK4, PK9, and PK10.

[0129] Based on this information, the scan operation will use a hash table to find the sequential chain of rows with primary keys PK4, PK9, and PK10. When the scan operation attempts to find a hash table entry for PK9, it will not find it because the hash table entry for PK9 has been deleted. Due to the missing hash table entry for PK9, the scan operation will skip PK9 and only return the data from the rows associated with PK4 and PK10.

[0130] Local indexes on non-primary key columns

[0131] Local indexes can be used to locate rows based on values ​​in columns other than the primary key column. For example, for table T, the primary key column could be a social security number, but a query could ask for the age of all people whose name is "Amy". To find rows that satisfy a predicate involving the "name" column, index 320 could be built based on values ​​from the "name" column. Index 320 could then be traversed based on the name "Amy" to find leaf nodes with entries about "Amy".

[0132] According to one embodiment, instead of containing pointers to one or more rows with the name "Amy", the index entries for "Amy" within index 320 include the primary key of each row with the name "Amy". After obtaining the primary key of the row with the name "Amy", the engine instance hashes the primary key to find the corresponding entry in hash table 350 and follows the pointers in those entries to obtain data from the chain of corresponding entries for those rows.

[0133] refer to Figure 7 This is a diagram illustrating how a secondary index 706, built on a non-primary key column, can be traversed based on a value to find the primary key of a row that has that value in a non-primary key column. For illustrative purposes, it will be assumed that the secondary index 706 is built on the "Name" column, the request is for the row with the name "Amy", and row R1 is the row that includes the name "Amy".

[0134] In these cases, secondary index 706 is traversed based on the key value "Amy" to locate the index entry 708 associated with the key value "Amy". The index entry includes the primary key of row R1. Then, the hash table entry for that primary key is looked up using the primary key of row R1. The hash table entry points to the end of the sequence of entries for row R1. Figure 7 In the example shown, the timing entry chain of line R1 includes only one entry, which is the line heap entry in line heap 702.

[0135] In contrast to the time sequence entry chain of row R1, Figure 7 The sequenced entry chain of line R2 includes an incremental log entry in incremental log 714 and a line heap entry in line heap 702. Therefore, the hash table entry of line R2 points to the incremental log entry of line R2, which in turn points to the line heap entry of line R2.

[0136] Global secondary index

[0137] When maintaining a local index for a non-primary column (such as "name"), this local index must be maintained for each slice, and the local index for a slice must exist on every copy of the slice. This is because it is impossible to know which slice(s) have, for example, a row named "Amy". Therefore, for each slice, its corresponding local "name" index must be searched to determine whether that slice has any row named "Amy".

[0138] As an alternative to maintaining a per-slice local index for non-primary columns, a single "global" index can be maintained for such columns. This global index is similar to a local index, but it will contain entries for all rows in the table, regardless of which slice they belong to. Therefore, the index entry associated with "Amy" in this type of index will include the primary key for all rows in the table whose "name" value is "Amy". Those primary keys can then be used in conjunction with a range-to-slice mapping to identify the slice to which each row retrieval request should point. Read requests can then be sent to a host that manages copies of those slices.

[0139] Perform DML on existing rows

[0140] Transactions that perform DML operations on existing rows (rows that already have a sequence of entries in the replica) are executed as follows:

[0141] • Create new log entries, including changes to the line.

[0142] • Store the new log entry in the incremental log of the primary replica containing the slice containing that line.

[0143] • Propagate new log entries to the host with the secondary copy of the slice.

[0144] • Make the new log entry point to the previous end of the sequence of entries for that line, and

[0145] • Make the hash table entries for this row point to the newly added entry (which is the new tail of the row's sequence entry chain).

[0146] For example, suppose the time-series entry chain of row R1 includes four entries, such as Figure 4 As shown in the diagram. In response to a DML operation that modifies row R1, the changes made by the DML operation are stored in a new incremental log entry 500. Figure 5 (As shown in the diagram). The new incremental log entry 500 points to the previous incremental log entry 332, which is the tail of the sequence of entries for line R1. The hash table entry for line R1 is then updated to point to the new incremental log entry 500, as shown in the diagram. Figure 5 As shown in the image.

[0147] Apply incremental logs to the line heap

[0148] As mentioned above, when a row is initially added to a replica, it is typically added as an incremental log entry. However, incremental log 304 serves as temporary storage for the row data that should eventually be applied to row heap 302. In one embodiment, a background process periodically applies incremental log entries to row heap 302.

[0149] For example, suppose incremental log 304 has a newly inserted incremental log entry for line R3. At this point, the sequenced entry chain for R3 contains only one incremental log entry (R3's hash table entry points to this entry). To apply R3's incremental log entry to line heap 302, a line heap entry containing the contents of R3's incremental log entry is stored in line heap 302. R3's hash table entry is updated to point to the new line heap entry, and the incremental log entry can be garbage collected / reused.

[0150] When a row's time-series entry chain includes one or more row-heap entries, the oldest incremental log entry (or entries) for that row must be applied before any newer incremental log entries for that row. For example, see reference... Figure 5 Incremental log entry 334 must be applied before incremental log entries 332 and 500 are applied. In these cases, applying the incremental log entry to the line heap involves:

[0151] • Create a new line heap entry containing the contents of the incremental log entry being applied.

[0152] as well as

[0153] • Repair the pointers in the sequence entry chain of the line so that:

[0154] The applied incremental log entries were removed from the time-series entry chain, and

[0155] ○ At the same location previously occupied by the applied incremental log entry, the new line heap entry is inserted into the line's sequence entry chain.

[0156] For example, refer to Figure 5In the time-series entry chain of line R1, incremental log entry 334 follows incremental log entry 332 and precedes line heap entry 342. To apply incremental log entry 334 to line heap 302, a new line heap entry (e.g., line heap entry 348) is created. Line heap entry 348 is filled with at least the contents of incremental log entry 334. Doing so creates sparse line heap entries (line heap entries containing fewer than all values ​​up to the commit time associated with the line heap entry). To create a fully filled line heap entry, line heap entry 348 can be further filled by merging all values ​​from line R1, as they existed when incremental log entry 334 was created. This can be achieved by obtaining missing values ​​from line heap entries further down the time-series entry chain of line R1 (e.g., line heap entries 342 and 346).

[0157] After creating and populating the new line heap entry 348, the pointer to R1's timing entry chain is updated so that the new line heap entry 348 replaces the applied incremental log entry 304 within R1's timing entry chain. In this example, updating the pointer involves making incremental log entry 332 point to the new line heap entry 348, and making line heap entry 348 point to line heap entry 342. Figure 6 The diagram illustrates these changes. After these pointer changes are completed, incremental log entry 334 is no longer part of the timing entry chain for line R1, and the space occupied by incremental log entry 334 can be freed / reused.

[0158] Propagate changes made by DML operations

[0159] As mentioned earlier, all DML operations affecting data in a slice are performed on the primary replica of the slice. However, in order for the secondary replica to be available for read operations, changes made on the primary replica must be propagated to the secondary replica. According to one embodiment, changes made to the primary replica of the slice are propagated to the secondary replica by sending log entries containing information about the changes to the host hosting the secondary replica.

[0160] For example, return to Figure 2 DML operations targeting data in slice S1 will be performed by engine instance 200B (which hosts S1D1, the primary replica of S1). In the primary replica (S1D1), changes can be made by (a) generating log entries for the changes, and (b) storing these log entries as incremental log entries in the incremental log of the primary replica (S1D1). The log entries are then propagated from host 200A to host 202A (which hosts S1D2, the secondary replica of S1).

[0161] Changes made to a slice are propagated to all secondary copies of the slice. Therefore, changes made to slice S3 will be made at the primary copy S3D1 on host 202A and propagated to host 204A (which has secondary copy S3D3) and host 208A (which has secondary copy S3D2).

[0162] Log contents

[0163] According to one embodiment, the content of the log records propagated to the secondary copy of the slice is similar to the content of the incremental log records of changes stored in the primary copy of the slice. Specifically, according to one embodiment, the content of the log records stored as incremental log records at the primary copy of the slice and propagated to the secondary copy of the slice includes:

[0164] • The transaction ID of the transaction that made the changes, as reflected in the execution log.

[0165] Primary keys of rows affected by DML operations

[0166] • Changes made by DML operations (e.g., updating one or more column values)

[0167] Which statement within the transaction specifies the instruction for the DML operation?

[0168] • Indication of the type of DML operation (e.g., insert, update, delete, etc.)

[0169] • An indicator of whether the log record is the last log record of the statement (last-log-)

[0170] (of-statement logo)

[0171] DML operations specify which statement within a transaction can be indicated by (a) assigning a number to each statement and (b) including the statement number in each log record generated when the statement is executed. For example, if transaction TX1 has three statements ST1, ST2, and ST3, these three statements can be assigned corresponding numbers 1, 2, and 3. All log records generated for changes made to statement ST1 will include statement number 1, while all log records generated for changes made to statement ST2 will include statement number 2. In cases where a statement (such as statement ST2) causes multiple log records to be generated, the last log record will contain the last-log-of-statement flag to indicate that it is the last log record for statement ST2. As will be described in more detail below, the statement number and the last-log-of-statement flag are used to ensure that the secondary replica has received all transaction log records when the transaction is committed or failover to the secondary replica's master.

[0172] The transaction ID in the log record can be used to determine the state of the transaction that made the changes reflected in the log record. For example, refer to... Figure 7 Suppose that the incremental log entry for row R2 includes the transaction ID TX2. That transaction ID can be used to look up transaction table entry 712 for transaction TX2 in transaction table 710. Transaction table entry 712 contains information about the status of transaction TX2. For example, the transaction table entry 712 can indicate whether transaction TX2 is in execution, preparation, commit, or has been committed. If it has been committed, transaction table entry 712 will indicate the commit time of transaction TX2.

[0173] Cleaning up entries

[0174] When a transaction commits, the incremental log entries and / or row stack entries corresponding to the changes made by the transaction can be updated to include the transaction's commit time. By including the commit time in the entry itself, the need to look up the transaction in the transaction table is avoided. The process of updating entries associated with committed transactions to reflect the transaction's commit time is called "sweeping." For example, sweeping can be performed by replacing the transaction ID in the entry with the transaction's commit time.

[0175] For entries that have not yet been cleaned up but correspond to committed transactions, the version of the data contained in the entry is determined by looking up the transaction in transaction table 710 to determine the transaction's commit time.

[0176] Once a cleanup operation is performed on an entry, the commit time stored in the entry is used to indicate the "version" of the row to which the associated entry belongs. Therefore, the version of the cleaned-up entry can be determined without using transaction table 710 to look up the status of the transaction that made the changes reflected in the entry.

[0177] Cleanup of the logs generated by transactions does not need to be performed when the transaction is committed. Rather, cleanup can be performed periodically, for example, by a background process.

[0178] Semi-synchronous propagation of log records

[0179] As mentioned above, when a DML operation changes data in a slice, the change is made by generating a log record reflecting the change, storing the log record in the incremental log of the slice's primary replica, and propagating the log record to the slice's secondary replicas. However, according to one embodiment, the performance of the DML operation is improved by performing the propagation of the log record to the secondary replicas "semi-synchronously".

[0180] The propagation of log entries to secondary replicas is semi-synchronous because the engine instance that propagates the log entry for the change reports a successful change to the client that requested the change, without waiting for confirmation that the log entry receiving the change has been successfully propagated to the secondary replica. For example, suppose a client requests to insert a row where the primary key of that row falls within the range mapped to slice S3. In these cases, the insert request is executed by engine instance 202B on host 202A, which manages the primary replica (S3D1) of slice S3. During the execution of the request, engine instance 202B inserts an incremental log entry into the incremental log of S3D1 and initiates propagation of the log entry to engine instance 208B (its secondary replica S3D2 of slice S3) and to engine instance 204B (its secondary replica S3D3 of slice S3). After initiating the propagation of the log entry, but before receiving confirmation from the secondary replica that the log entry has been received, engine instance 202B reports to the client that the statement making the change has been successfully executed. Because the completion of a statement is reported as successful before confirmation that the log record has been received at the secondary replica, changes at the primary and secondary replicas are not fully synchronized.

[0181] In one embodiment, the driver at the client end tracks which statements of a transaction have been reported as successfully executed. As will be described in more detail below, if a transaction fails over to a different host, the "last successfully executed statement" information maintained by the client is used to recover the transaction (without having to fully restart it). The fact that engine instance 202B does not wait for confirmation from secondary replicas before notifying clients that statements have been executed can lead to significant performance improvements because the execution of DML operations does not incur latency associated with confirmation.

[0182] According to one embodiment, the engine instance that made the changes specified in the statement reports to the client that the statement has been successfully executed under the following conditions:

[0183] • Changes are recorded in the incremental log of the primary replica, and

[0184] • One or more log entries have been placed "on the wire".

[0185] To be transferred to (one or more) secondary copies.

[0186] In this context, "online" refers to a situation where log records have been propagated to a fault domain other than the fault domain of the engine instance that made the changes. Because the engine instance waits until the log records are online, changes made at the primary and secondary replicas are not entirely asynchronous. Therefore, the term "semi-synchronous" is used to refer to the propagation of log records from the primary replica to the secondary replica.

[0187] In this example, assume host 202A has a network interface card (NIC) that connects host 202A to the network connected to hosts 204A and 208A. Under these conditions, engine instance 202B can report that the statement has been successfully executed for slice S3 when the NIC acknowledges receipt of the log record associated with the statement. The NIC can send this acknowledgment before actually sending the packet containing the log record, as long as the log record is stored in the memory associated with the NIC (as opposed to the memory in which the engine instance runs). Therefore, at the time of NIC acknowledgment, the packet containing the log record may not have yet been sent to the secondary replica host. In this example, the NIC constitutes a separate fault domain because after the NIC acknowledges receipt, the propagation of the log record from the NIC to hosts 204A and 208A should be successful, even if engine instance 202B subsequently fails or hangs.

[0188] Redundantly connected host

[0189] As explained above, the NIC and engine instance belong to different fault domains. Therefore, even if the engine instance that generates the logging fails or hangs, the logging will still be successfully propagated to the secondary replica by the NIC. However, a double failure (failure of the engine instance on the primary replica, and failure of the NIC / network) can still result in the following situation: even if the client has been notified that the corresponding statement was successful, the changes will not be propagated to the secondary replica.

[0190] Therefore, to reduce the possibility of secondary replicas failing to receive changes, each host in the system can be connected to every other host through multiple redundant networks. For example, such as... Figure 2 As shown, host 202A can connect to host 204A through two different networks (network 1, network 2), each network having its corresponding NIC (NIC1, NIC2) within host 202A. In these scenarios, engine instance 202B can simultaneously send log records of changes to slice S3 to host 204A through both NICs / networks. Therefore, log records would only fail to reach host 204A if both NICs / networks failed, which is an extremely unlikely event.

[0191] Although not in Figure 2 As shown, however, each of the two networks (Network 1, Network 2) can connect each host to every other host. In these scenarios, any communication between hosts will only fail if both networks fail. Furthermore, the number of networks between each host can be increased as needed to ensure successful communication. For example, in one embodiment, each host connects to every other host through at least three different networks.

[0192] Garbage collection on row heap entries

[0193] As explained above, as DML operations are performed on a row, the chain of time-series entries for that row grows, with each DML operation adding a new entry to the end of the chain. Similarly, as explained above, by creating new row heap entries for that row, the oldest incremental log entry for that row can be applied to the row heap, allowing the space occupied by incremental log entries to be reclaimed. However, if the chain of time-series entries for each row is allowed to grow indefinitely, the host will eventually run out of persistent storage space.

[0194] Therefore, according to one embodiment, the system periodically reclaims the space occupied by row stack entries residing at the head of the time-series entry chain of a row. The row stack entries residing at the head of the time-series entry chain of a row are referred to herein as "head entries." Head entries contain the oldest available version of the row. For example, the space occupied by a head entry can be reclaimed when the timestamp associated with it is older than a specified threshold age. For example, suppose the system's specified retention period is three days. In these cases, the system is able to handle read requests associated with snapshot times up to three days old. If the commit timestamp of a particular row's head entry is more than three days old, the system can deallocate the head entry and reuse its storage space.

[0195] When dealing with the header entry of a row, it may be necessary to rewrite the row heap entries preceding the header entry. The row heap entry preceding the header entry of a row is called the "second-to-last entry" of that row. The second-to-last entry may need to be rewritten, for example, when the second-to-last entry is a sparse entry.

[0196] For example, suppose row R1 has five columns: c1, c2, c3, c4, and c5. Further suppose the head entry of row R1 is a full-row version, containing values ​​for all five columns up to time T10. Finally, suppose the penultimate entry of row R1 is a sparse row-heap entry, with only values ​​for c1 and c2 up to time T50.

[0197] In these situations, deallocating the header entry of R1 without making any additional changes will cause problems. Specifically, after such deallocation, if the system receives a request to read row R1 with a deadline of T50, the values ​​of c3, c4, and c5 for deadline T50 will no longer be available.

[0198] According to one embodiment, when the header entry of a row with a sparse penultimate entry is deallocated, the system creates a new full-line version of the row reflecting the snapshot time of the penultimate entry. For example, if the penultimate entry of row R1 is a sparse entry with timestamp T50, the system creates a full-line version of R1 up to time T50. This full-line version of R1 replaces the sparse penultimate version of R1. Therefore, the pointers to the time-series entry chain of R1 are revised as follows:

[0199] • Remove the first and second-to-last entries from the sequential entry chain of the row, and

[0200] • Add the new full-line version to the head of the line's sequence entry chain.

[0201] After a new full-line entry has been added to the head of the sequence entry chain for that line, the space occupied by the old head entry and the old penultimate entry can be freed and reused.

[0202] Coordinator Selection

[0203] Refer again Figure 2 When a client requests to execute a transaction, system 200 selects an engine instance to coordinate the transaction. According to one embodiment, the system selects a coordinating engine instance based on the slice the transaction targets. In the simple case where the transaction operates only on data from a single slice, that slice is designated as the "control slice" of the transaction, and the engine instance managing the primary copy of the slice is selected as the coordinating engine instance for that transaction. Therefore, in Figure 2 In the system shown, transactions that operate only on data from slice S3 will be coordinated by engine instance 202B, which manages the primary replica S3D1 of slice S3.

[0204] When a transaction operates on data from more than one slice, the system selects a control slice from among those slices operated on by the transaction. For example, suppose a transaction operates on data from slice S1 and data from slice S2. In this case, the system will select either slice S1 or S2 as the control slice. If slice S1 is selected, the engine instance 200B that manages the primary replica S1D1 of slice S1 is designated as the transaction coordinator. If slice S2 is selected, the engine instance 204B that manages the primary replica S2D1 of slice S2 is designated as the transaction coordinator.

[0205] When a transaction operates on data from multiple slices, various techniques can be used to select the controlling slice. For example, if a transaction operates on slices S1 and S2, but performs more work on slice S1 than on slice S2, then slice S1 can be selected as the controlling slice. In cases where the slice(s)(s) to be modified cannot be determined from the transaction itself, the engine instance receiving the request from the client can simply select any slice that manages its primary replica as the controlling slice and then assume the role of the engine instance coordinating the transaction. Therefore, it is possible for a transaction's controlling slice to be one of the slices touched by the transaction.

[0206] Alternatively, when a transaction operates on multiple slices, the control slice can be selected based on a variety of factors, such as the amount of data on the slice, the slice the transaction first touches, and whether the slice is the most "reliable." For example, a slice may qualify as the most reliable if it has a larger replication factor, or if the primary copy of the slice is on a medium that has experienced fewer failures. The selection of the control slice can also be based on the current workload of the respective host. For example, if host 200A is heavily used while host 204A is relatively idle, slice S2 can be selected as the control slice so that the transaction can be coordinated by engine instance 204B on the less busy host 204A. Other embodiments may consider a variety of factors when selecting the control slice for a transaction, including but not limited to which host receives the request from the client, the current host workload, and the amount of work the transaction must perform on each slice.

[0207] In some embodiments, the client includes logic for selecting a host to which it sends transactions. The client may make the host selection based on factors described above, such as the primary key(s) to which the transaction operates and which hosts manage the primary replicas of the slices to which those primary keys are mapped. In this embodiment, the client may maintain a topology cache indicating the mapping between primary keys and the hosts that manage the primary replicas of the slices to which the primary keys are mapped. In embodiments where the client selects a host, after the host is selected:

[0208] The client connects to the host.

[0209] The slice whose primary replica resides on the host is selected as the control slice (based on one or more primary keys involved in the transaction), and

[0210] The engine instance that manages the primary replica acts as the coordinator for that transaction.

[0211] In an alternative embodiment, the logic for selecting the control slice resides on each host. In this embodiment, the host to which the client sends a transaction can select the control slice whose primary copy resides on a different host. In this case, the transaction and the responsibility for coordinating it can be forwarded to other hosts. Alternatively, the host that initially receives the transaction from the client can send a message back to the client instructing the client to send the transaction to a different host (i.e., the host that manages the primary copy of the slice selected as the control slice by the host that received the request).

[0212] Client-assisted failover

[0213] When the engine instance coordinating a transaction fails or stops working for any reason, the transaction becomes a "failover transaction," and a new coordinator is selected to resume the failover transaction. According to one embodiment, the new coordinator is selected from the engine instances that manage secondary copies of the control slice.

[0214] In addition to selecting a new coordinator, all the remaining hosts set their logical clocks to max(the current clock value, the highest timestamp generated by the failed host). The technique used to determine the highest timestamp generated by the failed host is described below.

[0215] Regarding the selection of a new coordinator for a failover transaction, if slice S3 is the control slice of the transaction, engine instance 202B will initially be designated as the coordinator for the transaction (because engine instance 202B manages the primary copy (S3D1) of slice S3). If engine instance 202B stops running, the system selects a new coordinator for the transaction. In this case, the candidates for coordinating the transaction are engine instance 204B (which manages access to a secondary copy S3D3 of the control slice S3) and engine instance 208B (which manages access to another secondary copy S3D2 of the control slice S3). When multiple secondary copies of the control slice exist, a new coordinator can be selected based on various factors, such as the busyness of the respective hosts and which host has the most log entries for the transaction.

[0216] According to one embodiment, if candidates to become the new coordinator for a transaction do not have the same number of log records for the transaction, the candidate with the highest number of transaction log records is selected. For example, suppose that for a failover transaction, engine instance 208B has more log records than engine instance 204B. In these cases, engine instance 208B will be selected as the new coordinator for the failover transaction.

[0217] Before transactions resume execution at the new coordinator, any missing transaction log records from other candidates are sent from the new coordinator to those candidates. In this example, engine instance 208B sends any missing transaction records from the failover transaction to engine instance 204B. Transaction log records and how they are used during failover will be described in more detail below.

[0218] Once a new coordination engine instance is selected, the secondary replicas of the control slice managed by the new coordination engine become the new primary replicas of the control slice. For example, if engine instance 208B is selected as the new coordination process for transactions in its control slice S3, then secondary replica S3D2 is designated as the new primary replica of slice S3. Furthermore, the client sends a message to engine instance 208B to allow engine instance 208B to resume transactions initiated by the now-failed engine instance 202B.

[0219] To enable engine instance 208B to recover the transaction, the client sends information about the transaction and an indication of the last change confirmed by the previous coordinator to host 208A. For example, prior to the failure, engine instance 202B would have sent a series of messages to the client related to the transaction's status. Each message confirms that a statement was successfully executed. Therefore, the client stores the highest statement number that was confirmed by engine instance 202A before the failure.

[0220] When engine instance 202B fails, the client sends a request to the new coordinator (engine instance 208B) to resume the transaction, along with the highest statement number confirmed for execution by the previous coordinator (engine instance 202B). The new coordinator (engine instance 208B) then resumes the transaction execution at the statement following the statement associated with the statement number received from the client.

[0221] For example, suppose transaction TX1 has ten statements (ST1 to ST10), and slice S3 is selected as the control slice for this transaction. In these cases, engine instance 202B (the primary replica managing slice S3) is selected as the coordinator for the transaction. During transaction execution, engine instance 202B successfully executes statements ST1 to ST4, sending an acknowledgment message to the client each time a statement is successfully executed. After failover, secondary replica S3D2 is designated as the primary replica, and engine instance 208B becomes the new coordinator for the transaction. The client notifies engine instance 208B that ST4 is the last statement to be successfully executed, therefore the new coordinator (engine instance 208B) resumes transaction execution at statement ST5.

[0222] Failover using semi-synchronous propagation with logging

[0223] As mentioned above, changes can be propagated to secondary replicas semi-synchronously. That is, the coordinator can indicate to the client that the statement has been successfully executed on the primary replica of the slice before the secondary replica's primary replica acknowledges receipt of the changes made by the statement.

[0224] In embodiments using semi-synchronous propagation with logging, the coordinator can send acknowledgments to clients after pushing changes to different fault domains, such as acknowledgments to the local NIC that the logs have been received by that NIC. However, even in these cases, the possibility that secondary replicas will not receive the logs is extremely small, but still possible. Therefore, in the example given above, the changes made by statement ST4 in transaction may not have been propagated to secondary replicas S3D2 and S3D3.

[0225] Therefore, according to one embodiment, after a failover, before the new coordinator resumes the transaction, the new coordinator verifies that its replica has log records for all statements up to and including the last confirmed statement. In this example, engine instance 208B does not resume the execution of the transaction at statement ST5 until it verifies that replica S3D2 has all log records for the transaction up to and including statement ST4 (including log records containing the end-of-statement flag for ST4). In the event of any missing log records, the new coordinator aborts the transaction, and the transaction can then be re-executed from the beginning. With all log records present, the new coordinator resumes the transaction at the next statement (ST5).

[0226] Adjusting the clock when the host fails

[0227] As mentioned above, when a host fails, all remaining hosts set their logical clock to max(the current clock value, the highest timestamp generated by the failed host). However, determining the highest timestamp generated by the failed host is not straightforward. Therefore, a leasing technique is used so that non-failed nodes always know a timestamp that is at least as high as the highest timestamp generated by the failed host.

[0228] According to the leasing technology, the "maximum clock value" is transmitted to all hosts in the system. A lease grants a host permission to generate timestamps up to the maximum clock value. Whenever the logical clock of any host in the system reaches the maximum clock value, the host must request an additional "lease." Upon receiving a new lease request, a new maximum clock value is selected, and a new lease is granted by transmitting the new maximum clock value.

[0229] In systems using this leasing technology, it is guaranteed that no host in the system will see a timestamp value greater than the current maximum clock value granted by the leasing mechanism. Therefore, when a host fails, all hosts in the system can set their clocks to the current maximum clock value, ensuring that their clocks are at least as high as any timestamps generated by the failed host node. When clocks are adjusted in this way after a host failure, a new maximum clock value is selected, and leases are granted to the remaining hosts to generate timestamps up to the new maximum clock value.

[0230] Deterministic response time

[0231] In one embodiment, the system responds to all commands within a specified maximum time. Otherwise, if the execution of a transaction exceeds the specified maximum time, the engine instance coordinating the transaction returns the results of the executed transaction statements to the client along with a recovery token. The recovery token contains information needed by the engine instance to recover the transaction, such as the number of the last executed statement.

[0232] Upon receiving the intermediate results and the recovery token, the client can resubmit the transaction along with the recovery token. The coordination engine instance assigned to the resubmitted transaction resumes the transaction's execution at the appropriate statements based on the contents of the recovery token. Using the recovery token in this way not only allows for guaranteed response times but also allows the host to largely forget the state of the transaction between the time the recovery token was sent to the client and the time the client resubmitted the transaction (and thus free up resources).

[0233] It's possible that to avoid exceeding a time threshold, the engine instance must stop processing statements in between commands. For example, a command might request a scan of the entire table. In these cases, it's possible that only a partial scan of the table is performed when the time threshold is reached. If the partially performed scan was performed in primary key-based order, the recovery token can include the primary key of the last scanned row. Therefore, when the operation resumes, the scan can resume to the next row. Similarly, if the scan is sorted by another column with a secondary index, the recovery token can indicate the last scanned value from the indexed column. When the operation resumes, the secondary index on that column can be used to resume the scan at the row containing the next value from the indexed column.

[0234] Multi-slice statements

[0235] A single statement can involve multiple slices. For example, a statement might request a scan of table T, which is divided into five slices S1-S5, with copies stored in [location missing]. Figure 2 In the system shown, as mentioned above, when a transaction operates on multiple slices, the system selects a control slice, and the engine instance that manages the primary replica of the control slice coordinates the transaction.

[0236] In the case of a table scan operation, a scan of a given slice can be performed by any engine instance that manages any copy of the slice. According to one embodiment, the engine instance coordinating the transaction distributes work among other hosts in a manner that maximizes parallelism and workload balancing between hosts. For example, if engine instance 202B is selected to coordinate a full scan of table T1, engine instance 202B can scan S1D2 and assign engine instance 200B to scan S2D2, engine instance 204B to scan S3D3, engine instance 208B to scan S4D1, and engine instance 206B to scan S5D1. These scan operations can be performed in parallel, and the results are returned to the coordinating engine instance 202B. The coordinating engine instance 202B then sends the results back to the client that requested the table scan.

[0237] The logic within an engine instance that coordinates the work of statements accessing multiple slices is referred to herein as a statement coordinator. According to one embodiment, the statement coordinator breaks down the work required by a statement into statement fragments, where each statement fragment specifies work to be performed by a different engine instance. These statement fragments are then sent to their respective engine instances so that these engine instances can perform the work specified therein. Thus, in this example, the statement coordinator of engine instance 202B creates five statement fragments for a statement requiring a full scan of table T and sends those statement fragments to the appropriate engine instances so that the table scan is performed in parallel.

[0238] transactions

[0239] As mentioned above, database operations performed by system 200 are often executed as part of a transaction. A transaction is a unit of work that must be executed atomically. Therefore, if system 200 has executed some of the work specified in a transaction but is unable to complete the rest, system 200 must abort the transaction and "roll back" the work already executed.

[0240] Each transaction can contain multiple statements. Within each transaction, statements are executed sequentially. However, the work of one transaction can be executed concurrently with the work of other transactions. Furthermore, the work specified within any given statement can be divided and executed in parallel. As mentioned above, the log records generated while executing transaction statements include both the statement number and the transaction ID.

[0241] When a transaction is committed, the commit time is assigned to the host based on the logical clock of the coordinating engine instance running the transaction. As mentioned above, the engine instance selected to coordinate the transaction is chosen by determining the control slice of the transaction and then selecting the engine instance that manages the primary copy of the control slice as the transaction coordinator.

[0242] As mentioned above, statements within a transaction can request DML operations to be performed on multiple slices. In these cases, the primary replicas of some of those slices can be managed by an engine instance residing on a host other than the host to which the coordination engine instance belongs. For example, suppose a transaction updates data residing in both slices S1 and S3. In this case, slice S1 can be chosen as the control slice. Since S1 is the control slice, engine instance 200B, which manages S1D1 (the primary replica of slice D1), is selected as the coordination engine instance.

[0243] To perform DML operations on slice S3, the coordination engine instance 200B sends a statement fragment to the engine instance managing the primary replica S3D1 of slice S3. Figure 2In the illustrated embodiment, the engine instance managing S3D1 is engine instance 202B on host 202A. The statement fragment specifies the work that must be completed on slice S3. Engine instance 202B executes the requested work and transmits the completed work back to coordinating engine instance 200B.

[0244] Prerequisites for committing a transaction

[0245] The transaction coordination engine instance cannot commit a transaction until it confirms that all the work required for the transaction has been successfully performed. In a system where each slice can have one or more secondary replicas and each transaction can perform DML on multiple slices, the work of a transaction includes (a) performing DML on the primary replica of the slice and (b) propagating log records to the secondary replicas.

[0246] Furthermore, performing DML on the primary copy of a slice can involve (a1) a coordination engine instance performing work on the primary copy of a control slice, (a2) a coordination engine instance performing work on the primary copies of one or more non-control slices, and (a3) ​​one or more non-coordination engine instances performing work on the primary copies of one or more other non-control slices.

[0247] Therefore, a commit protocol is needed to ensure that transactions are not committed until all of the following conditions occur:

[0248] The coordination engine instance has performed all the work requested on the primary replica of the control slice.

[0249] The coordination engine instance has performed all requested work on any non-controlling slices for which it manages the primary replica.

[0250] The non-coordination engine instance has already performed all requested work on the primary replica of any other slices that have been modified by the transaction, and

[0251] • The log records of all changes made by the transaction to the primary replica have been successfully propagated to the corresponding secondary replicas.

[0252] As mentioned earlier, the coordination engine instance can report to the client that a DML operation requested by the client has been successfully executed without waiting for confirmation from one or more hosts that the log record corresponding to the DML operation has actually been received. These hosts include one or more secondary copies of the slice modified in the DML operation. However, the coordination engine instance needs to obtain such confirmation before committing. For example, suppose statement ST1 in transaction TX1 requests changes to rows mapped to slice S1. In these cases, engine instance 200B can confirm to the client that statement ST1 has been executed after updating the primary copy S1D1 and sending the corresponding log record to the NIC of host 200A. However, before committing transaction TX1, engine instance 200B must receive confirmation that the log record has been successfully received by host 202A (which hosts the secondary copy S1D2 of slice S1).

[0253] Furthermore, before committing transaction TX1, the coordinator engine instance must receive confirmation that all other engine instances performing work as part of transaction TX1 are ready to commit. For example, if statement ST2 of TX1 specifies a DML operation on data in slice S3, then coordinator engine instance 200B will send a request to engine instance 202B (which manages the primary copy S3D1 of slice S3) to perform the DML operation on slice S3.

[0254] A non-coordinating engine instance performing DML on the primary replica of a non-controlling slice cannot report to the coordinating engine instance that it is ready to commit until it receives confirmation that the log record of its changes to the non-controlling slice has been successfully propagated to the engine instance managing the secondary replicas of the non-controlling slice. In this example, engine instance 200B sends a statement fragment to engine instance 202B to instruct engine instance 202B to perform the DML operation requested for slice S3 on data in S3D1. In these cases, engine instance 202B cannot report that it is ready to commit until engine instance 202B receives confirmation that the log record corresponding to its changes to S3D1 has been successfully propagated to S3D2 and S3D3, managed by engine instances 208B and 204B, respectively.

[0255] Communication during statement execution

[0256] Before explaining the operations involved in committing a transaction, we will provide an explanation of the various communications that occur during the execution of statements within the transaction. (Reference) Figure 9This is a block diagram of system 900 according to an embodiment, which will be used to interpret transaction commit protocols that can be used by a slice-based distributed database system. System 900 includes six hosts 950, 952, 954, 956, 958, and 960. Hosts 950, 952, 954, 956, 958, and 960 execute engine instances 902, 904, 906, 908, 910, and 912, respectively. Engine instance 902 manages the primary replica 930 of slice S1, while engine instance 912 manages the primary replica 932 of slice S2.

[0257] Slice S1 has two secondary copies 934 and 936 managed by engine instances 904 and 906, respectively. Slice S2 has two secondary copies 938 and 940 managed by engine instances 908 and 910, respectively.

[0258] For the purpose of explaining the commit protocol, it will be assumed that client 990 commits transaction TX1, which includes two statements ST1 and ST2, where statement ST1 performs DML on slice S1 and statement ST2 performs DML on slice S2. It will also be assumed that slice S1 is selected as the control slice for transaction TX1. Because engine instance 902 manages the primary copy of the control slice S1, engine instance 902 is designated as the coordinator for transaction TX1.

[0259] The coordinator's execution of statements

[0260] In this example, engine instance 902 executes statement ST1 in the following way:

[0261] • Modify the main copy 930 of slice S1

[0262] • Semi-synchronously send the changed log records to secondary copies 934 and 936 of slice S1.

[0263] • Semi-synchronously send the statement end savepoint message of statement S1 to the secondary copies 934 and 936 of slice S1.

[0264] • Send a statement completion confirmation message for statement S1 to client 990.

[0265] Non-coordinator execution statement

[0266] Because statement S2 involves performing DML operations on slices managed on a host other than the host of the control engine instance, its primary replica is located on a host other than the host of the control engine instance, additional communication is required. The communication required to execute statement ST2 is in Figure 9 As shown in the image.

[0267] refer to Figure 9 The presentation of statement S2 from client 990 to host 950 is illustrated as "(1) statement" to indicate that the client's presentation of statement ST2 is Figure 9 The first action is shown. According to one embodiment, engine instance 902 includes statement coordination logic and transaction coordination logic. If necessary, the statement coordination logic splits the statement ST2 received from client 990 into statement fragments, where each fragment operates on a slice whose primary replica is managed by a different host. The statement fragments are then sent to the engine instance that manages the primary replicas of those slices.

[0268] In this example, statement S2 contains only one fragment F1, which indicates the DML to be performed on slice S2. Engine instance 902 transfers fragment F1 to engine instance 912 (which manages the primary copy 932 of S2) for execution. Fragment F1 is then transferred to host 958. Figure 9 The text is labeled "(2) DML fragment" to indicate that the transmission of fragment F1 is... Figure 9 The second action shown.

[0269] The transaction coordination logic on host 950 tracks the state of transaction TX1 and stores the transaction log record indicating the transaction state of transaction TX1 in the primary replica of the control slice. In this example, the transaction coordination logic of engine instance 902 stores the transaction log record of transaction TX1 in primary replica 930. As will be described in more detail below, these transaction log records (different from the data log record containing incremental log entries) are propagated to secondary replicas 934 and 936 of slice S1 before TX1 is committed.

[0270] Because transaction log records are propagated to secondary copies of the transaction's control copy, any of the engine instances(s) managing those secondary copies(s) can be used as a backup coordinator for the transaction. Therefore, as mentioned earlier, if the transaction's coordinating engine instance fails, one of the backup coordinators is selected to restore the transaction.

[0271] After engine instance 912 receives statement fragment F1, it executes the statement fragment to perform the specified DML operation on the data in the primary replica 932 of slice S2. Log records containing incremental log entries reflecting those changes are then propagated to secondary replicas 938 and 940 of slice S2. The propagation of data log records to hosts 956 and 960 is illustrated as "(3) Data Log Recording" to indicate that the propagation of data log records is chronological. Figure 9 The third action shown in the diagram. As mentioned above, the propagation of log records is performed in a semi-synchronous manner, where engine instance 912 waits for the local NIC to acknowledge receipt of the log records used for transmission, but does not wait for acknowledgments of log record reception from hosts 956 and 960.

[0272] After semi-synchronously transmitting the log records to the secondary copies 938 and 940 of slice S2, engine instance 912 reports slice execution completion to coordinating engine instance 902. In the "Slice Complete" message, engine instance 912 includes information identifying the last log record generated for changes made to engine instance 912 during slice execution. For example, if the slice is associated with statement ST2 of a transaction, and the slice execution produced three log records, the slice complete message may include (a) statement number ST2, and (b) the log record sequence number of the last log record of that statement (i.e., 3). The transmission of the "Slice Complete" message is illustrated as "(4) Slice Complete + Last_LOGREC_ID" to indicate that the transmission of the "Slice Complete" message is chronologically ordered. Figure 9 The fourth action shown.

[0273] After receiving confirmation from all engine instances involved in the statement execution that their statement portions have been fully executed, the coordinator stores the "statement end-of-savepoint transaction log" in its transaction log and semi-synchronously sends the statement end-of-savepoint transaction log to the backup coordinator (the host controlling the secondary copy of the slice). According to one embodiment, the statement end-of-savepoint transaction log includes:

[0274] • The transaction ID of the transaction to which the statement belongs

[0275] • Report number

[0276] • Number of retries (the number of attempts made to execute the statement)

[0277] • For each slice touched by the statement, a slice-specific record is provided, which includes the slice ID, a LogRecID indicating the sequence number of the last log record, and the changes made to the specified slice by the statement.

[0278] In this example, after receiving a fragment completion message from engine instance 912, engine instance 902 stores the statement end savepoint transaction log and transmits it to engine instances 904 and 905, which respectively manage secondary replicas 934 and 936 of control slice S1. The semi-synchronous transmission of the statement end savepoint transaction log is as follows: Figure 9 The “(5) Statement End Savepoint Transaction Log” indicates that this transfer is in chronological order. Figure 9 The fifth action depicted in the text.

[0279] As mentioned earlier, the coordination engine instance sends an acknowledgment to the client that the statement has been successfully completed without waiting for the semi-synchronous transmission associated with the statement to actually arrive at its destination. Therefore, after providing the statement end-of-statement savepoint transaction log to the NIC for transmission to engine instances 904 and 906, engine instance 902 reports to client 990 that statement S1 has been successfully completed.

[0280] Coordination Engine Instance

[0281] At the start of a transaction, the coordination engine instance (engine instance 902 in the example above) stores the transaction record of the transaction in the transaction table of the primary copy of the control slice. The transaction record includes the transaction ID and status information about the transaction (e.g., whether the transaction is active, committed, aborted, etc.). Additionally, transaction entries may also include:

[0282] • Retry information for executed statements, and

[0283] • A list of “participating slices” and their corresponding “last LogRecID”.

[0284] The list of participating slices is a list of non-controlling slices that are the targets of DML operations within a transaction. In the example given above, slice S1 is the control slice, while slice S2 is a participating slice because S2 was updated during the transaction in which slice S1 was the control slice. The last LogRecID associated with a participating slice indicates the last LogRecID received by the coordination engine instance from the engine instance that performed DML on the participating slice. Figure 9 In the example shown, the LogRecID associated with the participating slice S2 is the LogRecID sent by the engine instance 912 to the host 950 after the engine instance 912 completes the execution of statement ST2 (i.e., action (4)).

[0285] As will be explained below, the coordinating engine instance uses state information to track statement completion and coordinate commit protocols. In one embodiment, the state information is maintained in the transaction record of the transaction, which may reside in the primary copy of the transaction's control slice. However, in an alternative embodiment, the state information may be stored elsewhere, provided it is accessible to the coordinating engine instance. References will follow below. Figure 10 Describe how the coordination engine instance uses transaction state information.

[0286] Branch Coordinator

[0287] When a transaction's coordinating engine instance sends a statement fragment to another engine instance to perform a DML operation, that other engine instance is responsible for coordinating the transaction's "branch." Figure 9In the example shown, engine instance 912 is the "branch coordinator" responsible for coordinating the branches of transaction TX1, which are coordinated by engine instance 902.

[0288] According to one embodiment, each branch coordinator stores an entry in a local transaction table that indicates:

[0289] Mapping from local transaction ID to global transaction ID

[0290] • The status of the local branch of a transaction (active, committed, aborted, etc.)

[0291] • Timestamp information

[0292] These "branch transaction entries" are propagated to the secondary copies of the primary copy that are updated during branch execution using semi-synchronous propagation. For example, during the execution of a branch by engine instance 912, engine instance 912 performs a DML operation on the primary copy 932 of slice S2. Therefore, the branch transaction entries of the branch are propagated to the secondary copies 938 and 940 of slice S2 using semi-synchronous propagation.

[0293] Transactions can see their own uncommitted changes. However, transactions can only see committed changes from other transactions. Additionally, if a change's commit time is equal to or earlier than its snapshot time, the transaction can only see those changes from other transactions. The combination of transaction state information and timestamp information allows reads to be performed on secondary replicas. For example, suppose a client sends a request to host 956 to read data from slice S2 up to snapshot time T10. Engine instance 908 checks the transaction state information to determine if changes made to slice S2 during the transaction branch have been committed. If the changes to slice S2 have been committed, engine instance 908 checks the timestamp information to determine if the changes made in the transaction branch fall within snapshot T10. If the changes are uncommitted or too recent (i.e., the branch's commit time is greater than T10), engine instance 908 skips entries in the time-series entry chain associated with changes made during the transaction branch to locate older entries containing data falling within snapshot T10.

[0294] Update logical clock

[0295] The commit times of two transactions updating the same data item must reflect the order in which changes were made to that data item. Therefore, if TX1 updates and commits a data item, and TX2 then updates and commits the same data item, then TX2 must be assigned a later timestamp than TX1, even if TX2 is coordinated by a different host than the one that coordinated TX1.

[0296] To ensure that no transaction is assigned a commit timestamp earlier than a subsequent transaction that touches the same data item, hosts include the current value of their logical clock in messages sent to other hosts. For example, according to one embodiment, the prepare confirmation message sent by all participants in a transaction to the coordination engine instance includes the current value of their respective host's logical clock.

[0297] All nodes involved in a transaction must report their logical clock values ​​to the coordinating engine instance to ensure that the commit time assigned to the transaction is later than the logical clocks of the participating nodes when they became involved in the transaction. Specifically, the nodes that must report their clock values ​​back to the coordinating engine instance include (a) all primary replicas affected by the transaction, and (b) all secondary replicas of the primary replicas affected by the transaction. As will be described in more detail below, the prepare confirmation message sent from backup coordinating engine instance 904 will include the current value of the logical clock of host 952. The prepare confirmation message sent from backup coordinating engine instance 906 will include the current value of the logical clock of host 954. The prepare confirmation message sent from branch coordinating engine instance 912 will include the current value of the logical clock of host 958.

[0298] If any prepare acknowledgment message includes a logical timestamp greater than the current value of the logical clock on the host (e.g., host 950) of the coordination engine instance, the logical clock of the host of the coordination engine instance is updated to reflect the time greater than the highest logical clock value it has received. For example, suppose the current value of the logical clock on host 950 is T500. In this case, if prepare acknowledgment messages from engine instances 904, 906, and 912 contain timestamps T300, T400, and T600, respectively, the logical clock on host 950 will be updated to at least T601. This clock adjustment is performed before the coordination engine instance obtains the candidate commit time from transaction TX1 using the logical clock on host 950.

[0299] Commit transaction

[0300] refer to Figure 10 It illustrates the method used in the already executed Figure 9 The protocol for committing transaction TX1 is followed by the actions shown. Initially, the transaction is active when the statements of the transaction are executed. The "active" state of the transaction is reflected in the transaction log maintained by the coordination engine instance (e.g., engine instance 902), the backup coordinator (e.g., engine instances 904 and 906), and the branch coordinator (e.g., engine instance 912).

[0301] After all statements in the transaction have been executed, the client submits a commit command. Upon receiving the commit command, the coordination engine instance 902 receives confirmation that all statements (ST1 and ST2) of TX1 have been executed. The coordination engine instance 902 has updated the transaction record of TX1 accordingly and has semi-synchronously sent the transaction record to the secondary copies 934 and 936 of control slice S1. Engine instance 902 also reports to the client that statements S1 and S2 have been fully executed (along with the last LogRecID of each statement).

[0302] In response to the commit command, the coordination engine instance transitions to the "ready" state through the following operations:

[0303] • Update the status information in the transaction record of TX1, and

[0304] • Send a readiness request message to all engine instances participating in the transaction, including the backup coordinator, the branch coordinator (which hosts the primary copy touched by the transaction), and the host of all secondary copies of all slices touched by the transaction.

[0305] To send a prepare request message to the participants in a transaction, the coordination engine instance reads a list of participants from the transaction's transaction record. According to an embodiment, the coordination engine instance also obtains the transaction's "preparation timestamp." The transaction's preparation timestamp can be the current value of the logical clock of the host where the coordination engine instance resides.

[0306] According to an embodiment, the preparation request message includes:

[0307] • Global transaction ID of the transaction being prepared

[0308] • Preparation timestamp

[0309] • Slice ID of each slice touched by the firm

[0310] • For each slice touched by the transaction, the last LogRecID of the log record associated with the changes made by the transaction to that slice.

[0311] In response to receiving the prepare request message, the transaction participants:

[0312] • If the global preparation timestamp is greater than the current value of the local logical clock, then update the local logical clock to the global preparation timestamp.

[0313] • Use the last LogRecID in the preparation request message to confirm that they have all the log records that the coordination engine instance believes they should have.

[0314] • Obtain the local preparation timestamp from its local logical clock

[0315] Regarding checking whether all required log records exist, it is assumed that the participating engine instance has a copy of slice S1. Further, it is assumed that changes made to slice S1 by a transaction are reflected in three data log records DL1, DL2, and DL3. In this example, the prepare request message sent to engine instance 912 includes the last LogRecID of DL3 for slice S1. In response to receiving the prepare request message from the coordinating engine instance 902, the participating engine instance verifies that it has log records for slice S1 up to and including log record DL3. How to handle errors (e.g., missing log records in the participating engine instance) will be described in more detail below.

[0316] After confirming that they have the required log entries, transaction participants obtain a local prepare timestamp and update the status information in their respective transaction records to indicate that the local prepare timestamp and transaction TX1 are in a "ready" state. After updating their respective transaction records, the transaction participants send a prepare acknowledgment message to the coordination engine instance. According to one embodiment, each prepare acknowledgment message includes a local prepare timestamp. This timestamp value represents the transaction's local "preparation time." As will be described in more detail below, the coordination engine instance uses these local prepare times to ensure that the commit time of transaction TX1 is greater than all local prepare times.

[0317] After receiving the readiness confirmation message from all transaction participants, the coordination engine instance:

[0318] • Update its local logical clock to a value greater than the highest local readiness time (if any local readiness time is greater than the current time of the coordination engine instance's logical clock).

[0319] • Obtain the candidate submission time based on the current value of its local logical clock.

[0320] • Update its transaction record with the candidate commit time and an indication that the transaction is in a "ready / committing" state, and

[0321] • Send a message containing candidate commit times to each backup coordinator.

[0322] In one embodiment, the "candidate commit time" sent by the coordination engine instance to the backup coordinator is obtained by incrementing the logical clock value of the coordination engine instance and then using the new value of the coordinator's logical clock as the candidate commit time. In other words, the candidate commit time is set to max(local ready time, current local clock) + 1.

[0323] In this example, the logical clock on host 950 increments, and its value is then sent to engine instances 904 and 906 as the candidate commit time for TX1. Because this occurs after the logical clock has been updated based on the local prepare times of all hosts involved in the transaction, the candidate commit time is guaranteed to be higher than the prepare times of all participants in the transaction.

[0324] In response to a message containing a candidate commit time, the backup coordinator updates their transaction logs with the candidate commit time and an indication that the transactions are in a "ready / committing" state. The backup coordinator then sends a "confirm commit time" message back to the coordination engine instance.

[0325] After receiving confirmation commit time messages from all backup coordinators, the coordinating engine instance:

[0326] • Change the transaction status to "committed".

[0327] • Inform the customer that the transaction has been committed (and provide the commit time).

[0328] • Send committed transaction records to the backup coordinator

[0329] In response to receiving a committed transaction log record, the backup coordinator updates its transaction log record to reflect that the transaction has been committed.

[0330] After a transaction is committed, a commit confirmation message is sent (asynchronously) to all participants in the transaction. In response to the commit confirmation message, cleanup is performed on all entries corresponding to the transaction (incremental log entries and / or row heap entries). After cleanup, the participants in the transaction send a commit confirmation message back to the coordination engine instance.

[0331] The report has been submitted.

[0332] In the above commit protocol, the coordinating engine instance does not inform the client that the transaction has been committed until it receives a commit confirmation message from the backup coordinator. The reason the coordinating engine instance waits until it receives the commit confirmation message from the backup coordinator before reporting the transaction's commit is that, in some cases, the transaction's commit time may change from the candidate commit time initially set by the coordinating engine instance.

[0333] For example, suppose a transaction is initially assigned a candidate commit time T121. Further suppose the coordinating engine instance (e.g., engine instance 902) marks the transaction as "committed" with a deadline of T121 before receiving a commit time confirmation from the backup coordinators (e.g., engine instances 904 and 906). At this point, a read operation with a snapshot time of T125 will see the changes made by the transaction.

[0334] However, the coordinating engine instance may crash at time T130 before it has received a commit time confirmation from the backup coordinator. In these cases, the backup coordinator (e.g., engine instance 904) can be assigned as the new coordinating engine instance for the transaction.

[0335] A new coordination engine instance that may not be aware of the candidate commit times for T121:

[0336] • Re-prepare the transaction to be committed (confirm that all participants are ready to commit the transaction).

[0337] • Select a new candidate submission time (e.g., T135).

[0338] Send candidate commit times to the backup coordinator.

[0339] • When confirmation is received, the new submission time will be communicated to the client.

[0340] Because the transaction now has a commit time T135 greater than T125, read operations performed before the coordination engine instance crashed (which used a snapshot time of 125) cannot be repeated. Specifically, performing the same read operation with the same snapshot time will produce results that do not include the changes made by the transaction (because the new commit time is after the snapshot time of the read operation).

[0341] Blackout range of recovered transactions

[0342] According to one embodiment, the commit protocol is modified so that the coordinating transaction reports to the client that the transaction committed when the candidate commit time message is "online," without waiting for confirmation of the commit time message from the backup coordinator. To avoid non-repeatable read problems that occur when the backup coordinator recovers such a transaction without having received the original candidate commit time, the backup coordinator for such recovery transactions assigns an "interruption range" to the transaction.

[0343] According to one embodiment, the interruption range of a transaction recovered by the backup coordinator is the time period between (a) the transaction's maximum preparation time and (b) the commit time assigned to the transaction by the recovery backup coordinator. In the example given above, assume that the maximum preparation time received by the original coordination engine instance 902 is T124. Therefore, coordination engine instance 902 selects T125 as the commit time. Coordination engine instance 902 reports that transaction TX1 has been committed (before receiving confirmation of the commit time from backup coordinators 904 and 906), and then crashes.

[0344] Then, you can choose to back up the coordinator engine instance 904 to restore transaction TX1. Engine instance 904, acting as the new coordinator, re-prepares the transaction. The maximum preparation time received by engine instance 904 will again be T124. However, the internal logical clock at engine instance 904 can be T134. Therefore, engine instance 904 can assign a commit time of T135 to transaction TX1. In this example, the "interruption range" for restoring transaction TX1 is the time range between, but not including, T124 (maximum preparation time) and T135 (commit time of the restored transaction).

[0345] After an interruption range is assigned to a committed and recovered transaction, if a read operation with a snapshot time within the interruption range attempts to read data items touched by the recovered transaction, an error will be reported. For example, if transaction TX1 commits at commit time T135 and the interruption range is T125-T134, an error will be generated if any read operation with a snapshot time within the T125-T134 range attempts to read rows touched by transaction TX1. In response to receiving such an error, the client can recommit read operations with snapshot times outside the interruption range (e.g., reading snapshot time T157).

[0346] Change state and timestamp information atomically

[0347] Often, it's necessary to make multiple changes atomically (so that a process either sees all the changes or sees none). For example, after obtaining a candidate commit time, the reconciliation engine instance must change the transaction's state to "committing" within the transaction record and change the timestamp to the candidate commit time. Importantly, no process sees one of these changes while ignoring the others.

[0348] According to an embodiment, if the hardware does not support making these two changes atomically, atomicity can be achieved by (a) storing an invalid timestamp in the transaction record, (b) changing the state information in the transaction record, and (c) storing the candidate commit time on an invalid timestamp in the transaction record. Using this technique, the process reading the transaction record will see:

[0349] • Status = Ready, Timestamp = Ready Timestamp

[0350] • Status = Ready, Timestamp = Invalid Timestamp

[0351] Status = Submitting, Timestamp = Invalid Timestamp

[0352] • Status = Committing, Time = Candidate Commit Timestamp

[0353] If the read process sees an invalid timestamp (e.g., timestamp 0), it knows the transaction record is being updated and therefore does not use that information. Instead, the read process waits until it sees a valid timestamp, at which point it will see a "candidate commit timestamp" and its status will be "committing".

[0354] Channels and Auto-commit Transactions

[0355] In the example given above, the commit protocol is initiated when a commit command is received from the client. However, in the case of auto-commit transactions, the commit command can be implicit. According to one embodiment, the configuration parameter of the "channel" between the client and the host is set to "auto-commit". A "channel" is a logical entity that connects the client to the host and ensures the order of statements committed through the channel. Therefore, a channel can be thought of as a "pipeline" through which the client sends commands to the host.

[0356] If a channel's configuration parameter is set to auto-commit, each statement received through that channel is treated as a single-statement transaction. Therefore, upon receiving confirmation from that channel that the operation specified in the statement has been completed, the coordination engine instance automatically initiates a commit protocol, even without receiving an explicit commit command.

[0357] Although a channel is initially established as a conduit between the client and the host, it can be considered a client-side entity because the client associated with the channel does not change, but the host associated with the channel can change. For example, when a transaction fails over from one host to another, the channel through which the client commits commands for the transaction connects to the new host, and the client continues to use the same channel to commit commands for the recovered transaction to the new host.

[0358] Urgent preparation

[0359] According to one embodiment, for auto-commit transactions, it is possible to embed messages typically sent during the preparation phase of a transaction onto messages sent during the active phase of the transaction. For example, the transmission of data log entries containing changes made by the transaction can be combined with the transmission of a prepare request message. In these cases, the "preparate request message" may simply be a flag in the data log entry. When this flag is set, the receiver knows the changes made by the preparation transaction and embeds a prepare timestamp into its acknowledgment message for the received data log entry.

[0360] Refer again Figure 9Suppose that an auto-commit transaction specifies changes to data in slice S1. S1 is selected as the control slice, and therefore engine instance 902 is selected as the coordinating engine instance. After changes are made to the primary replica 930 of S1, engine instance 902 sends a data log record containing those changes to hosts 952 and 954, which respectively store secondary replicas 934 and 936 of slice S1. For urgent preparation, the message including that data log record also serves as a preparation request message. Therefore, hosts 952 and 954 respond to the message by determining the preparation time of the transaction and sending back an acknowledgment to engine instance 902, which not only indicates that the data log records have been received but also includes their corresponding preparation times for the transaction. Based on the preparation times, engine instance 902 selects a candidate commit time and proceeds with the steps of committing the transaction (e.g., ...). Figure 10 (As shown in the diagram). Therefore, the need for a separate "preparation" phase is avoided. If an autocommit transaction changes data in two slices (e.g., S1 and S2), the engine instances responsible for the primary replicas of those slices will make their corresponding changes and send their corresponding log entries to the appropriate secondary replicas. All secondary replicas will respond with the data log entry as a preparation request message, and with the transaction's preparation time.

[0361] For transactions that are not automatically committed, eagerly preparing the transaction can be inefficient. For example, if a transaction consists of 20 statements, it is inefficient to require participants to "prepare" after each of the 20 statements. If a participant instructs the transaction to be prepared after the first of the 20 statements, then from that point onward, read operations targeting data touched by the transaction will have to treat the transaction as ready, not active. As explained in more detail below, read operations on data that are ready can require additional steps (when the snapshot time of the read is greater than the ready timestamp) to increase the likelihood that the transaction will eventually be assigned a commit time greater than the snapshot time of the read operation.

[0362] In non-auto-commit transactions where the number of statements is known or can be accurately estimated, it may still be worthwhile to eagerly prepare the transaction. In such cases, combining the data log message with the prepare request message will only execute on the content of the last log record known or estimated to be the last statement in the transaction. For example, in a transaction with 20 statements, the prepare request message can be combined with the data log message of the last log record whose changes were made by the 20th statement.

[0363] In cases where the work of the last statement will be performed by the non-coordination engine instance, the coordination engine instance will prepare the request message combined with the DML fragment sent to the non-coordination engine instance. For example, refer again... Figure 9Suppose engine instance 902 is coordinating a transaction in which the last statement operates on slice S2. In these cases, the DML fragment sent from engine instance 902 is combined with a readiness request message. After executing the DML fragment, engine instance 912 includes the readiness request message in the data log records sent to secondary replicas 938 and 940 of slice S2. In this example, engine instances 908 and 910 not only acknowledge receipt of the log records, but also (a) acknowledge that they are ready to commit the transaction and (b) include their local readiness timestamps in the acknowledgments. After receiving those acknowledgments, engine instance 912 updates its local logical clock to obtain its own readiness time for the transaction and acknowledges to coordinating engine instance 902 that it is ready to commit. In this context, "updating the local logical clock" involves setting its logical clock to the larger of (a) its current value and (b) its local readiness timestamp. In its acknowledgment message to engine instance 902, engine instance 912 includes its readiness timestamp.

[0364] According to one embodiment, when a message that is normally sent asynchronously is combined with a preparation request, the message is instead sent synchronously. For example, engine instance 912 does not report to engine instance 902 that it has successfully executed its DML fragment until it receives acknowledgment and preparation timestamps from secondary replicas 938 and 940.

[0365] As explained above, during the preparation phase of a transaction, a prepare message must be sent to any transaction participant that has not received the "eagerly prepared" message. On the other hand, if the eagerly prepared message (a prepare request message carried on other messages such as log entries) is sent to all participants, there is no separate preparation phase. In these cases, when the commit instruction is received, the coordination engine instance will already have the preparation times for all participants. At that point, the coordination engine instance selects a candidate commit time and proceeds to the commit phase, as described above.

[0366] Urgent preparation for auto-commit of two slices

[0367] Eager preparation can even be used for auto-commit transactions involving data from more than one slice. For example, suppose in Figure 9 The system shown automatically commits transactions that touch slices S1 and S2. Further assume that slice S1 is selected as the control slice. Because slice S1 is selected as the control slice, engine instance 902 is designated as the coordinating engine instance. Engine instance 902 executes the portion of the transaction that touches slice S1 and sends a DML fragment to engine instance 912 so that engine instance 912 executes the portion of the transaction that touches slice S2. Engine instance 902 carries a prepare request message on the DML fragment sent to engine instance 912.

[0368] Engine instance 902 sends one or more log records to secondary replicas 934 and 936, so the secondary replicas will reflect the changes made by engine instance 902 to the primary replica 930 of S1. The last of those log records sent to each secondary replica is sent synchronously and includes a preparation request.

[0369] Similarly, engine instance 912 sends one or more log records to secondary replicas 938 and 940, so the secondary replicas will reflect the changes made by engine instance 912 to the primary replica 932 of S2. The last of those log records sent to each secondary replica is sent synchronously and includes a preparation request.

[0370] Engine instances 904 and 906 respond to a data logging / preparation request from engine instance 902 by obtaining a preparation timestamp and sending it to engine instance 902. Similarly, engine instances 908 and 910 respond to a data logging / preparation request from engine instance 912 by obtaining a preparation timestamp and sending it to engine instance 912. Engine instance 912 then increments its clock as needed and obtains a preparation timestamp. Engine instance 912 carries its preparation timestamp on the message confirming the execution of the DML fragment.

[0371] At this point, the coordination engine instance 902 has received the preparation timestamps directly or indirectly from all participants in the transaction. Based on those preparation timestamps, the coordination engine instance 902 increments the current clock as needed and determines candidate commit times. The coordination engine instance 902 can then proceed directly to the transaction commit phase.

[0372] Downgrade from ready state

[0373] When using eager readiness, a transaction is considered "ready" by each engine instance that receives the prepared request message. Because the prepared request message is combined with messages sent before the entire transaction is ready to commit, received read operations may be handled inefficiently. That is, they cannot safely ignore uncommitted changes, whereas they could safely do so if the transaction were not yet ready. According to one embodiment, the system tracks how many read operations attempt to read data touched by transactions that are already eagerly prepared. If the number of read operations exceeds a threshold, the transaction automatically transitions from eager readiness to non-eager readiness. In response to this transition, participants in a ready transaction change their transaction state back to "active." Furthermore, the coordination engine instance discards the readiness time already provided by the participants.

[0374] As a result of a transaction returning to an "active" state, new read operations can access older versions of the data items touched by the transaction, based on the assumption that transactions will be assigned commit times greater than their snapshot times. Therefore, special handling is avoided for cases where the snapshot time is greater than the preparation time. Any transaction transitioning from eager preparation to non-eager preparation must undergo a full preparation phase after the execution of the transaction's last statement, such as... Figure 10 As shown in the image.

[0375] Read data items updated by transactions that have not yet been committed.

[0376] The fact that a data item has been updated by an uncommitted transaction does not necessarily halt the progress of a read operation targeting that data item. For example, suppose the time-series entry chain for row R1 in slice S1 has ten entries, the first five of which are associated with an uncommitted transaction. Further suppose the client submits a request to read row R1 with a deadline of T20. As explained above, any copy of slice S1 can be used to service the read operation. How the read operation proceeds depends on the state of the uncommitted transaction.

[0377] If the uncommitted transaction is still "active," the first five entries can be skipped, and the row version of R1 corresponding to time T20 can be obtained from one or more of the earlier five entries of that row. In this case, it is safe to assume that the read operation does not need to see any changes made by the uncommitted transaction because it is guaranteed that the commit time assigned to that transaction is greater than the snapshot time T20 of the read operation.

[0378] If an uncommitted transaction is in the "ready" phase and its ready timestamp is greater than T20, the first five entries can be skipped, and the row version corresponding to time T20 can be obtained from one or more of the earlier five entries of that row. In this case, it is safe to assume that the read operation does not need to see any changes made by the uncommitted transaction because it is guaranteed that the commit time assigned to that transaction is greater than the prepare time, which is known to be greater than the snapshot time T20.

[0379] On the other hand, if an uncommitted transaction is in the ready phase and its ready timestamp is less than T20, a commit time less than T20 may be assigned to the uncommitted transaction. For example, suppose a read operation reads rows from slice S2 from secondary replica 938 of S2. If the transaction is in the ready state and its ready time is T15, engine instance 908 will have already sent a ready confirmation message with a ready time of T15 to coordinating engine instance 902. Therefore, the commit time of the transaction is guaranteed to be greater than T15, but not necessarily greater than T20. In these cases, the read operation can be paused until a commit time is assigned to the transaction. In some cases, the read operation can be paused until the entries generated by the transaction are cleaned up because the coordinator of the read operation does not know the commit time of the transaction until the entries generated for the transaction are cleaned up on the host executing the read operation.

[0380] If the commit time is greater than T20, the read operation can continue by skipping the first five entries generated by the transaction. On the other hand, if the commit time is less than T20, the changes made by the transaction must be seen by the read operation, so the values ​​from the first five entries will be used as needed to obtain the data required by the read operation.

[0381] Transactions in the "Ready / Committing" state are handled similarly to "Ready" transactions. Specifically, if a read operation has a snapshot time lower than the commit timestamp, it can be performed by skipping the entries in the timeline chain associated with the committing transaction, since the committing transaction is guaranteed to have a snapshot time at least as high as the commit timestamp. On the other hand, if the commit timestamp is lower than the snapshot time of the read operation, the read operation must wait until a commit time has been assigned to the transaction.

[0382] Optimization for read operations during preparation time

[0383] As mentioned above, read operations whose snapshot time is greater than the preparation time of a transaction that updated the items they must read typically have to wait until that transaction commits and a commit time is assigned. Once the transaction has committed and a commit time has been assigned, if the commit time is greater than the snapshot time of the read operation, the changes made by the transaction are skipped. On the other hand, if the commit time is less than the snapshot time of the read operation, the read operation sees the changes made by the transaction.

[0384] According to one embodiment, a technique is employed to prevent read operations that require seeing a specific version of a specific data item from waiting during the preparation phase of a transaction that has performed DML on that specific data item. Specifically, when (read snapshot time > TXN preparation time), the engine instance performing the read operation sends an increment clock message to the host executing the coordination engine instance. The increment clock message can be sent immediately or after a short delay.

[0385] The increment clock message includes the snapshot time of the read operation. If the logical clock at the coordinating host is not yet higher than the snapshot time in the increment clock message, the host receiving the increment clock message responds by incrementing its logical clock value to a value higher than the snapshot time included in the message. After receiving confirmation that the host of the transaction has incremented its logical clock, the read operation can continue under the assumption that the changes made by the transaction are not part of the snapshot of the read operation, and therefore can be safely skipped by the read operation. At this time, the preparation time at the replica can also be incremented to the snapshot time of the read operation (because since the clock of the coordinating engine instance has been maximized with the snapshot time of the read operation, it is guaranteed that the transaction will be assigned a commit time higher than the snapshot time).

[0386] For example, suppose TX1 updates a specific row in slice S2, TX1's status at secondary copy 938 is "Preparing," and the preparation time for TX1 at secondary copy 938 is T10. Further suppose engine instance 908 receives a request to read that specific row with a deadline of T20. In these cases, instead of waiting for TX1 to commit, engine instance 908 can send an increment clock message with a timestamp T20 (or higher) to host 950. In response, host 950 increments its local logical clock value to a time later than the timestamp included in the increment clock message. Incrementing host 950's clock in this way ensures that host 950 will assign TX1 a commit time higher than the snapshot time T20 of the read operation. Host 950 will send an acknowledgment clock message back to host 956. After receiving the acknowledgment of the increment clock message, the preparation time can be increased to T20, and the read operation can proceed, skipping the entry associated with TX1, because it is guaranteed that the changes made by TX1 are not in the snapshot associated with time T20.

[0387] According to one embodiment, the clock increment operation is accomplished using a series of Remote Direct Memory Access (RDMA) calls. RDMA calls can be made to read the relevant transaction table entries on the host on which the coordination engine instance is running. From the transaction entries, the coordinator of the read operation can obtain the global readiness time of the transaction. If the read operation snapshot is small, the change can be safely ignored because the commit time will only increase. If the read operation snapshot is larger than the global readiness time, an RDMA write operation can be used to change the global readiness time to the snapshot time of the read operation. Changing the global readiness time in this way ensures that the transaction coordination engine instance will eventually assign a commit time to the transaction that is higher than the snapshot time of the read operation.

[0388] Handling submission time read operation

[0389] As explained above, read operations arriving during the transaction's preparation phase can continue reading older versions of the data items touched by the transaction after ensuring the transaction will be assigned a commit time later than the snapshot time of the read operation. Unfortunately, similar optimizations cannot be performed on read operations when the problematic transaction is in a ready / committing state.

[0390] When a transaction is in the ready / committing state, the transaction coordinator has already sent a candidate commit time to the backup coordinator. Therefore, based on the snapshot time of the read operation, the logical clock of the host that adds the coordinating engine instance at this time may not have any impact on the commit time assigned to the transaction. Similarly, changing the prepare time of a transaction in the global transaction table entry will change the candidate commit time that the coordinating engine instance has already sent to the backup coordinator.

[0391] In one embodiment, when a transaction is in the "ready / committing" state, the coordinator of a read operation can still send an increment clock message to the coordination engine instance. However, the coordination engine instance will not adjust its clock or change the global readiness time, but will wait until it receives a confirmation commit time message from the backup coordinator. At this point, the coordination engine instance will not only change the transaction state to "committed," but will also respond to the shelved read operation by providing the commit time assigned to the transaction.

[0392] Because the commit time is sent directly to the coordinator of the read operation, the read operation is only suspended before the transaction is committed, not during the cleanup of the data log entries in question. By comparing the transaction's commit time with the snapshot time of the read operation, the coordinator of the read operation determines whether to obtain data from the time-series entry chain generated for the transaction or skip those entries.

[0393] In one embodiment, when a transaction is in the ready / commit phase and a read operation is suspended, the coordination engine instance can send a candidate commit time of the transaction to the coordinator of the suspended read operation. This candidate commit time can be sent without the coordination engine instance waiting for all backup coordinators to confirm the commit time. In these cases, if the candidate commit time is less than the snapshot time of the read operation, the read operation must continue to wait (because there is no guarantee that the transaction will commit). On the other hand, if the candidate commit time is greater than the snapshot time of the read operation, the read operation can continue under the assumption that it has not seen the changes made by the transaction. This is possible because if the transaction does commit, its commit time will be at least as high as the candidate commit time.

[0394] Optimistic preparation time

[0395] As explained above and Figure 10As shown, after receiving a readiness confirmation message from all participants in a transaction, the coordination engine instance selects a candidate commit time that is higher than (a) its current logical clock and (b) the highest readiness time received from the transaction participants. In the above embodiment, the readiness time sent by each transaction participant is the value of the logical clock at the participant's host when the participant has prepared the changes associated with the transaction.

[0396] For example, if transaction TX1 is paired with Figure 9 If slices S1 and S2 in system 900 execute DML, then all hosts in system 900 will become participants in TX1, since each host has a copy of slice S1 or slice S2. Because the logical clocks at these hosts are independent of each other, the preparation time for transaction TX1 can differ at each host.

[0397] As mentioned above, if their snapshot time is greater than the preparation time of TX1, then read operations targeting data items touched by TX1 during the preparation phase of TX1 must perform additional work (e.g., sending a clock increment message to the control engine instance) before reading the data items. However, if their snapshot time is less than the preparation time of TX1, no additional work is required.

[0398] According to one embodiment, to increase the likelihood that the snapshot time of a later received read operation will be less than the preparation time of the transaction being prepared, transaction participants can assign a preparation time higher than their current clock. For example, suppose engine instance 910 has received a preparation request for TX1. In response, engine instance 910 verifies that the secondary copy 940 of S2 has a log record of all changes made by TX1 to slice S2. Engine instance 910 then changes the state of its locally stored TX1 to "ready," stores the local ready time of TX1, and returns the local ready time in the preparation confirmation message. To assign an optimistic preparation time, engine instance 910 chooses a preparation time that is significantly higher than the current value of the logical clock of host 960.

[0399] For example, if the current value of the logical clock of host 960 is T1000, engine instance 910 can choose a preparation time of T10000. By selecting an optimistic preparation time in this way, engine instance 910 increases the likelihood that any read operation targeting a data item touched by TX1 received during the preparation state of TX1 will have a snapshot time less than the local preparation time of TX1. Because the snapshot time of those read operations will be less than the local preparation time of TX1, the read operations can proceed (looking at the version of the data item prior to TX1) without performing any additional work to ensure that TX1 will be assigned a commit time greater than the snapshot time of the read operation.

[0400] Logical rollback

[0401] In various situations, it may be necessary to "roll back" or "undo" changes made to a slice. According to the implementation, instead of creating a new version of the data with the changes removed, the system only stores data indicating which log records will be considered "undoed".

[0402] For example, suppose statement ST3 makes changes to multiple slices whose primary replicas are distributed across multiple hosts. During the execution of the statement, log entries for statement ST3 are generated by each of those hosts. Each of those log entries is marked with the identifier of statement ST3. If the coordinating engine instance fails before the execution of statement ST3 completes, the transaction fails over to another engine instance, which becomes the new coordinating engine instance for the transaction. To undo the changes made to statement ST3, the new coordinating engine instance generates a "rollback log entry." The rollback log entry indicates that all log entries marked with the transaction's statement number ST3 will be considered "undone."

[0403] After storing the rollback log entries, the new orchestration engine instance re-delivers the statements for execution. However, instead of reusing the same statement numbers, the new orchestration engine assigns new statement numbers (e.g., ST4) to the statements. Because new statement numbers are used for the re-executed statements, it is easy to distinguish the log entries from those generated when the statement execution was aborted from those generated when the statement was re-executed.

[0404] When an incremental log entry is applied to the line heap, incremental log entries in "undone" log records are skipped. Therefore, applying such a log record involves removing the incremental log entry from its corresponding sequence of entries without creating any new heap line entries. After an undone incremental log entry has been removed from its sequence of entries, the space allocated to the undone log record in the incremental log can be deallocated / reused.

[0405] Recover from lost data log records

[0406] Many different types of failures can occur during transaction execution. One such error is that the secondary replica fails to receive data log entries reflecting changes made to the primary replica. When this error is detected depends on a variety of factors, such as whether the host containing the primary replica has failed.

[0407] Refer again Figure 9Suppose that engine instance 912 generates three data log records (DL1, DL2, and DL3) during the execution of a DML fragment associated with statement ST2 of transaction TX1. Further assume that hosts 960 and 956 of secondary replicas 940 and 938 of S2 do not receive data log record DL2.

[0408] If hosts 960 and 956 receive subsequent data log entries (e.g., DL3) but not DL2, they can determine that a data log entry is missing. Under these conditions, hosts 960 and 956 can request the missing log entry (DL2) from host 958.

[0409] In some cases, the failure may not be discovered until later in the transaction execution. For example, suppose hosts 956 and 960 receive log records DL1 and DL2, but do not receive log record DL3 sent semi-synchronously from host 958. Further suppose host 958 acknowledges the completion of the statement fragment to host 950, and then crashes. The fragment completion acknowledgment message sent from host 958 to host 950 includes the last-LogRecID of the changes made by engine instance 912 to the primary replica 932 of slice S2. Therefore, after the crash, coordination engine instance 902 will have the LogRecID of log record DL3 generated by engine instance 912 prior to the crash.

[0410] Unfortunately, with the crash of master 958, the actual log record DL3 will be lost. After the crash, secondary replica 938 can be designated as the new primary replica of S2. In these cases, changes associated with log record DL3 will not be reflected in the new primary replica 938 of S2. During the preparation phase of the commit protocol for transaction TX1, control engine instance 902 will send a prepare message to all participants in transaction TX1. In this example, control engine instance 902 will ask master 956 whether it is ready for all changes to slice S2 up to the changes reflected in data log record DL3. Because engine instance 908 only has data log records up to DL2, engine instance 908 will report an error during the preparation phase of TX1. In these cases, the entire transaction TX1 may need to be rolled back and re-executed.

[0411] Engine Cluster

[0412] The term "engine cluster" is used in this article to collectively refer to a group of entities that work together as a database command service from a client. For example, in Figure 2In the system 200 shown, the engine cluster includes engine instances 200B-208B. The members of the engine cluster can change dynamically. For example, if engine instance 202B fails, it will no longer be a member of the engine cluster. Conversely, to increase the capacity of the engine cluster, new hosts and engine instances can be added.

[0413] When the membership of an engine cluster changes, replica management responsibilities need to be reassigned. For example, because replicas managed by engine instances on a failed host are no longer accessible, secondary replicas on different hosts will be promoted to primary status for each primary replica on the failed host. As mentioned above, this promotion of secondary replicas can be performed as part of a transaction failover operation. Similarly, when a new host is added to the engine cluster, the engine instances on those new hosts need to be assigned to host replicas so that some of the system's workload can be distributed to the new host.

[0414] Control Cluster

[0415] According to an embodiment, in addition to the engine cluster, the distributed database system also includes a control cluster. The control cluster comprises a set of control instances. The number of control instances in the control cluster is typically odd, but an even number can also be used. (See reference...) Figure 12 The diagram illustrates a distributed database system comprising six hosts: 1200, 1210, 1220, 1230, 1240, and 1250. Two engine clusters and one control cluster run on those hosts. Specifically, the control cluster includes control instances 1202, 1222, 1232, and 1242, which run on hosts 1200, 1210, 1220, 1230, and 1240, respectively. Engine instances 1204, 1214, 1224, 1234, and 1254, which run on hosts 1200, 1210, 1220, 1230, and 1250, respectively, form an engine instance cluster (EC1). Engine instances 1206, 1216, 1236, 1246, and 1256, which run on hosts 1200, 1210, 1230, 1240, and 1250, respectively, form another engine instance cluster (EC2).

[0416] The control cluster is responsible for tracking the current state and membership of each engine cluster. Specifically, the control cluster tracks which hosts are currently operating as part of the distributed database system, and the neighbor monitoring relationships between hosts. Neighbor monitoring will be described in more detail below.

[0417] According to one embodiment, the control instance operates as a High Availability Metadata Infrastructure (HAMI) community. In one embodiment, HAMI provides a tiered key-value store across a sufficient number of machines using only local storage to achieve high availability. A write-enabled community typically has 3-9 voting members. Non-voting observers (who can handle client write requests) may also be present for additional read scaling. In one embodiment, HAMI improves scalability by directing read operations to replicas and non-voting observers. A HAMI community is a collection of individual instances of the HAMI engine, referred to as members. A community is configured with a certain number of members, and in simple cases, a quorum of those configured members must be operational to provide service. In some embodiments, a configured number of shared (non-local) storage locations may also exist. When shared storage is present, the community can reach a quorum of configured members, or, if the number of members is less, a quorum of shared storage locations can be reached. This allows a configured member to provide service when sufficient shared storage is available, even if a majority of configured members are unavailable. Configured members have fixed network locations and ports. The configuration is stored in a replicated object repository visible to all members and requires nothing other than name resolution.

[0418] When tracking the status and membership of the engine cluster, the control cluster must be able to make decisions about engine cluster membership very quickly in response to changes affecting membership. Furthermore, the control cluster itself should be fault-tolerant so that a failure requiring changes to engine cluster membership does not simultaneously cause the control cluster to fail.

[0419] The control cluster's view of the host cluster is considered the "truth" for investigating faults. As will be described in detail below, investigation is necessary because it is unsafe to assume a host is faulty simply because another host reports a fault. For example, if host 1200 reports that host 1210 has failed, host 1210 may indeed have failed. However, host 1200 might also be experiencing network problems (and therefore unable to read host 1210's health counters), while host 1210 is functioning normally. The use of health counters to detect faults will be described in more detail below.

[0420] Health counter

[0421] refer to Figure 13 It illustrates in more detail Figure 12The host 1200. According to one embodiment, each host maintains a set of health counters 1330. The health counters 1330 are values ​​periodically updated by components within the host 1200 and stored in the host's volatile memory. When a counter stops being updated for a threshold period, it is likely that the component responsible for incrementing the counter has failed. Therefore, according to one embodiment, a health checker monitors the health of each component by periodically checking the counters associated with the components to ensure that the counters increment as expected.

[0422] Health checker and monitoring tree

[0423] According to one embodiment, various "health checkers" are used to monitor the health of various components of a distributed database system. A health checker is an entity whose function is to detect when a component fails. According to one embodiment, health checkers are arranged in a hierarchy, where higher-level health checkers monitor the health of one or more lower-level health checkers. The hierarchy of health checkers within a host forms a "monitoring tree," with the "root" of the monitoring tree being the health checker responsible for monitoring the health of the host itself.

[0424] exist Figure 13 In the illustrated embodiments, engine instances 1204 and 1206 include instance checkers 1312 and 1314, respectively. In one embodiment, the instance checker is the lowest-level health checker in the monitoring tree. Instance checkers monitor scheduler group-specific health counters, which are incremented by the scheduler groups executing within their respective engine instances. When a scheduler group-specific health counter indicates that the corresponding scheduler group is executing correctly, the instance checker increments its own engine-specific health counter.

[0425] In addition to instance checkers 1312 and 1314, host 1200 also includes host checker 1310 and neighbor checker 1340. Host checker 1310 monitors the health of host 1200 based on engine-specific health counters set by instance checkers 1312 and 1314. Besides monitoring engine-specific health counters, host checker 1312 can also obtain information from host manager 1350. Host manager can perform various checks related to the health of host 1200, including but not limited to:

[0426] • Check the health of the operating system running on host 1200 (e.g., check kernel memory fragmentation).

[0427] • Check the registers to detect any problems with the host controller adapter (e.g., one or more NICs connected to the host 1200).

[0428] • Check the health of any SSDs and NVRAM locally on host 1200.

[0429] • Check the health of host service components. When the host manager 1350 and the engine-specific health counters indicate that the engine instance is operating normally, the host checker 1310 can increment the host-specific health counters.

[0430] Neighbor checker 1340 monitors the health of neighboring hosts by periodically reading the values ​​of host-specific health counters on the neighboring hosts, as will be described in more detail below. In addition to monitoring the host-specific health counters of neighboring hosts, neighbor checker 1340 can also periodically read the health counters of neighbor checkers running on the neighboring hosts. Therefore, neighbor checker 1340 is able to detect when its neighboring hosts are not functioning properly and when its neighboring hosts are not correctly monitoring their respective neighboring hosts.

[0431] According to one embodiment, the neighbor checker 1340 determines whether the neighbor host is operating normally by reading a health counter from the neighbor host's memory using an RDMA operation. For example, the neighbor checker 1340 can use an RDMA operation to read the health counter in the volatile memory of host 1210 (see [link to RDMA operation]). Figure 12 To determine whether host 1210 is operating normally.

[0432] Finally, within control instance 1202 is the logic for using HAMI heartbeat to detect the health of other control instances. How to monitor the health of the control cluster will be described in more detail below.

[0433] Host monitoring ring

[0434] As mentioned above, the neighbor checker within each host uses RDMA to read host-specific health counters from neighboring hosts to determine if the neighboring hosts are operating correctly. According to one embodiment, the control cluster performs neighbor monitoring assignments to ensure that the health of each host is being monitored by another host. Such neighbor monitoring assignments can be performed such that neighbor monitoring relationships form a ring encompassing all hosts.

[0435] Refer again Figure 12 It illustrates a neighbor-monitoring relationship 1270 forming a ring including all hosts 1200-1230. In the illustrated embodiment, host 1200 monitors the health of host 1210. Host 1210 monitors the health of host 1220. Host 1220 monitors the health of host 1250. Host 1250 monitors the health of host 1240. Host 1240 monitors the health of host 1230. Host 1230 monitors the health of host 1200.

[0436] Network Topology

[0437] refer to Figure 16It illustrates the support Figure 12 The network topology shown illustrates communication between the entities. Figure 16 In this setup, each of the eight hosts is running a host monitor. The host monitor is part of a host monitoring ring, where each host monitors the health of its neighboring hosts. Additionally, some of these hosts are running control instances.

[0438] The network topology connecting the hosts allows each host to connect to every other host through two different networks. For example, the leftmost host can communicate with the rightmost host through a network containing switches 1, 2, and 3, or through a second network containing switches 4, 5, and 6.

[0439] Responding to engine instance failures

[0440] When an engine instance fails, the failure will be detected by a host checker running on the same host as the failed engine instance. For example, if engine instance 1204 fails, instance checker 1312 will detect the failure and stop incrementing the health counter of engine instance 1204. Host checker 1310 will see that the health counter of engine instance 1204 has not changed and will know that engine instance 1204 has failed.

[0441] After a host inspector detects that an engine instance on its host has died, it sends messages to the surviving engine instances in the engine cluster, notifying them that the sibling engine instance has died. For example, if there are 100 hosts and 10 engine instances in the engine cluster, and one engine instance has died, the host inspector will send 9 messages. Engine eviction messages instruct other engine instances to evict the failed engine instance from the engine instance cluster. According to one embodiment, RDMA (Random Access Path) is used to send engine eviction messages, and the host inspector sending the engine eviction message does not wait for the message to be acknowledged. Therefore, these end-to-end eviction messages are "unreliable".

[0442] In addition to sending engine eviction messages to surviving engine instances, the host checker running on the host of the failed engine instance also sends messages to the control cluster. However, unlike the engine eviction messages sent to surviving engine instances, after sending the engine eviction message to the control cluster, the host checker waits for acknowledgment from the control cluster that the message has been received.

[0443] Upon receiving an engine instance eviction message, the control cluster records the eviction in the control directory and broadcasts the eviction notification to all surviving members of the engine instance cluster. In contrast to end-to-end eviction messages, eviction messages from the control cluster (slow path) are reliable. Therefore, if all host connections are functioning correctly, each surviving engine instance will receive the eviction notification twice: once via RDMA from the host of the failed engine instance, and again via the eviction broadcast from the control cluster.

[0444] After being notified that an engine instance will be evicted, the surviving engine instances update their engine instance cluster configuration information to evict the specified engine instance, adjust slice hosting assignments to resolve the removal of the failed engine instance, etc.

[0445] Responding to host failure

[0446] When a host fails, the host assigned to monitor that failed host as a neighbor will detect the failure by noticing that the health counter on the failed host is not increasing. For example, suppose... Figure 12 The diagram shows the establishment of a host monitoring ring. If host 1210 fails, the neighbor checker on host 1200, which is currently assigned to monitor host 1210, will detect the failure.

[0447] Upon detecting a host failure, the host that detected the failure sends a message to the control cluster to report the failure. After reporting the failure to the control cluster, the host that detected the failure will begin neighbor monitoring of hosts that the failed host had previously monitored. Therefore, after reporting a failure in host 1210, host 1200 will begin neighbor monitoring of host 1220 (the host previously monitored by the failed host 1210).

[0448] Upon receiving notification of a failed host, the control cluster initiates an investigation. According to one embodiment, the investigation is conducted in a tiered manner.

[0449] refer to Figure 14 The diagram illustrates a distributed database system comprising four hosts: 1401, 1402, 1403, and 1404. For illustrative purposes, it will be assumed that the neighbor checker on host 1403 detects a problem with host 1404. The problem could be that the host health counter of host 1404 has stopped incrementing, or that host 1403 is unable to communicate with host 1404. Host 1403 attempts to read the health counter of host 1404 as shown in (1), indicating that it is in chronological order. Figure 14 The first action shown.

[0450] After detecting a problem on host 1404, the neighbor checker on host 1403 determines who is the current leader in control cluster 1450. In one embodiment, this can be achieved using the RDMA of any control instance. In the illustrated example, the RDMA of the follower control instance (CI-F) is shown as (2).

[0451] Once the leader of the control cluster is identified, the neighbor inspector of host 1403 sends a host eviction alert to the control instance leader (CI-L). The transmission of the host eviction alert is shown in (3).

[0452] In response to receiving a host eviction alert, control the instance leader:

[0453] • Determine the expulsion list (4)

[0454] • Send a suspicious notification to each surviving host (5)

[0455] • Record the eviction list in the control directory

[0456] Update the status information corresponding to the expelled host.

[0457] • Send an eviction notice to each surviving host (6).

[0458] Determine the eviction list

[0459] As referenced above Figure 14 As explained, when one or more hosts fail, the leader controlling the cluster determines which hosts will be evicted from the cluster. As a general rule, the leader attempts to maintain the largest set of hosts in the cluster that can communicate with each other. This set is referred to herein as the "maximum fully connected host set." The maximum fully connected host set is determined based on the connectivity between hosts, as will now be referenced. Figure 15 As described.

[0460] See Figure 15 This is a flowchart illustrating the determination of the maximum set of fully connected hosts according to one embodiment. In step 1502, it is determined whether the "local investigation" was successful. A local investigation refers to a troubleshooting step taken when an engine instance detects a problem with another engine instance it is communicating with. The final result of the local investigation is one of the following three options:

[0461] The problem was resolved (e.g., the suspended thread was restarted).

[0462] • The instance is evicted (as a proxy for eviction by the scheduling group, since we don't have the latter).

[0463] If connectivity is suspected to be the root cause, then the scope of the investigation should be expanded. In this case, we will switch to a focused investigation approach.

[0464] For example, a local investigation might involve attempting to restart one or more components. If the local investigation is successful, control is transferred to step 1550, where the problem is considered resolved and normal operation can continue. If the local investigation fails, control is transferred to step 1504.

[0465] At step 1504, a “centralized investigation” begins. This investigation determines which engine instances need to be evicted from the host cluster. According to one embodiment, engine instances that do not belong to the largest set of fully connected engine instances are evicted.

[0466] For illustrative purposes, it will be assumed that the centralized investigation is triggered by a situation where host A is unable to communicate with host B. In this scenario, at step 1508, host A determines whether it can reach the current leader of the control cluster. If host A cannot reach the current leader of the control cluster, host A knows it will be evicted. Therefore, at step 1552, host A evicts itself. After evictment, host A stops processing client data requests until recovery is performed to re-add host A to the host cluster.

[0467] If host A can reach the current leader of the control cluster, host A notifies the current leader that it cannot reach host B. At step 1510, the leader of the control cluster attempts to contact host B. If the leader cannot communicate with host B, host B is added to the eviction list at step 1554.

[0468] If the leader can reach host B, then at step 1512, the leader determines whether hosts A and B can access the same network. If A and B cannot access the same network, control moves to step 1518 and a tie-breaking heuristic is used to determine which of A and B will be expelled. At step 1556, the loser is added to the expulsion list.

[0469] If hosts A and B can access the same network, control is passed to step 1514, where one or more other members of the control cluster are used to probe the connectivity between hosts A and B. As a result of the probe performed by one or more other members of the control cluster, the control cluster determines connectivity scores for hosts A and B. According to one embodiment, the connectivity score of host A reflects how many members of the control cluster can communicate with host A, while the connectivity score of host B reflects how many members of the control cluster can communicate with host B.

[0470] At step 1516, it is determined whether the connectivity scores of hosts A and B are equal. If the connectivity scores are not equal, the host with the lower connectivity score is considered a "loser" and is added to the eviction list at step 1556. If the connectivity scores are equal, a tie-breaking trial-and-error method is used at step 1518 to determine the loser to be added to the eviction list in step 1556.

[0471] Steps 1552, 1554, and 1556 are followed by step 1558, in which the expulsion list is ready. Once ready, the expulsion list is submitted to the control directory in step 1560, and the expulsion agreement is initiated at 1562.

[0472] Handling eviction is a daunting task for surviving instances. Furthermore, eviction must be completed quickly. For example, survivors must reconfigure the slices affected by eviction. Survivors may need to build additional copies to compensate for the lost ones.

[0473] As mentioned above, centralized investigations are conducted in pairs. In the case of multiple failures, eviction based on pairwise investigations may lead to unnecessary reconfiguration work. Furthermore, in multi-failure scenarios, pairwise investigations can result in significantly suboptimal global configurations.

[0474] To avoid problems that can arise from pairwise investigations in multiple failure scenarios, a "broad investigation" technique is employed if an alert arrives at the control cluster during (or shortly after) a centralized investigation. Receiving such an alert during a centralized investigation can indicate the presence of multiple failure scenarios.

[0475] During the "extensive investigation," the control cluster acts as both investigator and arbitrator, and all hosts in the cluster are subject to the investigation. The extensive investigation examines the connections between the control cluster and the hosts, not the entire set of peer connections between hosts. Evicts resulting from the extensive investigation are processed in batches within a single eviction list.

[0476] In a broad survey, the network connectivity available to the control cluster leader is categorized. For example, suppose the control cluster leader is connected to two networks (Network 1 and Network 2). Hosts accessible to the control cluster leader through both Network 1 and Network 2 can be categorized as "fully connected hosts." Hosts indirectly accessible to the control cluster leader through a single network can be categorized as "edge hosts." Hosts that the control cluster leader cannot access directly or indirectly are categorized as "unreachable." After categorization, unreachable hosts are removed.

[0477] Edge hosts in one network typically cannot communicate with edge hosts in another network. In these situations, the control cluster leader decides which network's edge hosts to evict. For example, the control cluster leader might decide to retain edge hosts in network 1 while evicting edge hosts in network 2. According to one embodiment, when selecting which network's edge hosts to retain, the control cluster leader prefers to retain hosts that include the control instance, rather than hosts that do not include the control instance. If a broad investigation was initially triggered by an alert from one host regarding another host, the control cluster instance ensures that at least one of those two hosts is included in the recommended eviction list.

[0478] Respond to other types of failures

[0479] Host and engine instance failures are not the only types of failures that can occur in a distributed database system. For example, software components such as host services (e.g., host checker 1310) and instance services (e.g., engine instance 1204 or control instance 1202) can fail. When a software component fails, the system automatically attempts to restart the failed component.

[0480] If an engine instance dies (fails and cannot be restarted), the host inspector of the engine instance running on the same host as that engine instance detects the failure and reports it to the control cluster. For example, if engine instance 1204 dies, host inspector 1310 reports engine instance 1204 dead to the control cluster. Reporting a dead engine instance in this way is called "one-sided reporting" because it does not involve components on any host other than the host on which the failed engine instance was running. In response to being notified that an engine instance has failed, the control cluster reconfigures the engine cluster by evictring the failed engine instance.

[0481] Engine instance eviction

[0482] For various reasons, it may be necessary to evict an engine instance from the engine cluster. For example, as explained above, an engine instance may need to be evicted because the instance checker has detected that it has stopped performing useful work. Evicting an engine instance will also be necessary if the host on which it is running fails (which can be detected by neighbor monitoring). Finally, eviction will be necessary if the host on which the engine instance is running loses connectivity to other hosts in the host cluster (e.g., due to link or switch failure).

[0483] Evicting an engine instance involves reconfiguring the engine cluster by excluding the engine instance. Reconfiguring the engine cluster can involve:

[0484] • Determine the primary replica managed by the evicted engine instance.

[0485] Designate the sub-dungeons of those main dungeons as the new main dungeons.

[0486] • Create new secondary copies to replace those that have become primary copies.

[0487] • Update the slice mapping to engine instance to reflect the changes.

[0488] Control instance health monitoring

[0489] As described above, when a host fails can be detected by monitoring neighboring hosts in a host monitoring ring. In one embodiment, a separate health monitoring mechanism is used to detect failures of the control instance. Specifically, according to an embodiment, the control cluster uses the RAFT protocol to detect failures among its members. The RAFT protocol is described in detail in "In Search of an Understandable Consensus Algorithm" by Diego Ongaro and John Ousterhout of Stanford University, which can be found at www.usenix.org / conference / atc14 / technical-sessions / presentation / ongaro, the contents of which are incorporated herein by reference.

[0490] According to the RAFT protocol, at any given time, a control instance is designated as the "leader instance." However, the designation of a leader instance is temporary and changes over time. In one embodiment, the leadership period has a fixed duration. Once a leader's leadership period ends, leadership designation is automatically passed to another leader. According to an embodiment, the sequence of leadership designations forms a ring that includes all control instances in the control cluster. Therefore, each control instance has an equal "chance of being chosen" when it becomes the designated leader instance. In an alternative embodiment, leadership does not change automatically over time. More precisely, once elected as a leader, a control instance will remain in leadership until it fails, at which point a follower can be elected as the new leader. In one embodiment, when a new leader is chosen, each control instance has the same chance of being elected as the new leader.

[0491] In one embodiment, to detect control instance failures, the leader instance sends heartbeat messages to all other control instances, and all other control instances send heartbeat messages back to the leader instance. Failure to receive a heartbeat message within a threshold time period indicates that the control instance that did not receive the heartbeat is not operating correctly. Therefore, the leader instance is able to detect when any other control instance (“follower instance”) fails, and all follower instances are able to detect when the leader instance fails.

[0492] In an alternative implementation, the control cluster can use health counters and RDMA to detect failures in a manner similar to that of the host cluster. Specifically, instead of the leader instance sending heartbeat messages to all follower instances, the leader instance can increment its health counter, and follower instances can use RDMA to check the leader instance's health counter. Similarly, instead of follower instances sending heartbeat messages to the leader instance, follower instances can update their local health counters, and the leader instance can use RDMA to check if the corresponding health counters of the follower instances are increasing.

[0493] In one embodiment, each control instance has an instance checker that monitors the health of the control instance. The host checker reads the health counters updated by the control instance's checker. If the control instance checker detects a failure within the control instance, or if the control instance terminates, the health counters are not updated. This is detected by the host checker, which sends an alert to the remaining members of the control cluster indicating that one of the members of the control cluster has died. If the dead member is known to be the previous leader, the remaining members can initiate an election. This ensures a rapid initiation of leader election.

[0494] Handling and controlling instance failures

[0495] If the controlling instance misses a heartbeat from the leader instance, a HAMI election is initiated to determine a new leader. In contrast, HAMI followers continue to be considered part of the HAMI group even when they stop responding to requests from the HAMI leader.

[0496] When a HAMI leader receives an update request, the following events occur:

[0497] The leader attempts to persist the request to a quorum of members (including themselves).

[0498] Once the quorum is reached, the update request is committed, even if the followers do not respond.

[0499] The leader will continue to send log entries asynchronously to any unresponsive followers.

[0500] Asynchronous requests sent to unresponsive followers will not affect the latency of update requests, as long as the quorum is met.

[0501] Whenever a follower removed from the group is able to rejoin the group (after a reboot or network partition repair, etc.), the follower will begin receiving log entries (or snapshots of the log entries) to catch up. Additionally, control instances can be added to or removed from the control cluster via administrative operations.

[0502] Hardware Overview

[0503] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute the techniques, or may include digital electronic devices permanently programmed to execute the techniques, such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs), or may include one or more general-purpose hardware processors programmed to execute the techniques according to program instructions in firmware, memory, other storage devices, or combinations thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement the techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement the techniques.

[0504] For example, Figure 11 This is a block diagram illustrating a computer system 1100 on which embodiments of the present invention may be implemented. The computer system 1100 includes a bus 1102 or other communication mechanism for transmitting information and a hardware processor 1104 coupled to the bus 1102 for processing information. The hardware processor 1104 may be, for example, a general-purpose microprocessor.

[0505] Computer system 1100 also includes main memory 1106, such as random access memory (RAM) or other dynamic storage devices, coupled to bus 1102 for storing information and instructions to be executed by processor 1104. Main memory 1106 may also be used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 1104. When such instructions are stored in non-transitory storage media accessible to processor 1104, such instructions make computer system 1100 a dedicated machine customized to perform the operations specified in the instructions.

[0506] Computer system 1100 also includes a read-only memory (ROM) 1108 or other static storage device coupled to bus 1102 for storing static information and instructions of processor 1104. Storage device 1110, such as a disk or optical disk, is provided and coupled to bus 1102 for storing information and instructions.

[0507] Computer system 1100 can be coupled via bus 1102 to a display 1112, such as a cathode ray tube (CRT), for displaying information to a computer user. Input device 1114, including alphanumeric keys and other keys, is coupled to bus 1102 for transmitting information and command selections to processor 1104. Another type of user input device is cursor control 1116, such as a mouse, trackball, or arrow keys, for transmitting directional information and command selections to processor 1104 and for controlling cursor movement on display 1112. Such input devices typically have two degrees of freedom on two axes (a first axis (e.g., x) and a second axis (e.g., y)) to allow the device to specify a position in a plane.

[0508] Computer system 1100 may implement the techniques described herein using custom hard-wired logic, one or more ASICs or FPGAs, firmware, and / or program logic, which, in combination with the computer system, make computer system 1100 a special-purpose machine or program the computer system 1100 as a special-purpose machine. According to one embodiment, the techniques herein are executed by computer system 1100 in response to processor 1104 executing one or more sequences of one or more instructions contained in main memory 1106. These instructions may be read into main memory 1106 from another storage medium, such as storage device 1110. Execution of the sequence of instructions contained in main memory 1106 causes processor 1104 to perform the processing steps described herein. In alternative embodiments, hard-wired circuitry may be used instead of software instructions or in combination with software instructions.

[0509] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs or magnetic disks, or solid-state drives, such as storage device 1110. Volatile media include dynamic memory, such as main memory 1106. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or magnetic tape cassettes.

[0510] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including wires containing bus 1102. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0511] Various forms of media can involve carrying one or more sequences of one or more instructions to processor 1104 for execution. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 1100 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 1102. Bus 1102 carries the data to main memory 1106, from which processor 1104 retrieves and executes the instructions. The instructions received by main memory 1106 may optionally be stored on storage device 1110 before or after execution by processor 1104.

[0512] Computer system 1100 also includes a communication interface 1118 coupled to bus 1102. Communication interface 1118 provides bidirectional data communication coupled to network link 1120, which is connected to local network 1122. For example, communication interface 1118 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity to a corresponding type of telephone line. As another example, communication interface 1118 may be a LAN card providing data communication connectivity to a compatible local area network (LAN). A wireless link may also be implemented. In any such implementation, communication interface 1118 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0513] Network link 1120 typically provides data communication to other data devices via one or more networks. For example, network link 1120 may provide a connection to host computer 1124 or to data devices operated by Internet Service Provider (ISP) 1126 via local network 1122. ISP 1126, in turn, provides data communication services via a global packet data communication network now commonly referred to as the "Internet" 1128. Both local network 1122 and Internet 1128 use electrical, electromagnetic, or optical signals that carry digital data streams. Signals through various networks, as well as signals on network link 1120 and through communication interface 1118, are example forms of transmission media that carry digital data to or from computer system 1100.

[0514] Computer system 1100 can send messages and receive data, including program code, through one or more networks, network links 1120, and communication interfaces 1118. In the Internet example, server 1130 can transmit requested code for the application through the Internet 1128, ISP 1126, local network 1122, and communication interface 1118.

[0515] The received code can be executed by processor 1104 when it is received, and / or stored in storage device 1110 or other non-volatile storage device for later execution.

[0516] cloud computing

[0517] The term "cloud computing" is generally used in this article to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0518] Cloud computing environments (sometimes called cloud environments or the cloud itself) can be implemented in various ways to best meet different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the public. In contrast, private cloud environments are generally used only by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public clouds) that are bound together by data and application portability.

[0519] Generally, cloud computing models enable functions that were previously provided by an organization's own IT departments to be delivered as service layers within a cloud environment for consumer use (depending on the public / private nature of the cloud, or whether it is within or outside the organization). Depending on the specific implementation, the precise definition of the components or functions provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure, while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications, while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything below the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provision processing, storage, networking, and other basic computing resources, while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service in which consumers use database servers or database management systems running on cloud infrastructure, while DBaaS providers manage or control the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0520] In the foregoing description, embodiments of the invention have been described with reference to numerous specific details, which may vary between implementations. Therefore, the description and drawings should be considered illustrative rather than restrictive. The unique and exclusive indication of the scope of the invention, and the content that the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the set of claims published from this application in the specific form of such claims, including any subsequent corrections.

Claims

1. A method comprising: Multiple copies of a slice are stored on a set of hosts belonging to a distributed database system, wherein the slice has multiple assigned rows, and each of the multiple copies is a copy of the slice; The plurality of copies mentioned above includes at least: A primary copy residing on persistent storage on the local machine of the first host in the group of hosts; and A secondary copy residing on persistent storage on the local machine of the second host in the group of hosts; Execute a transaction that updates specific data items in the slice; Use a coordination engine instance to coordinate the execution of the transaction; During the execution of the transaction, the coordination engine instance causes: The first host stores the update in the primary replica; and The second host stores the update in a secondary copy; To bring the transaction into a ready state, the following steps are taken: The coordination engine instance stores data indicating that the transaction is in a ready state; and The coordination engine instance sends a preparation message to the second host; In response to the prepared message, the second host stores data indicating that the transaction is in a prepared state and determines the prepared timestamp for the update of the secondary replica; When the transaction is in the ready state, the second host receives a read request to read a specific data item up to a specific snapshot time; If the specific snapshot time is less than the update's preparation timestamp, the second host allows the read request to read the pre-update version of the specific data item that existed before the update to the secondary copy; if the specific snapshot time is greater than the preparation timestamp, then: The second host sends an increment clock message to the coordination engine instance, so that the first logical clock used by the coordination engine instance is set to an update value that is at least as high as the specific snapshot time; and After sending the increment clock message and before the transaction is committed, the second host allows read requests to read the previous version of a specific data item; and When the transaction is committed, a commit time at least as high as the updated value is assigned to the transaction.

2. The method of claim 1, wherein: The second host has a local logical clock; and Determining the preparation timestamp for the update includes selecting a preparation timestamp that is significantly higher than the current value of the local logical clock, in order to increase the likelihood that any read operations received by the second host during the preparation state of the transaction will be assigned a snapshot time less than the preparation timestamp.

3. The method as described in claim 1, wherein, If a specific snapshot time is greater than the preparation timestamp, the second host will only allow a read request to read the pre-update version of a specific data item after receiving confirmation that the increment clock message has been successfully processed.

4. The method of claim 1, wherein setting the first logical clock used by the coordination engine instance to an update value that is at least as high as a specific snapshot time comprises: Change the global preparation time of the transaction to the updated value; as well as When the transaction is committed, the first logical clock used by the coordination engine instance is set to a value that is at least as high as the global readiness time.

5. A method comprising: Store the first multiple copies of the first slice in the first group of hosts belonging to the distributed database system; The first plurality of copies include at least: The first primary copy residing on persistent storage on the local machine of the first host in the first group of hosts; and The first secondary copy resides on the persistent storage device of the second host in the first group of hosts; The second host is different from the first host; The second plurality of copies of the second slice are stored in a second group of hosts belonging to the distributed database system, wherein the first slice has a plurality of assigned rows, the second slice has a plurality of assigned rows, each of the first plurality of copies is a copy of the first slice, and each of the second plurality of copies is a copy of the second slice. The second plurality of copies includes at least: A second primary copy residing on persistent storage on the local machine of the third host in the second set of hosts; and A second secondary copy residing on persistent storage on the local machine of the fourth host in the second group of hosts; The fourth host is different from the third host; Execute a transaction that includes the statement: Perform a first update on the first version of the first data item in the first slice; and Perform a second update on the first version of the second data item in the second slice; Executing the transaction includes: The first host stores the first update in the first primary copy; The first host causes the second host to store the first update in the first secondary copy; The third host stores the second update in the second primary copy; and The third host causes the fourth host to store the second update in the second auxiliary copy.

6. The method of claim 5, wherein: The transaction is presented to the first host by the client application; The method also includes, before the transaction is presented to the first host, the client application determines whether the first slice or the second slice will become the controlling slice of the transaction; as well as After determining that the first slice will become the control slice and the first host hosts the first primary copy of the first slice, the client application presents the transaction to the first host.

7. The method of claim 6, wherein the first slice is determined as a control slice based on at least one of the following: The transaction performs more work on the first slice than on the second slice; The amount of local data in the first slice; The transaction first touches the first slice; or The first slice is qualified to be the most reliable slice relative to the slice touched by the transaction.

8. The method of claim 6, further comprising sending a DML fragment from a first host to a third host, the first host causing the third host to store a second update in a second primary copy, the DML fragment causing the third host to store the second update in the second primary copy when executed by the third host.

9. The method of claim 6, further comprising establishing the second host as a backup coordinator for the transaction based on the second host having an auxiliary copy of the slice selected as the control slice.

10. The method of claim 5, further comprising: The transaction is put into a ready state by storing transaction status data indicating that the transaction is in a ready state at a first host. The first host receives the readiness confirmation directly or indirectly from all hosts hosting the primary copy of the slice updated by the transaction and all hosts hosting the secondary copy of the slice updated by the transaction. In response to receiving a readiness confirmation directly or indirectly from all hosts hosting the primary replica of the slice updated by the transaction and all hosts hosting the secondary replica of the slice updated by the transaction, the transaction is brought into the committed state by changing the transaction state data at the first host to indicate that the transaction is in the committed state. as well as During the commit state, the first host sends a candidate commit time to the second host that is at least as high as the current value of the first logical clock at the first host.

11. The method of claim 10, wherein: Preparation confirmation includes preparing the timestamp; and The first host generates the candidate commit time in a manner that ensures the candidate commit time is at least as high as the highest preparation timestamp received in the preparation confirmation.

12. The method of claim 10, further comprising the first host submitting the transaction only after receiving confirmation from the second host that the candidate submission time has been received.

13. The method of claim 10, further comprising: The fourth host receives a read request to read a second data item up to the snapshot time, the snapshot time being greater than the preparation time associated with the second update of the second secondary copy; In response to the read request to read the second data item, the fourth host sends an increment clock message to the first host to request the first host to increment the first logical clock on the first host to a value that is at least as high as the snapshot time; as well as In response to the transaction being in a commit state, the first host delays acknowledgment of the increment clock request message from the fourth host until the first host receives confirmation that all coordinator backup hosts have received the candidate commit time. as well as After the first host receives confirmation that all coordinator backup hosts have received the candidate commit time, the first host establishes a commit time for the transaction and responds to the increment clock message by sending a commit time to the fourth host.

14. The method of claim 13, further comprising: Before the fourth host sends the clock increment message, the fourth host waits for a predetermined amount of time to check whether the transaction information on the fourth host's local machine has changed to indicate that the transaction has been committed; as well as The fourth host sends the clock increment message only after waiting for the predetermined amount of time.

15. The method of claim 10, further comprising submitting the transaction by the first host in response to receiving confirmation from all coordinator backup hosts that the candidate commit time has been received.

16. The method of claim 10, further comprising, after committing the transaction, the first host sending a committed log record of the transaction to all backup coordinators.

17. The method of claim 10, further comprising: The fourth host receives a read request to read a second data item up to the snapshot time, the snapshot time being greater than the preparation time associated with the second update to the second secondary copy; In response to the read request to read the second data item, the fourth host sends an increment clock message to the first host to request the first host to increment the first logical clock on the first host to a value that is at least as high as the snapshot time; as well as In response to the transaction being in a committed state, the first host determines whether the snapshot time is less than the candidate commit time; as well as In response to the snapshot time being less than the candidate submission time, the first host sends a specific message to the fourth host; as well as In response to the specific message, the fourth host responds to the read request by providing a first version of the second data item, wherein the first version does not include the second update.

18. The method of claim 5, further comprising: After the second update is stored in the second secondary copy, the fourth host generates the preparation time for the update; Specifically, when the local clock at the fourth host is at a specific time value, the generation preparation time is executed; The preparation time for the update generation involves a preparation time that is significantly longer than the specific time value; and The fourth host will assign the preparation time to the second update.

19. The method of claim 18, further comprising: When the transaction is in the committed state, the fourth host receives a read request to read the second data item up to a snapshot time greater than a specific time value and less than the preparation time, and In response to the snapshot time being less than the preparation time, the fourth host responds to the read request with a version of the second data item that does not include the second update.

20. A non-transitory computer-readable medium storing one or more instructions, which, when executed by one or more computing devices, cause the method described in any one of claims 1-19 to be performed.

21. A system comprising: Multiple computing devices, each having one or more processors; The plurality of computing devices are capable of accessing a non-transitory computer-readable medium containing one or more instructions, which, when executed by the plurality of computing devices, cause the method described in any one of claims 1-19 to be performed.

Images