Rdma-enabled key-value store

Abstract

According to one or more embodiments, lookup, insert, and delete operations are allowed to continue during the actions required to remedy a conflict. When a relocation operation is used to resolve a conflict, information encoded in the header portion of a hash table entry storing a key-value pair indicates when the associated key-value pair was relocated. This information facilitates continued access to the RKVS during the relocation process by allowing other processes accessing the RKVS to handle relocations without failure. Furthermore, when a hash table expansion is needed in order to resolve a conflict, a second, larger hash table is allocated and lookup operations continue to be performed on both the old hash table and the new hash table. One or more embodiments also use flags to prevent insert, lookup, and delete failures in the RKVS, which are encoded in the header information of a hash table entry that reflects the state of the corresponding key-value pair in the store.

Description

TECHNICAL FIELD

[0001] The present invention relates to the management of key-value stores, and more specifically to improving the availability of a remote direct memory access (RDMA)-enabled key-value store during conflict resolution, and reducing the failure of operations within the store concurrently performed by multiple processes. BACKGROUND

[0002] Key-value storage is essential in many applications, such as in the design of scalable industry data management. The performance of key-value stores plays an important role in enabling high-speed data processing. Innumerable techniques have been developed to improve the efficiency of delivering to client applications data requested by the client applications from a key-value store. Such techniques typically involve a database server instance executing on a server device that receives requests, gathers the requested data from the key-value store, and provides the data to the database application. Numerous client applications will frequently send requests to the same database server instance. The larger the workload in the database server instance, the less likely it is that the database server instance will be able to efficiently respond to all requests.

[0003] Unfortunately, if the database server instance executing on the server device fails, the data in the volatile memory of the server device is typically rendered unavailable to the client applications. In these situations, the client applications either wait for the database server instance to be restored, or, if possible, obtain the data they need from other sources (e.g., by sending requests to another database server instance that can access another copy of the desired data).

[0004] Alternatively, the client applications can use remote direct memory access (RDMA) requests to retrieve data managed by the database server instance from the volatile memory of the server machine on which the database server instance is executing. This technique allows the client to retrieve data from the volatile memory of the host server machine without involving the database server instance in the retrieval operation.

[0005] RDMA requests initiated by the client machine or client are unilateral operations and do not require CPU interrupts or OS kernel involvement on the host server machine (RDBMS server). That is, RDMA reads and writes are performed without the involvement of any remote database server instance. For example, InfiniBand is a widely used switching architecture network in high-performance computing systems. RDMA over Converged Ethernet (RoCE) is a network protocol that allows direct memory access over Ethernet. InfiniBand and RoCE network interface cards are designed to achieve low latency by providing RDMA and kernel bypass in addition to implementing several layers of network stack in hardware.

[0006] RDMA enables zero-copy transfers and significantly reduces latency and CPU overhead. It allows a process to directly access the memory of a remote device without involving any processes running on the device. Therefore, because data retrieval does not involve the database server instance, retrieval can succeed even if the database server instance (or the host server machine itself) has stopped or become unresponsive.

[0007] Compared to traditional messaging methods, RDMA achieves the lowest round-trip latency, highest throughput, and zero CPU overhead. Research has been conducted to explore RDMA implementations to improve the performance of in-memory data repositories. To this end, various methods have been proposed for maintaining key-value repositories with high throughput and low CPU overhead utilizing RDMA. However, most designs do not support multiple processes performing RDMA read and write operations in parallel on the key-value repository. Generally, such systems use single-sided RDMA reads to access data and leave write operations to the server instance managing the data, which prevents proper scaling as more and more processes access the data.

[0008] FaRM is a general-purpose distributed computing platform that exposes the memory of a cluster of machines to a shared address space. FaRM uses a variant of the Hopscotch hash to create a location-aware hash table for its key-value store design. For a GET operation, the client reads several consecutive Hopscotch entries, one of which contains the expected key with a high probability. For a PUT operation, the client writes its request to a circular buffer in volatile memory accessible to the server instance. The server instance polls this buffer to detect new write requests. Therefore, FaRM's PUT operation design prevents multiple processes from performing RDMA write operations directly in the key-value store.

[0009] Pilaf is another key-value store design aimed at achieving high performance and low CPU utilization. For GET operations, the client uses RDMA read operations to access a cuckoo-based hash table stored in memory accessible to the server instance. However, for PUT operations, the client uses a SEND message to send its write request to the server instance. Like FaRM, Pilaf provides multiple PUT operations by sending write requests to the server instance. Pilaf does not support multiple processes performing RDMA write operations directly in the key-value store.

[0010] HERD is a key-value system designed to optimize the use of RDMA networks. It employs a hybrid approach, utilizing both RDMA requests and messaging. In HERD, clients send their requests to the server instance via an unreliable connection (UC) using RDMA write requests. These write operations place PUT and / or GET requests into each client's memory area accessible to the server instance. The server instance polls these memory areas for new requests. Upon detecting a new request, the server instance executes the request in a regular manner using its local data structures. The server instance then sends a reply to the client using a SEND message over unreliable datagrams. Like FaRM and Piaf, HERD also provides multiple PUT operations via a process of sending requests to the server instance. However, HERD does not support multiple procedures directly performing write operations to the key-value store.

[0011] Nessie is a key-value system that includes a hash table design that supports multiple procedures performing RDMA read and write operations directly within the key-value repository. In Nessie, an indexed hash table maps each key to one or more locations in an auxiliary data table where the corresponding key-value pair is stored. The key is mapped to an entry in the indexed hash table based on one or more hash functions. The entries in the data table also include a validity flag indicating whether the corresponding entry belongs to an ongoing PUT operation.

[0012] Each client is given ownership of a set of data items in the data table, and a client can only update the data items it owns. The high-level structure for each operation on the data table includes: reading the primary and secondary indexes in the index table for the specified key to identify the location of the key-value pair in the data table; performing the operation on the appropriate item at the identified location in the data table; checking that the location listed for the key-value pair has not been modified by another client, i.e., modified via entries in the index table; and setting a valid bit on any new data item in the data table. However, Nessie's design has several weaknesses, including potential insertion failures, power outages during conflict resolution, and lookup failures.

[0013] The Nessie system may experience power outages during conflict resolution. Specifically, if key-value pairs (Key...) x VAL x A write operation fails because both the primary and secondary entries in the index hash table are filled with position data for different key-value pairs. Therefore, a relocation operation is performed to clear the entries in the index hash table to allow the write operation to complete. The first key-value pair that can be relocated is the one used for the key. x The data in the primary entry. If the relocation operation fails because the alternative entry used for the relocation candidate is populated, a second relocation operation is performed to relocate the data item in the secondary entry. Failure to clear the secondary entry causes a message to be sent to the server instance, which then revokes access to its local storage and performs a series of migrations or table resizing before allowing the client to continue. In this design, the server instance prevents the client from performing any read or write operations on the key-value store until conflict resolution is complete, which could lead to unnecessary power outages during which the key-value store is unavailable to the client.

[0014] The Nessie system also has the possibility of insertion failure. For example, in the Nessie system, for an operation where a client writes a value to a key-value store, the client first writes the data to a free entry in a secondary data table owned by the client. This data entry includes a flag marking the data as invalid. Once the position of the entry in the secondary data table is recorded in the index table, the write operation is complete and the flag for the key-value pair used for insertion is updated to indicate that the data is now valid. However, insertions in the Nessie system can fail, such as when two processes A and B attempt to simultaneously insert the same key-value pair (Key... x VAL x Insert a key-value store and the third process C attempts to delete it after it is inserted (Key). x VAL x Only when either process A or B should successfully insert the key-value pair should the inserted key-value pair be deleted by process C, thus causing it to be removed from the key-value repository. x VAL x However, if one of the processes for inserting key-value pairs is undesirably delayed until after process C deletes the key-value pairs from the repository, the insertion may fail, resulting in (Key...) x VAL x The data remains in the key-value store after it should have been deleted by process C.

[0015] The Nessie system also has the possibility of lookup failures. In this design, to perform a read operation on a specific key, the client first reads the position data in the main entry of the key stored in the indexed hash table. Because the Cuckoo Hash uses multiple potential indexes for each key, the position data mapped to the entry for the requested key might be for a different key. Therefore, when reading an entry in the indexed hash table, the client verifies whether the stored key associated with the position data matches the requested key. If the key matches, the corresponding value is returned to the client from the specified position in the data table. Otherwise, secondary entries in the indexed hash table are checked. Moreover, if the main entry is empty, the secondary entries for the key are checked. If the secondary entry is not empty and is associated with the requested key, the client continues reading the value stored at the specified position in the secondary data table.

[0016] Lookups in this design may fail because the latency between reading the location information in the index hash table and reading the data at that location within the data table is not accounted for. Specifically, there may be a latency between the client reading the location of a given entry and reading the data stored at that indicated location. During this latency, the data stored at the indicated location within the auxiliary data table may be modified by another process or moved to another location within the auxiliary data table. In this case, the client may return a value not associated with the requested key.

[0017] Therefore, it would be beneficial to support the possibility of multiple processes performing RDMA read and write operations directly in the key-value store without lookup failures, power-down failures, and / or insert failures.

[0018] The methods described in this section are possible methods, but not necessarily methods that have been previously conceived or adopted. Therefore, unless otherwise indicated, any method described in this section should not be assumed to be prior art simply because it is included in this section. Furthermore, any method described in this section should not be assumed to be easily understood, routine, or conventional simply because it is included in this section. Attached Figure Description

[0019] In the attached diagram:

[0020] Figure 1 This is a block diagram of a system configured to enable one or more clients to remotely access a key-value store managed by a database server instance, which is located in the volatile memory of the server machine executing the database server instance.

[0021] Figure 2 A flowchart is depicted for retrieving header information from hash table entries during the first sub-operation of a key-value pair operation and terminating the second sub-operation of the key-value pair operation based on one or more set flags in the header information.

[0022] Figure 3 The example hash table entry configuration and example hash table content are described.

[0023] Figures 4-7 The corresponding time series of snapshots of hash tables that implement key-value repositories are depicted.

[0024] Figure 8 A second hash table for implementing a key-value repository is described for use in key-value migrations related to conflict resolution.

[0025] Figure 9 A flowchart is depicted for migrating at least a portion of a key-value store from a first hash table in volatile memory to a second hash table in volatile memory.

[0026] Figure 10 It is a block diagram of a computer system on which embodiments can be implemented.

[0027] Figure 11 A software system that can be used in the embodiments is described. Detailed Implementation

[0028] In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of the invention. However, it will be apparent that the invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form to avoid unnecessarily obscuring the invention.

[0029] General Overview

[0030] According to one or more embodiments, remote direct memory access (RDMA) requests and local requests, potentially originating from multiple processes, can be used on an RDMA-enabled key-value store (RKVS). Therefore, RKVS provides lock-free multi-process operation for local and remote read and write operations without client involvement in managing the RKVS processes. Accordingly, embodiments allow fully parallel hash table insert, lookup, and delete operations.

[0031] The embodiments eliminate the need to restrict access to RKVS during conflict resolution. Specifically, according to one or more embodiments, lookup, insert, and delete operations are allowed to continue during actions required for conflict remediation. Conflict remediation actions include relocating key-value pairs to make room for inserts and / or expanding the size of the hash table storing RKVS. The increased availability of RKVS during conflict remediation allows databases requiring high availability and performance to continue operating without power outages that negatively impact the performance of the key-value store.

[0032] When a relocation operation is used to resolve conflicts, information encoded in the header of the hash table entry storing the key-value pairs indicates when the associated key-value pairs are being relocated. This information facilitates continued access to RKVS during the relocation process by allowing other processes accessing RKVS to handle relocations without failure. Furthermore, when a hash table needs to be expanded to resolve conflicts, a second, larger hash table is allocated, and lookup operations continue on both the old and new hash tables. When key-value pairs are migrated from the old hash table to the new hash table, the header information is used to handle relocation operations between the two tables, as described above. Any insert operations in RKVS are performed on the new hash table, and delete operations are performed on both the old and new hash tables as needed. Once the data migration is complete, the old hash table is released, and operations continue using the new hash table as before.

[0033] One or more embodiments also use flags encoded in the header information of hash table entries to prevent insertion and lookup failures in RKVS; these flags reflect the state of the corresponding key-value pairs in the repository. Specifically, the header information in a given hash table entry includes one or more of four single-bit flags. Such flags provide information about one or more of the following:

[0034] (a) Whether the associated entry is populated with key-value data, which allows the process to determine whether the entry is occupied without accessing any separate data structure other than the RKVS hash table; (b) Whether the data in the associated entry is being written to a particular entry, which helps determine whether to delay an operation on a particular entry until the data is written—such information allows the process to avoid failures caused by another process writing to one of the entries involved in the operation while the operation continues; (c) Whether the data in the associated entry is being relocated, which allows RKVS to continue operating during conflict remedies; and / or (d) Whether another process has priority to write a particular key-value pair that matches the key information stored in the associated entry, which helps prevent double insertion of key-value pairs.

[0035] According to an embodiment, each operation on RKVS is a multi-part process comprising a sub-operation that reads header information from a target entry in RKVS. The process initiating such an operation can utilize the retrieved header information to abort additional portions of the sub-operations of the operation, thereby avoiding potential failure conditions. According to one or more embodiments described below, the process follows the aborted operation with one or more further operations, at least in part based on the retrieved header information.

[0036] System and RDMA Overview

[0037] Figure 1This is a block diagram of system 100, which is configured to enable one or more clients to remotely access a key-value store managed by a database server instance, located in the volatile memory of the server machine executing the database server instance. Figure 1 In this configuration, machine 120 is configured with multiple clients 122-126, which may be implemented differently as one or more database applications, one or more database server instances, or any other type of process configured to send RDMA requests. Additionally, server machine 102 is configured with volatile memory 106 in which database server instance 108 runs. The cache 110 of volatile memory 106 includes an RDMA-enabled key-value store (RKVS) 112 managed by database server instance 108.

[0038] Database applications, other database server instances, and other types of processes typically retrieve key-value data and / or write key-value data to a remote key-value repository (such as RKVS 112) by sending database commands to database server instance 108. However, by using the techniques described herein, some or all of the key-value operations required by clients 122-126 are performed via RDMA-based access to the volatile memory 106 of server machine 102. Figure 1 As shown, RKVS 112 is implemented by at least hash table 114. Figure 1 As shown, key-value pairs can be written to or read from RKVS 112 by any of clients 122-126 via RDMA requests, to read from or write to hash table 114. Any such RDMA request can be received in parallel with other RDMA requests from other clients.

[0039] Server machine 102 and machine 120 are communicatively coupled to network 130. Network 130 can be implemented using any network protocol or architecture that facilitates RDMA requests.

[0040] In addition to transmitting RDMA access information to one or more clients among clients 122-126, the preliminary steps for enabling RDMA access may also include database server instance 108 providing certain information to network interface circuit system (NIC) 140 through which server machine 102 is coupled to network 130. NIC 140 can be implemented as any type of network interface circuit system, including as a chip on a server blade associated with server machine 102, as a network interface card of server machine 102, etc. Specifically, according to an embodiment, database server instance 108 registers with NIC 140 which areas of volatile memory 106 are suitable for RDMA access. According to one embodiment, in response to RDMA registration, NIC 140 provides database server instance 108 with one or more authentication keys. According to an embodiment, these keys are ultimately provided to clients so that clients can prove their authorization when performing RDMA access to volatile memory 106.

[0041] The RDMA technology described herein assumes an underlying network architecture that allows data transfer between one or more clients 122-126 and server machine 102, wherein server machine 102's volatile memory stores cached data. This underlying network architecture could be, for example, an InfiniBand interconnect. According to one embodiment, RDMA requests to RKVS 112 are serviced by NIC 140, through which server machine 102 is coupled to network 130 using a transport protocol implemented in NIC 140. According to another embodiment, RDMA requests to RKVS 112 are serviced by the management layer of network 130. However, the technology described herein is not limited to any particular underlying network architecture.

[0042] Hash table collision avoidance scheme

[0043] According to an embodiment, hash table 114 employs open addressing, which avoids memory allocations or pre-allocations at different addresses that would occur with linking, resulting in more shared cache lines. According to an embodiment, hash table 114 also implements one or more schemes to avoid hash mapping collisions. An embodiment is described herein as implementing a cuckoo hash collision avoidance scheme, in which multiple hash functions are used sequentially to identify entries in hash table 114 for a given key-value pair. With this scheme, the hash table has an 80% load factor when using two hash functions, and a 91% load factor when using three hash functions.

[0044] To illustrate the cuckoo hash scheme, System 100 ( Figure 1Client 122 applies the first hash function in the sequence of functions to the key of a specific key-value pair. The application of the first hash function identifies the main entry P in hash table 114 for that specific key-value pair. If entry P does not store the specific key-value pair (in the case of a read request) or is populated with another key-value pair (in the case of a write request), then client 122 applies the second hash function of the sequence to the key of the specific key-value pair, which identifies the auxiliary entry S in hash table 114 for that key-value pair. In this way, according to the Cuckoo Hash scheme, the client traverses the sequence of hash functions to identify multiple entries in the hash table for a single key-value pair.

[0045] However, given that different conflict avoidance schemes are more suitable for different applications that can use RKVS 112, the embodiments are not limited to the cuckoo hash conflict avoidance scheme. Specifically, according to one or more embodiments, any of a plurality of thread-safe hash table conflict avoidance schemes can be applied to hash table 114, including cuckoo hash, hopscotch, red-black tree, split ordered list, etc. Therefore, the embodiments provide multiple options for hash table conflict management.

[0046] Head marker

[0047] According to one or more embodiments, entries in hash table 114 include one or more header portions that serve as markers for processes (such as clients 122-126) that request information from the hash table. Figure 2 A flowchart 200 is depicted for retrieving header information from a hash table entry during a first sub-operation of a key-value pair operation and aborting a second sub-operation of the key-value pair operation based on one or more flags in the header information being set. Specifically, at step 202 of flowchart 200, the first process performs the first sub-operation of a specific key-value pair operation by executing steps 202A and 202B of flowchart 200. At step 202A, a Remote Direct Memory Access (RDMA) request is sent to obtain header information from a hash table in the volatile memory of the computing device for a specific entry of a specific key-value pair, wherein the header information includes a set of flags including one or more of the following: an Allocd flag indicating whether the specific entry is occupied, an Influx flag indicating whether an incomplete write operation to the specific entry is in progress, a Time flag indicating whether data in the specific entry is being migrated, and a Fence flag indicating whether the second process, other than the first process, has priority in inserting the specific key-value pair into the hash table.

[0048] For example, client 122 sends an RDMA request to NIC 140 via network 130 to retrieve the key-value pair (K) from hash table 114. x Vx The client 122 applies a hash function to K to obtain the information. x To identify hash table entries used for key-value pairs, which returns a mapping to (K x V x The identifier of the entry. According to an example embodiment where hash table 114 is managed using a cuckoo collision avoidance scheme employing sequence hash functions H1, H2, and H3, client 122 in K x Then run H1 to identify the first candidate entry in hash table 114 for a specific key-value pair.

[0049] Figure 3 An example hash table entry configuration 300 is depicted, containing 6 bytes of key information 320 representing the key of a key-value pair and 8 bytes of value information 330 representing the value of the key-value pair. The entry configuration 300 also includes 16 bits of header information 310. Figure 3 In the embodiments depicted herein, the information includes four 1-bit flags, a five-bit node identifier 322, and a seven-bit reserved header space 324.

[0050] The four 1-bit flags include an assigned flag 312, an injection flag 314, a time flag 316, and a fence flag 318. The utility of each flag is summarized below based on one or more embodiments (further details are provided below):

[0051] • When new data is to be inserted into the associated hash table entry, the assigned flag 312 is set. This flag serves to prevent multiple processes from simultaneously attempting to insert data into the entry. The associated assigned flag 312 remains set when the entry is used to store key-value pairs.

[0052] • When data insertion in an associated hash table entry is in progress but not yet complete, the injection flag 314 is set. The effect of this flag is to stop the lookup at this entry until the data is fully inserted. After the data insertion is complete, the injection flag 314 is deset, i.e., deset by the data insertion process. Using this flag helps avoid potential lookup failures.

[0053] • During conflict resolution, a timestamp 316 is set when the movement of data from the associated hash table entry to another location begins. This timestamp serves to notify clients requesting data at the entry whether the key-value pair is in the process of being relocated. After the relocation operation is complete, the timestamp 316 is de-set by the process that performed the relocation (such as database server instance 108). Using this timestamp allows the process to continue operations in RKVS 112 during conflict resolution.

[0054] • When a process inserting a specific key-value pair into another entry in hash table 114 determines that a different process is also inserting the same specific key-value pair into an associated hash table entry, a fence flag 318 is set. The utility of this flag includes allowing one of several processes attempting to simultaneously insert the specific key-value pair into RKVS 112 to take priority over the insertion operation by instructing other processes to abort the key-value pair insertion, thereby preventing double insertion of the key-value pair into RKVS 112.

[0055] According to an embodiment, the node identifier 322 in the header 310 is used to identify the node, i.e., machine 102, on which hash table 114 is stored. In this embodiment, the node identifier 322 is used in conjunction with a distributed hash table that implements RKVS 112, i.e., to track nodes in a multi-node system that store different key-value pairs.

[0056] Figure 3 An example of the content 350 of a hash table 114 implementing RKVS 112 is further depicted. The example hash table content 350 includes entries 340-348 and column type descriptions to indicate a list of abbreviations for different kinds of data within each entry. Although columns are described, according to the embodiment, the entries of the hash table implementing RKVS 112 include strings of data, such as 1000011100000000 6_BYTE_KEY 8_BYTE_VALUE, without separating the data by columns.

[0057] Therefore, each of entries 340-348 in the example hash table content 350 includes data for flags 312-318 (described in that order), key information 320, and value information 330. In this document, the assigned flag 312, injection flag 314, time flag 316, and fence flag 318 are represented as an ordered sequence of four bits, such as 1000, where each corresponding flag is represented by its position within the ordered bit sequence in the order indicated by the hash table entry configuration 300. Entries in hash table 114 may include other data, such as node identifier 322, which is not included in the example hash table content 350. Node identifier 322 (such as “01110” included in the example data string above) identifies the node intended to store key-value pairs, such as machine 102 in the case of entries belonging to hash table 114.

[0058] According to one or more embodiments, any RDMA request that allows retrieval of data from hash table entries and evaluation of retrieved data and / or writing to hash table entries can be used to implement operations within RKVS 112, as described herein. According to embodiments, compare and exchange (CAS) requests are used, as needed, to write or identify the contents of the header 310, key 320, and / or value 330 for entries in hash table 114. CAS requests can be executed remotely using RDMA-based CAS requests, or they can be executed locally, for example, by the database server instance 108 maintaining RKVS 112.

[0059] The following pseudocode (1) describes an example implementation of the CAS operation:

[0060]

[0061] Pseudocode (1): Example implementation of CAS operation

[0062] This example CAS operation determines whether the value at the accum pointer is the same as the value at the dest pointer. If so, the operation returns true and swaps the information at the dest pointer with the newval argument. If not, the operation returns false and fills the position identified by the accum pointer with the value at the dest pointer for later review.

[0063] According to an embodiment, the CAS operation operates on 8 bytes of data. Therefore, a single CAS operation can (a) read and / or write both header 310 and key 320 information at a given entry in hash table 114, or (b) read and / or write value 330 at the entry. Thus, any operation at least partially implemented by a CAS operation requires at least two sub-operations for the target entry, which includes at least a first sub-operation of reading / writing header 310 and key 320 and a second sub-operation of reading / writing various portions of the target entry as needed.

[0064] A given suboperation may require the execution of multiple entries in one or more hash tables. For example, as described below, an insert operation requires the client to attempt to retain all entries mapped to the key-value pair being inserted. According to one embodiment, the requests required to execute a given suboperation are performed in parallel by the client. This embodiment is used to describe the example RKVS operation described herein. However, according to another embodiment, the various requests required to execute a given suboperation are performed serially by one or more clients accessing RKVS 112.

[0065] Search operation design

[0066] Returning to the discussion of step 202A in flowchart 200, the key-value pair operation performed by client 122 reads the key K from RKVS 112. x Associated value V x The search operation. Client 122 in K x Then, run hash function H1 to identify the first candidate entry for key-value pairs in hash table 114. For example, H1 will assign K... x Entry 340 is mapped to the example hash table content 350. Client 122 initiates the first sub-operation of the lookup operation by sending an RDMA CAS request to NIC 140 via network 130, wherein the CAS request reads 8 bytes of the identified entry 340, including header 310 information and key 320 information. According to embodiments, the description herein of the sub-operation for evaluating or setting the state of the header 310 and key 320 information from the target entry relates to such a CAS request (or RDMA or local).

[0067] CAS-type operations are useful for this purpose because the operation of a CAS request inherently provides information about the state of the header and whether the key information from the target entry matches the expected state. Therefore, in the case of a first sub-operation configured to explore the state of the target entry, a CAS request allows for easy understanding of the entry's state. Furthermore, in the case of a sub-operation configured to set the header and / or key information of the target entry, existing information is only overwritten if the entry is in the expected state. This inherent check of the entry state by the CAS request before writing to the entry prevents accidental overwriting of changes made by other processes.

[0068] The CAS request compares the first 8 bytes of entry 340 with the 8 bytes of data that would be present in entry 340 if it were filled with specific key-value pairs and ready to be read. Specifically, if the entry is ready to be read, the flag in the header 310 information for that entry will be 1000, i.e., a set allocated flag indicating that the entry is occupied, while other flags are not set to indicate the absence of any special circumstances that would cause the entry to be in an intermediate state. Therefore, in the example RDMA CAS operation from client 122, the accum pointer points to the value 1000 01110 0000000K. x K x This is the 6-byte key value of the specific key-value pair that client 122 is trying to find. This value, used for the accum pointer, indicates that the request is for a specific key-value pair associated with a node identifier 322 01110 (e.g., the identifier of machine 102) that is in a read-ready state. Figure 3As indicated in the diagram, entry 340, referenced by the dest pointer in the CAS operation, has the value 1100011100000000K. x It indicates that the entry is related to K. x This is associated, but the set injection flag indicates that a write operation to the entry is in progress and the entry is not yet ready to be read. Insert operations using injection flags are described in more detail below.

[0069] Finally, the newval argument requested by the CAS is the same as the value located at the accum pointer, because client 122 does not need to change the 8-byte value retrieved from entry 340. According to an embodiment, client 122 enables multiple threads to map each of the hash functions H1, H2, and H3 in hash table 114 to K in parallel. x Each of the entries in the execution instruction performs a CAS operation.

[0070] Return to Figure 2 The discussion of flowchart 200, at step 202B, determines one or more flags from the set of flags that are set. For example, in the example CAS request above, the values ​​stored at the accum and dest pointers are not the same, and the CAS operation returns false, which indicates to client 122 that entry 340 is not yet ready to be read. Due to the failed comparison, the memory referenced by the accum pointer is filled with the value at the dest pointer, i.e., 110001110 0000000K. x Client 122 determines, based on the value referenced by the accum pointer, that the injection flag 314 is set for entry 340, indicating that the insertion of data in entry 340 is in progress and has not yet been completed.

[0071] At step 204 of flowchart 200, based on the setting of one or more flags, the first process aborts a second sub-operation of a specific key-value pair operation. For example, client 122 determines that a lookup operation for a specific key-value pair should be aborted for entry 340 based on the setting of injection flag 314 for entry 340. Accordingly, client 122 aborts a second sub-operation that would include an RDMA read request to read value 330 from entry 340.

[0072] As another example, the second hash function H2 used for hash table 114 will K xEntry 346 is mapped to the example hash table content 350. As described above, client 122 determines that a timestamp 316 has been set for entry 346, indicating that data in entry 346 is being relocated to another entry and that the relocation operation has not yet been completed. Therefore, client 122 determines, based on the timestamp 316 set for entry 346, that a lookup operation for a specific key-value pair should be aborted for entry 346. Accordingly, client 122 aborts a second sub-operation of the lookup operation that would include an RDMA read request to read value 330 from entry 346.

[0073] Therefore, client 122 does not read from RKVS 112 when another process performs an insert or relocation operation on the key-value pair. x V x The value of ) could cause the lookup to fail, instead waiting a predetermined amount of time for other processes to complete the current operation affecting the target entry before trying the lookup operation again.

[0074] Search operation timed out

[0075] In some cases, there may be a delay between a client finding the desired key in a specific entry of hash table 114 and reading the associated value from the entry. During this delay, the entry may have changed, causing the client to fail to find the changed value. Accordingly, in an embodiment, each client calculates the amount of delay between the start of the first sub-operation of a given lookup operation and the start of the second sub-operation of the lookup operation. If this delay becomes greater than a predetermined lookup timeout period, then the client restarts the lookup operation from the first sub-operation.

[0076] As described in further detail below, the operation of changing the value of a target entry requires three sub-operations: (1) a first sub-operation (e.g., CAS type) to set an appropriate flag in the header 310 of the target entry and set key 320 to the appropriate key if necessary; (2) a second sub-operation (e.g., write type) to set the value 330 of the target entry to the value of the key-value pair being written; and (3) a third sub-operation (e.g., CAS type) to de-set the appropriate flag in the header 310 to indicate the completion of the write operation. Accordingly, according to one or more embodiments, the predetermined lookup timeout period represents a time amount greater than the latency of one RDMA operation and less than the latency of two RDMA operations performed back-to-back.

[0077] To illustrate, the successful first sub-operation of a lookup operation ensures that no write operation has been initiated on the target entry. The second sub-operation, initiated immediately after the first sub-operation completes, ensures that any write operation initiated after the first sub-operation of the lookup operation cannot complete the two sub-operations required to change the value before the second sub-operation retrieves the correct value from the entry. In other words, adhering to a predetermined lookup timeout ensures that the data in the target entry of a given lookup operation cannot be modified before the lookup operation completes.

[0078] Explanation of how to avoid search failures

[0079] The following is an illustration of potential lookup failures that can be caused by latency between sub-operations of a lookup operation in the Nessie system. In this example, according to the Cuckoo Scheme with two hash functions, there are two entries P and S in the index table, which can be used (K x V x The main entry P of the index table is stored (K). x V x The location information of ) will identify the first entry in the auxiliary data table as the location of the key-value pair. When entry P stores (K x V x When processing location data, process A reads (K) from entry P. x V x The location of ). After reading the location data from entry P, and then reading the key-value pair (K) from the location indicated in the auxiliary data table. x V x Before that, change the data stored in the first entry of the auxiliary data table to different key-value pairs. For example, process B, which controls the first entry in the data table, will change (K... x V x The process moves to the second entry in the auxiliary data table and populates the first entry with a different key-value pair. After process B changes the data in the auxiliary data table, for example in time slot P, process B updates the index table (K). x V x The position of ). However, process A has already read the previous (and now incorrect) (K) from the index table. x V x The system attempts to retrieve the location data and read the key-value pair from the first entry in the auxiliary data table, but this does not produce the expected key-value pair. Therefore, the lookup operation of process A cannot retrieve V from the key-value repository. x .

[0080] The above embodiments address this potential search failure. Specifically, according to one or more embodiments, given the above-mentioned method where (K) x V xIn a scenario where the primary entry P is empty and key-value pairs are stored in auxiliary entries S, client 122 initiates a lookup operation on the key-value pairs in entry P of hash table 114. Client 122 determines that the assigned flag of entry P is not set, indicating that entry P is empty. Client 122 also initiates a lookup operation on the auxiliary entries, entries S, for the key-value pairs.

[0081] After reading the header and key of entry S, and before reading its value, database server instance 108 initiates a relocation operation to relocate (K) x V x ) Move from entry S to entry P in hash table 114, for example, due to conflict resolution. Combine (K) x V x Redirecting to entry P, database server instance 108 sets the assigned flag and injection flag in the header 310 of entry P, and also sets the key 320 of entry P to K. x Database server instance 108 also sets a timestamp in the header 310 of entry S, where the allocated bit of entry S is kept set. This information in the header 310 and key 320 of entries P and S shows (K x V x It is currently in the process of being inserted into entry P and is also in the process of being repositioned from entry S, and these processes are not yet complete.

[0082] If the scheduled lookup timeout for the lookup operation performed by client 122 on entry S has not expired and the lookup operation has completed, then client 122 can retrieve the correct value from entry S via an RDMA read request before database server instance 108 changes the value stored in entry S. However, if the second sub-operation of the lookup operation is not initiated before the scheduled lookup timeout expires, then it is not guaranteed that client 122 can retrieve the value V before entry S is changed. x Therefore, if the lookup operation times out, client 122 aborts the lookup operation. If client 122 then reads entries P and S before completing the relocation operation, client 122 determines from the entry header 310 that they are not yet ready to be read. Therefore, client 122 waits for the amount of time configured to allow write operations to terminate before retrying the lookup operation. In this way, client 122 is guaranteed to retrieve the correct value of the key-value pair.

[0083] Insertion operation design: Injection flag

[0084] According to one or more embodiments, the utilization of injection flag 314 and fence flag 318 allows the process to avoid insertion failures. Specifically, embodiments allow multiple processes to reliably write to and delete from hash table 114, thereby avoiding double insertion and deletion problems.

[0085] The following illustrates a potential insertion failure that could result from multiple processes inserting the same key-value pair into RKVS 112, as might be seen in a Nessie system. In this example, two processes, A and B, attempt to insert the same key-value pair (K...) into RKVS 112 within the same timeframe. z V z Insert it into the key-value store. Additionally, a third process C attempts to delete (K) after its insertion. z V z In this example, (K) z V z Mapping (K) to two entries P and S in the index table. Combining (K) z V z To write to the key-value store, process A first subscribes to entries P and S in the index table by setting their flags to invalid. Then, it sets the data for entry P in the auxiliary data table to indicate (K). z V z After the location data of ) is obtained, and in process A it will be compared with (K) z V z Before the associated flag is set to valid, process B attempts to set the same (K) flag to valid. z V z Write entry P. It first checks the relationship with (K). z V z The associated flag is not yet valid. Since the flag has not been updated to indicate that the pair is valid, based on the proposed design, process B will repeat the insertion operation after a certain delay. The amount of this delay is not specified in the Nessie design. If this delay is longer than the completion time of process A, then (K) z V z If the time required for inserting entry P and for procedure C to complete the deletion operation from entry P is long, then after procedure C deletes the data (K... z V z The deletion operation performed by process C should have been performed by process A or B to insert the pair (K). This result is undesirable because the deletion operation performed by process C should have been performed by process A or B to insert the pair (K). z V z Then remove (K) from the hash table. z V z ).

[0086] The embodiments address potential insertion failures, such as the one explained above. Specifically, according to the embodiments, the header 310 for each entry in hash table 114 includes an injection flag and a fence flag for communicating the current state of the ongoing insertion operation to other processes. Figure 4 A series of snapshots 430-438 depict the hash table 114 (with entries 402-410) on the progress timeline. According to an embodiment, snapshots 430-438 depict how clients 122-126 interact in the aforementioned scenario. Specifically, in Figure 4 In the example depicted, (K) z V z This maps to entries 406 and 410. In this example, the collision avoidance scheme for hash table 114 uses hash functions H1 and H2 to map key-value pairs to entries in the hash table, respectively. z V z Based on H1, entry 406 is mapped to H2. Accordingly, entry 406 takes precedence over entry 410, which is mapped to a key-value pair based on H2.

[0087] At snapshot 430, client 122 determines that entries 406 and 410 are empty via the first sub-operation of reading the header 310 of the entries in the insert operation. Client 122 then sets their injection and assigned flags and K... z These entries are reserved by inserting them into the key 320 information used for the entries, as depicted in snapshot 432. Therefore, the embodiment prevents double insertion of key-value pairs by retaining all available entries mapped to key-value pairs in hash table 114.

[0088] Before client 122 completes the insert operation, at snapshot 432, client 124 attempts to insert the same (K) z V z Insert entries 406 and 410. Because the first sub-operation of this insertion involves reading at least header 310 and key 320 from the entries to determine the current state of the entries, client 124 determines that entries 406 and 410 are both reserved by another process for the same key K. z Based on this determination, client 124 aborts the second sub-operation of the insert operation and does not perform any further action to insert the key-value pair. If client 126 attempts to look up (K) in entries 406 and 410 before the insert operation completes... z V z As the first sub-operation of the delete operation, client 126 determines the target (K) based on the injection bits of the two entries. z V zThe insertion operation of the first sub-operation has not yet been completed, as shown in snapshot 432. Therefore, client 126 delays for a predetermined amount of time, sufficient for another process to complete the insertion operation, and then restarts the deletion operation from the first sub-operation, as described in further detail below.

[0089] As shown in snapshot 434, after completing the first sub-operation of the insert operation, client 122 performs the second sub-operation of the insert operation (e.g., using an RDMA write-type request) to set the value 330 in entry 406 to V. z According to an embodiment, client 122 selects entry 406 as the entry for inserting a key-value pair based on a sequence of hash functions upon which the collision avoidance scheme of hash table 114 is based.

[0090] As shown in snapshot 436, after completing the second sub-operation, client 122 performs a third sub-operation of the insert operation to set a flag in the header 310 of entry 406 to 1000, thereby indicating that the information in entry 406 has been finalized and is ready to be read. The third sub-operation also involves setting a flag in the header 310 of entry 410 to 0000 (and, according to an embodiment, zeroing out the key 320 information in the entry) to indicate that the entry is empty. Therefore, hash table 114 only includes (K z V z A copy of the key-value pair, although multiple processes attempt to insert the key-value pair at a time.

[0091] Moreover, as shown in snapshot 438, the aforementioned insertion failure is reliably avoided and can be successfully removed, for example, by client 126 from entry 406 of hash table 114 (K). z V z For example, if client 126 attempts to delete (K) at any time before the insert operation is completed. z V z The first sub-operation that performs the deletion operation, retrieving at least header 310 and key 320 information from entries 406 and 410, reveals that for both entries 406 and 410, the assigned and injected flags are set, and the key 320 information shows K. z As shown in snapshot 432. This configuration of the header flags and key information indicates that entries 406 and 410 are currently configured for insertion (K). z V z The insert operation was scheduled. This information caused client 126 to abort the second sub-operation of the delete operation (used to remove the key-value pair from RKVS112) and wait for a predetermined amount of time to allow the injection flag to be cleared.

[0092] After waiting for a predetermined amount of time, during which time the injection flag is cleared from entries 406 and 410, as shown in snapshot 436, client 126 performs a second attempt at the first sub-operation of the deletion operation, which reads header 310 and key 320 from entries 406 and 410. Based on the result of the first sub-operation, client 126 determines (K z V z The key-value pair is stored in entry 406 and is available for deletion, i.e., it has a flag bit of 1000. The client 126 performs a deletion operation by removing the key-value pair from RKVS 112 by zeroing out the header 310 and key 320 information of entry 406, as shown in snapshot 438.

[0093] Insertion Operation Design: Fence Sign

[0094] Fence flag 318 is useful in preventing insertion failures by allowing communication between procedures regarding the priority of the procedure inserting a specific key-value pair. For example, Figure 5 A series of snapshots 520-536 of hash table 114 (with example entries 502-510) over time are depicted to illustrate an example of the usefulness of fence markers. Specifically, client 122 performs a process of reading header 310 and key 320 information from three entries 502, 506, and 510 in the hash table mapped to key-value pairs. y V y This is the first sub-operation of the operation to insert into hash table 114. In this example, client 122 performs the first sub-operation, and hash table 114 is shown in snapshot 520. Therefore, client 122 finds that entry 502 is inserted into another key-value pair (K). x V x The system was found to be in use, and entries 506 and 510 were found to be empty. Client 122 then performed an insert operation to set the key 320 information of entries 506 and 510 to K. y And set the assigned and injected flags in the header 310 information of those entries to indicate that they are being targeted at (K) y V y The second sub-operation of the insertion operation allows the move operation to continue on empty entries. The result of the second sub-operation is shown in snapshot 522.

[0095] In client 122, entry 502 is determined to be (K) x V xAfter being occupied, the key-value pair is deleted from entry 502, as indicated by the zeroing flag in header 310 and the zeroing key 320 of entry 502 in snapshot 524. In this example, such deletion could occur after client 122 checks entry 502 via the first sub-operation of the insert operation, before client 122 completes the insert operation, and when a second client, such as client 124, initiates the deletion of (K... y V y ) Any time before the second insert operation in hash table 114. Note that, for ease of interpretation, snapshot 524 also indicates the time before deleting (K) from the entry. x V x Afterwards, the value 330 of entry 502 is zeroed out. Once the header information of a given entry is set to indicate that the entry is empty (i.e., 0000), the information stored in the rest of the entry is considered garbage information until another key-value pair is inserted into the entry. In this way, it is not necessary to perform the operation of zeroing out the value 330 of an entry in order to delete the contents of an entry.

[0096] continue Figure 5 For example, when hash table 114 is in the state indicated by snapshot 524, client 124 initiates a process to transfer (K... y V y The second insertion operation is performed by inserting the data into hash table 114. Specifically, client 124 performs the insertion operation from the mapping to (K... y V y The client 124 reads the first sub-operation of header 310 and key 320 from entries 502, 506, and 510. In this case, the client 124 finds that entry 502 is empty, and that both entries 506 and 510 have the injection flag set, and that the key 320 information indicates K. y .

[0097] If the client discovers that the injection bit has already been set in one of the target entries for the insert operation, the response depends on whether the key 320 information matches the key the client is inserting. If the key 320 information of a given target entry indicates a different key-value pair, then the entry is locked for the insertion operation of a different key-value pair. As shown in snapshot 522 of entries 506 and 510 held by client 122, the first sub-operation of the insert operation is performed using one or more CAS requests that attempt to set the header 310 and key 320 of an empty target entry in hash table 114 to header information with the assigned and injection bits set and key information indicating the key to be inserted. This action holds the entry for inserting the desired key-value pair. If at least one entry is successfully held for key-value pair insertion during the CAS operation, then the client continues to insert values ​​into one of the held entries.

[0098] However, if none of the CAS operations used as the first sub-operation of the insert operation successfully reserve an entry for the insert, the client waits for a period of time to clear the injection bits that list the target entries with different keys. This allows entries to be reserved to avoid double insertions rather than to allow another key-value pair to be inserted into the desired entry. If, after the waiting period, the client finds that all target entries for the insert operation are full, the client initiates conflict remedies, as described in detail below.

[0099] On the other hand, if client 124 attempts to retain (but finds the injection bit set) one or more target entries whose key 320 indicates that client 124 is inserting the same key, this means that another process has locked one or more entries to insert the same key-value pair. In this case, if client 124 is able to retain at least one target entry, then the client attempts to prioritize inserting the key-value pair by using the fence bit to prevent another insertion. To use Figure 5 To illustrate with an example, client 124 discovers that entries 506 and 510 have injection flags set and also have the same key in key 320 as the key that client 124 is attempting to insert. Therefore, the attempt to retain these entries 506 and 510 fails. However, as shown in snapshot 526, the attempt to retain entry 502 succeeds, considering that the entries were empty when client 124 performed the first sub-operation of the insert operation. In snapshots 526-536, the dashed boundary around entry 502 and the absence of dashed boundaries around entries 506 and 510 illustrate the ownership difference between entries 502 and 506 and 510. Specifically, client 124 controls entry 502, while client 122 controls entries 506 and 510.

[0100] When multiple different clients reserve entries in hash table 114 to insert the same key-value pair, one client has priority to perform the insert operation. According to one embodiment, priority is determined by a client that prevents one or more other clients from performing the insert, or by a client that completes the insert operation before another client has the opportunity to prevent it. According to another embodiment, priority passes to the client that reserved the entry with the lowest index for the key-value pair. Because priority is asserted by one of the inserting clients, multiple insert operations on the same key-value pair may not occur simultaneously. Specifically, a client that has lost priority detects an action by another client establishing priority and aborts any further sub-operations required to complete a non-priority insert operation.

[0101] To prevent client 122 from completing the insertion operation that began in entries 506 and 510, client 124 attempts to set the fence bits for entries 506 and 510. In this example, the attempt to set these fence bits succeeds, as shown in snapshot 528, because client 122 has not yet set (K) y V y Successfully inserted entry 506 or 510. Client 124 sent V y An RDMA write request for value 330 of entry 502 is sent to continue the second sub-operation of the insert operation, as shown in snapshot 532. Client 124 performs the third sub-operation of the insert operation by setting header 310 of entry 502 to 1000, as shown in snapshot 534, indicating that the entry is occupied and ready to be read.

[0102] When client 122 attempts to continue its pair (K) y V y When inserting V, client 122 uses an RDMA write request to insert V. y The second sub-operation for performing the insert operation is written as the value 330 of entry 506, as shown in snapshot 530. However, when client 122 attempts to perform the third sub-operation of changing the header 310 of entry 506 from "1100" to "1000" using an RDMA CAS request to perform the insert operation, the request fails because the fence bit is set. That is, the comparison performed by the CAS request between the expected value (1100) and the actual value (1101) of the header 310 of entry 506 causes the CAS request to return false and no swap is performed. Based on the fence bit being set in the header 310 of entry 506, client 122 determines that it did not insert (K y V y Priority of ). According to an embodiment, in response to determining that it has not inserted (K) y V y Prioritizing the insertion of key-value pairs, client 122 clears the header 310 and key 320 of entries 506 and 510, as shown in snapshot 536. According to another embodiment, as part of its third sub-operation, client 124 clears the header 310 and key 320 of entries 506 and 510. Therefore, only one client successfully inserts the key-value pair.

[0103] Figure 6 A series of snapshots 620-630 of hash table 114 over time are depicted, where events are similar to Figure 5 Specifically, those events at snapshots 520 / 620, 522 / 622, and 524 / 624. One difference in the snapshot sequence is that client 122 continues the insert operation (K...). y V yThe second sub-operation is inserted into hash table 114. Specifically, as shown in entry 606 of snapshot 624, client 122 sends V y An RDMA write request is made to the value 330 of entry 606. Before client 122 sets the header portion of entry 606 to "1000", client 124 retains entry 602 and determines that entries 606 and 610 were previously retained by another client, as shown in snapshot 626. After client 124 retains entry 602 and before client 124 sets the fence bits for entries 606 and 608, client 122 performs a third sub-operation of insert operation, setting the header portion of entry 606 to "1000" and zeroing the header portion and key 320 of entry 610, as shown in snapshot 628. When client 124 attempts to set the fence bits of entries 606 and 610 using a CAS-type RDMA request, a CAS request is made because (K y V y The header 310 and key 320 in entry 602 have been successfully inserted into entry 606 by another client. That is, the header portion (1100) expected by client 124 in the CAS request does not match the actual header portion of the entry at snapshot 628 ("1000" for entry 606 and "0000" for entry 610). Therefore, client 124 aborts the remaining sub-operations of the insertion operation and zeros the header 310 and key 320 in entry 602, as shown in snapshot 630.

[0104] Insertion operation design: Time stamp

[0105] During the first sub-operation of the insert operation, the insert client may discover that one of the target entries has a time bit set, indicating that the specific target entry is involved in conflict resolution, as described further below. If the key 320 information of the specific target entry has the same key as the one being inserted, then the key-value pair already exists in RKVS 112 and the client aborts the insert operation. If the key 320 information of the entry indicates a different key than the one being inserted, and if the client has successfully reserved one or more other entries in hash table 114 for the insert operation, then the client continues the insert operation using the reserved one or more entries. However, if the client has not reserved any other entries for the insert operation, then the client waits to have the time bit cleared from the specific target entry before continuing the insert operation.

[0106] Conflict resolution: key-value pair relocation

[0107] As indicated above, conflict avoidance schemes are generally used within hash table 114, which implements RKVS 112. However, even with this scheme applied, conflicts can still occur within hash table 114, i.e., when a client attempting to write a key-value pair to hash table 114 does not find a free entry mapped to that key-value pair. In this case, if the client is not database server instance 108, the client sends a conflict resolution request for the key-value pair to database server instance 108. After identifying the conflict (either locally or based on the received conflict resolution request), database server instance 108 resolves the key-value pair conflict and inserts the key-value pair into RKVS 112.

[0108] According to an embodiment, in order to limit the processing power required to resolve conflicts, database server instance 108 determines whether the relocation operation limits the amount of time spent resolving conflicts (as described below) to a predetermined amount of time for conflict remediation. Therefore, in order to resolve a conflict, database server instance 108 utilizes at most the predetermined amount of time for conflict remediation to determine whether relocating one or more key-value pairs in hash table 114 results in space being cleared for key-value pairs of one of the entries to be inserted into its mapping.

[0109] According to one embodiment, this determination includes performing relocation within hash table 114. According to another embodiment, this determination includes performing analysis of potential relocations during the determination process without performing relocations. In this embodiment, if database server instance 108 determines that relocation results in space for key-value pairs before the timer expires, then database server instance 108 performs the relocation required to clear the space as described above. According to one embodiment, conflict resolution using the determined relocation sequence must also be completed before the time expires to avoid hash table resizing.

[0110] According to one or more embodiments, a flag in the header 310 of an entry in the hash table used to implement RKVS 112 allows conflict resolution actions to be performed within RKVS 112—including relocating key-value pairs within hash table 114 and reallocating the hash table to increase its size—while maintaining the availability of data stored in RKVS 112. Specifically, database server instance 108 sets a timestamp associated with the relocated key-value pairs for conflict remedy, so that the occurrence of the relocation operation is communicated to other processes utilizing information in RKVS 112. Therefore, other processes accessing RKVS 112 can continue to access data while correctly handling the entries involved in the relocation operation.

[0111] The relocation operation involves three RDMA requests (which can be executed in parallel on the destination and source entries as needed): (1) a first CAS-type sub-operation for each entry to set appropriate flags in the headers of the destination and source entries and set key 320 to the appropriate key as needed; (2) a second write sub-operation to set the value 330 of the destination entry to the value of the relocated key-value pair; and (3) a third CAS-type sub-operation to adjust the appropriate flags in the headers 310 of the source and destination entries to complete the relocation operation.

[0112] To illustrate the relocation operation, Figure 7 Snapshots 720-728 of hash table 114 over time are depicted, comprising five entries 702-710. Specifically, at snapshot 720, entries 702 and 708 are filled with key-value pairs (K... x V x ) and (K y V y Therefore, these entries are associated with the flag value "1000", where the assigned flag is the only flag that is set. Entries 704, 706, and 710 are empty and are therefore associated with the flag value "0000".

[0113] With entries 702-710 in the state indicated at snapshot 720, client 122 attempts to transfer the key-value pair (K... z V z Insert into hash table 114. The collision avoidance scheme implemented for hash table 114 has two hash functions, which respectively hash (K) z V z ) maps to entries 702 and 708. Because (K z V z The two entries mapped to are occupied by other key-value pairs, so client 122 sends a request to database server instance 108 for (K). z V z The database server instance 108 determines whether entry 702 or 708 can be relocated within hash table 114 to resolve the conflict. y V y This is mapped to entries 706 and 708, where entry 706 is not occupied, as shown in snapshot 720.

[0114] In order to make (K) y V yTo relocate from source entry 708 to destination entry 706, database server instance 108 performs the first sub-operation of the relocation operation by executing two local CAS operations (e.g., in parallel), the result of which is depicted in snapshot 722. Specifically, one of the CAS operations changes the header 310 bits to "1100" and the key 320 to K in entry 706. y To indicate K by setting the injection bit y The value is being written to entry 706. Another CAS operation in entry 708 changes header bit 310 to "1010" to indicate the current content of the entry via the set time bit (K). y V y It is currently being relocated.

[0115] Once the first sub-operation of the relocation operation is completed, database server instance 108 will then relocate via V. y The second sub-operation of the relocation operation is performed by locally writing the value 330 of entry 706, as shown in snapshot 724. Once the second sub-operation of the relocation operation is complete, database server instance 108 performs the third sub-operation of the relocation operation via (e.g., executed in parallel) two local CAS operations, the result of which is depicted in snapshot 726. Specifically, one of the CAS operations changes the header 310 bit in entry 706 to "1000" to indicate that the entry is occupied and ready for other operations. Another CAS operation changes the header 310 bit in entry 708 to "0000" to indicate that entry 708 is currently empty, and according to an embodiment, other information within entry 708, such as key 320, is also set to zero. Thus, the relocation operation releases entry 708 for insertion (K z V z Database server instance 108 performs insert (K) z V z The operation, as described in detail above, has its final result depicted in snapshot 728.

[0116] According to one or more embodiments, if database server instance 108 determines during the first sub-operation of a relocation operation that the destination entry has an injection bit set, then database server instance 108 delays for a predetermined amount of time and then retryes the relocation operation. This delay allows the client that retains the target destination entry time to complete the current operation and may release the entry for use as a relocation destination, for example, in cases where the entry is retained to avoid double insertion.

[0117] Conflict resolution: Hash table resizing

[0118] If the predetermined time for conflict remedy expires before the detected conflict is resolved by the relocation operation, then database server instance 108 automatically stops the relocation operation and adjusts the size of the hash table implementing RKVS 112 in volatile memory 106. For illustration, database server instance 108 allocates another hash table 800 in cache 110 to implement RKVS 112, as... Figure 8 The same reference numerals in the figures refer to... Figure 1 The corresponding description and illustration are provided. Hash table 800 is larger than hash table 114, for example, twice the size. According to an embodiment, stopping the relocation operation includes unsetting any time bits that the database server instance 108 has set in connection with the conflict remediation relocation operation.

[0119] Figure 9 A flowchart 900 is depicted for migrating at least a portion of a key-value store from a first hash table in volatile memory to a second hash table in volatile memory. At step 902 of flowchart 900, one or more messages are sent to one or more clients configured to request access to the key-value store at the first hash table using RDMA, wherein one or more messages include information identifying the second hash table. For example, after determining that key-value pairs of RKVS 112 will be migrated from hash table 114 to hash table 800, database server instance 108 sends messages to each client authenticated as having access to hash table 114 via RDMA, such as clients 122-126. These messages indicate the migration of the key-value store and identify hash table 800. These messages allow clients to continue accessing RKVS during the migration between the two hash tables.

[0120] Based on these messages, clients 122-126 (and any other clients registered to access hash table 114 via RDMA) register to access hash table 800 via RDMA, as described in detail above. Therefore, database server instance 108 and any client authorized to access hash table 114 are able to access both hash table 114 and hash table 800 during the migration.

[0121] At step 904, multiple key-value pairs in the key-value store are relocated from the first hash table to the second hash table. For example, database server instance 108 migrates key-value pairs stored in hash table 114 to hash table 800 without requiring clients in RKVS 112 to stop operating during the migration. Except that the source entry for the relocation operation is in hash table 114 and the destination entry for the relocation operation is in hash table 800, database server instance 108 performs the migration between hash table 114 and hash table 800 in a manner similar to the relocation operation described above.

[0122] At step 906, both the first and second hash tables are maintained while multiple key-value pairs are relocated from the first hash table to the second hash table. For example, database server instance 108 maintains both hash table 114 and hash table 800 in volatile memory 106 during the migration of key-value pairs.

[0123] At step 908, while both the first hash table and the second hash table are maintained, one or more first RDMA requests from one or more clients for the first hash table are serviced, and one or more second RDMA requests from one or more clients for the second hash table are serviced. For example, NIC 140 service RDMA requests from one or more clients 122-126 for both hash table 114 and hash table 800.

[0124] Specifically, during the migration, lookup operations are performed in both hash table 114 and hash table 800. Furthermore, insertion operations are performed in hash table 800, not hash table 114. Deletion operations are also performed in both hash table 114 and hash table 800. During the migration process, when a client encounters an entry in hash table 114 that has a timestamp set (which is the target of a specific operation), the client waits for a period of time to allow relocation to complete before attempting the operation in hash table 800 (instead of both hash table 114 and hash table 800).

[0125] At step 910, after relocating multiple key-value pairs from the first hash table to the second hash table, the first hash table is released. For example, after relocating all key-value pairs from hash table 114 to hash table 800, database server instance 108 sends another message to each client with RDMA access to RKVS 112 to notify the client that the migration is complete. Database server instance 108 then releases hash table 114, and hash table 800 subsequently acts as the hash table implementing RKVS 112 on server machine 102.

[0126] Therefore, although some operation types may be slower during the migration for conflict resolution due to the need to perform operation types on hash tables 114 and 800, all types of operations (including insert, delete, and lookup) can continue at each step of conflict resolution in RKVS 112.

[0127] Other system details

[0128] refer to Figure 1One or more clients 122-126 execute on machine 120, for example, as parallel processes. Clients 122-126 can be implemented differently, for example, as processes, database applications, or other database server instances that execute on machines other than the host server machine. A client can also be a client-side database server process linked to a database application, or a combination of such a process and the database application to which it is linked.

[0129] Although clients 122-126 are described herein as residing on a single machine, clients that can access RKVS112 can execute on any number of computing devices coupled to machine 102, for example, via network 130. Applications, processes, or services such as any of clients 122-126 or database server instance 108 run on computing devices and comprise a combination of software and resource allocations from the computing devices. Specifically, an application or service is a combination of integrated software components and allocations of computing resources, such as memory and / or processes on the computing devices for executing the integrated software components on a processor, the combination of software and computing resources dedicated to performing the prescribed functions of the application or service.

[0130] Hardware Overview

[0131] According to one embodiment, the techniques described herein are implemented by one or more dedicated computing devices. The dedicated computing device may be hardwired to execute these techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) persistently programmed to execute these techniques, or may include one or more general-purpose hardware processors programmed to execute these techniques according to program instructions in firmware, memory, other storage devices, or a combination thereof. Such a dedicated computing device may also combine custom hardwired logic, ASICs, or FPGAs with custom programming to implement these techniques. The dedicated computing device may be a desktop computer system, a portable computer system, a handheld device, a networking device, or any other device that combines hardwired and / or program logic to implement these techniques.

[0132] For example, Figure 10 This is a block diagram illustrating a computer system 1000 on which embodiments of the present invention can be implemented. The computer system 1000 includes a bus 1002 or other communication mechanism for transmitting information, and a hardware processor 1004 coupled to the bus 1002 to process information. The hardware processor 1004 may be, for example, a general-purpose microprocessor.

[0133] Computer system 1000 also includes main memory 1006, such as random access memory (RAM) or other dynamic storage devices, coupled to bus 1002, for storing information and instructions to be executed by processor 1004. Main memory 1006 can also be used to store temporary variables or other intermediate information during the execution of instructions by processor 1004. When stored in non-transient storage media accessible to processor 1004, these instructions make computer system 1000 a dedicated machine customized to perform the operations specified in the instructions.

[0134] The computer system 1000 also includes a read-only memory (ROM) 1008 or other static storage device coupled to the bus 1002 for storing static information and instructions for the processor 1004. A storage device 1010 (such as a disk, optical disk, or solid-state drive) is provided and coupled to the bus 1002 for storing information and instructions.

[0135] Computer system 1000 can be coupled to display 1012 (such as a cathode ray tube (CRT)) via bus 1002 for displaying information to the computer user. Input device 1014, including alphanumeric keys and other keys, is coupled to bus 1002 for transmitting information and command selections to processor 1004. Another type of user input device is cursor control 1016 (such as a mouse, trackball, or arrow keys) for transmitting directional information and command selections to processor 1004 and for controlling cursor movement on display 1012. Such input devices typically have two degrees of freedom on two axes (a first axis (e.g., x) and a second axis (e.g., y)), allowing the device to specify a position in a plane.

[0136] Computer system 1000 may implement the techniques described herein using custom hardwired logic, one or more ASICs or FPGAs, firmware and / or program logic (which, in combination with the computer system, make computer system 1000 a special-purpose machine or program computer system 1000 as a special-purpose machine). According to one embodiment, computer system 1000 performs the techniques described herein in response to processor 1004 executing one or more sequences of one or more instructions contained in main memory 1006. These instructions may be read into main memory 1006 from another storage medium (such as storage device 1010). Execution of the instruction sequence contained in main memory 1006 causes processor 1004 to perform the processing steps described herein. In alternative embodiments, hardwired circuitry may be used instead of software instructions or in combination with software instructions.

[0137] As used herein, the term "storage medium" refers to any non-transient medium that stores data and / or instructions that enable a machine to operate in a particular manner. Such storage media can include non-volatile media and / or volatile media. Non-volatile media include, for example, optical discs, magnetic disks, or solid-state drives, such as storage device 1010. Volatile media include dynamic memory, such as main memory 1006. Common forms of storage media include, for example, floppy disks, flexible disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, PROMs and EPROMs, FLASH-EPROMs, NVRAMs, any other memory chips, or magnetic tape cassettes.

[0138] Storage media differ from transmission media but can be used in conjunction with them. Transmission media participate in the transfer of information between storage media. For example, transmission media include coaxial cables, copper wires, and optical fibers, including conductors containing bus 1002. Transmission media can also take the form of sound waves or light waves, such as those generated during radio wave and infrared data communication.

[0139] Carrying one or more sequences of instructions to processor 1004 for execution can involve various forms of media. For example, the instructions may initially be carried on a disk or solid-state drive of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit them over a telephone line using a modem. A modem local to computer system 1000 may receive data over the telephone line and convert the data into an infrared signal using an infrared transmitter. An infrared detector may receive the data carried in the infrared signal, and appropriate circuitry may place the data on bus 1002. Bus 1002 transfers the data to main memory 1006, from which processor 1004 retrieves and executes the instructions. The instructions received by main memory 1006 may optionally be stored on storage device 1010 before or after execution by processor 1004.

[0140] Computer system 1000 also includes a communication interface 1018 coupled to bus 1002. Communication interface 1018 provides bidirectional data communication coupled to network link 1020, which is connected to local network 1022. For example, communication interface 1018 may be an Integrated Services Digital Network (ISDN) card, a cable modem, a satellite modem, or a modem providing data communication connectivity with a corresponding type of telephone line. As another example, communication interface 1018 may be a Local Area Network (LAN) card to provide data communication connectivity with a compatible LAN. A wireless link may also be implemented. In any such implementation, communication interface 1018 transmits and receives electrical, electromagnetic, or optical signals carrying streams of digital data representing various types of information.

[0141] Network link 1020 typically provides data communication to other data devices via one or more networks. For example, network link 1020 can provide a connection via local network 1022 to host computer 1024 or to data devices operated by Internet Service Provider (ISP) 1026. ISP 1026 then provides data communication services via a global packet data communication network (now commonly referred to as "the Internet" 1028). Both local network 1022 and Internet 1028 use electrical, electromagnetic, or optical signals carrying digital data streams. Signals through various networks, as well as signals on network link 1020 and through communication interface 1018 (which carries digital data to and from computer system 1000), are example forms of transmission media.

[0142] Computer system 1000 can send messages and receive data, including program code, through one or more networks, network links 1020, and communication interfaces 1018. In the Internet example, server 1030 can send requested code to the application through the Internet 1028, ISP 1026, local network 1022, and communication interface 1018.

[0143] The received code can be executed by processor 1004 when it is received, and / or stored in storage device 1010 or other non-volatile storage device for later execution.

[0144] Software Overview

[0145] Figure 11This is a block diagram of a basic software system 1100 that can be used to control the operation of computer system 1000. Software system 1100 and its components, including their connections, relationships, and functions, are intended to be exemplary only and are not intended to limit the implementation of one or more example embodiments. Other software systems suitable for implementing one or more example embodiments may have different components, including components with different connections, relationships, and functions.

[0146] Software system 1100 is provided to guide the operation of computer system 1000. Software system 1100, which may be stored on system memory (RAM) 1006 and fixed storage device (e.g., hard disk or flash memory) 1010, includes a kernel or operating system (OS) 1110.

[0147] OS 1110 manages the low-level aspects of computer operations, including managing process execution, memory allocation, file input and output (I / O), and device I / O. One or more applications, designated 1102A, 1102B, 1102C...1102N, can be "loaded" (e.g., transferred from fixed storage device 1010 to memory 1006) for execution by system 1100. Applications or other software intended for use on computer system 1000 can also be stored as downloadable computer-executable instruction sets, for example, for downloading and installing from Internet locations (e.g., web servers, app stores, or other online services).

[0148] Software system 1100 includes a graphical user interface (GUI) 1115 for receiving user commands and data graphically (e.g., "click" or "touch gestures"). These inputs can then be processed by system 1100 according to instructions from operating system 1110 and / or (one or more) applications 1102. GUI 1115 also displays the results of operations from OS 1110 and (one or more) applications 1102, allowing the user to provide additional input or terminate the session (e.g., log off).

[0149] OS 1110 can execute directly on the raw hardware 1120 of computer system 1000 (e.g., one or more processors 1004). Alternatively, a hypervisor or virtual machine monitor (VMM) 1130 can be inserted between the raw hardware 1120 and OS 1110. In this configuration, VMM 1130 acts as a software “buffer” or virtualization layer between OS 1110 and the raw hardware 1120 of computer system 1000.

[0150] VMM 1130 instantiates and runs one or more virtual machine instances (“guest machines”). Each guest machine includes a “guest” operating system (such as OS 1110) and one or more applications (such as application(s) 1102) designed to run on the guest operating system. VMM 1130 presents a virtual operating platform to the guest operating system and manages the execution of the guest operating system.

[0151] In some instances, VMM 1130 can allow a guest operating system to run as if it were running directly on the bare hardware 1120 of computer system 1000. In these instances, the same version of the guest operating system configured to run directly on the bare hardware 1120 can also run on VMM 1130 without modification or reconfiguration. In other words, VMM 1130 can provide full hardware and CPU virtualization to a guest operating system in some situations.

[0152] In other instances, the guest operating system can be specifically designed or configured to run on the VMM 1130 for improved efficiency. In these instances, the guest operating system is "aware" that it is running on the virtual machine monitor. In other words, the VMM 1130 can provide paravirtualization to the guest operating system under certain circumstances.

[0153] Computer system processes involve the allocation of hardware processor time and the allocation of memory (physical and / or virtual). Memory allocation is used to store instructions executed by the hardware processor, data generated by the execution of those instructions, and / or to store hardware processor state (e.g., register contents) between hardware processor time allocations when the computer system process is not running. Computer system processes run under the control of the operating system and can also run under the control of other programs executing on the computer system.

[0154] cloud computing

[0155] This article generally uses the term "cloud computing" to describe a computing model that enables on-demand access to a shared pool of computing resources, such as computer networks, servers, software applications, and services, and allows for the rapid provisioning and release of resources with minimal management effort or service provider interaction.

[0156] Cloud computing environments (sometimes called cloud environments or the cloud itself) can be implemented in various ways to best suit different requirements. For example, in a public cloud environment, the underlying computing infrastructure is owned by an organization that makes its cloud services available to other organizations or the general public. In contrast, private cloud environments are generally used only by a single organization or within a single organization. Community clouds are designed to be shared by several organizations within a community; while hybrid clouds include two or more types of clouds (e.g., private, community, or public) bound together by data and application portability.

[0157] Generally, cloud computing models enable some of the responsibilities that might have previously been provided by an organization's own IT departments to be delivered as service layers within the cloud environment for use by consumers (inside or outside the organization, depending on the public / private nature of the cloud). Depending on the specific implementation, the precise definition of the components or features provided by or within each cloud service layer can vary, but common examples include: Software as a Service (SaaS), where consumers use software applications running on cloud infrastructure while the SaaS provider manages or controls the underlying cloud infrastructure and applications; Platform as a Service (PaaS), where consumers can use software programming languages ​​and development tools supported by the PaaS provider to develop, deploy, and otherwise control their own applications while the PaaS provider manages or controls other aspects of the cloud environment (i.e., everything in the runtime execution environment); and Infrastructure as a Service (IaaS), where consumers can deploy and run arbitrary software applications and / or provide processes, storage devices, networks, and other basic computing resources while the IaaS provider manages or controls the underlying physical cloud infrastructure (i.e., everything below the operating system layer). Database as a Service (DBaaS) is a service in which consumers use database servers or database management systems running on cloud infrastructure, while the DbaaS provider manages or controls the underlying cloud infrastructure, applications, and servers, including one or more database servers.

[0158] In the foregoing description, embodiments of the invention have been described with reference to numerous specific details, which may vary depending on implementation. Therefore, the description and drawings should be considered illustrative rather than restrictive. The unique and exclusive indication of the scope of the invention, and the content that the applicant intends to define as the scope of the invention, is the literal and equivalent scope of the set of claims published from this application in the specific form of such claims, including any subsequent corrections.

Claims

1. A computer-executed method, comprising: The first process executes the first sub-operation of a specific key-value pair operation through the following steps: Send a Remote Direct Memory Access (RDMA) request to obtain header information from a hash table in the computing device's volatile memory for a specific entry of a particular key-value pair. Each entry in one or more entries of the hash table includes header information, a key for storing key-value pairs in each entry, and a value for storing key-value pairs in each entry; The header information includes a set of flags, which includes one or more of the following: An assigned flag indicates whether the specific entry is occupied. An injection flag indicates whether an incomplete write operation to a specific entry is in progress. A time stamp indicates whether data in a specific entry is being migrated, and A fence marker indicates whether a second process, other than the first, has priority in inserting a specific key-value pair into the hash table. One or more flags in the set of flags are set; and Based on the first sub-operation of the specific key-value pair operation, if it is determined that one or more flags of the specific key-value pair are set, the first process suspends the second sub-operation of the specific key-value pair operation. The method is performed by one or more computing devices.

2. The method of claim 1, wherein: The specific key-value pair includes a specific value; The specific key-value pair operation is a lookup operation that retrieves the specific value from a specific entry in the hash table. The one or more flags include time flags; as well as The method further includes, after a second sub-operation of suspending the specific key-value pair operation and after a predetermined amount of time after one or more flags in the set of flags determined have been set, the first process performs a second lookup operation to retrieve the specific value.

3. The method of claim 2, wherein the first process performs a second lookup operation to retrieve the specific value from the hash table.

4. The method of claim 2, wherein: Hash tables are the first type of hash tables, and hash tables implement key-value storage. The method also includes determining that the key-value store is migrating from a first hash table to a second hash table; The first process is at least in part based on the determination that the key-value store is migrating from the first hash table to the second hash table, and performs a second lookup operation to retrieve the specific value on the first hash table and the second hash table.

5. The method of claim 1, wherein: The specific key-value pair operation is an insertion operation that inserts a specific key-value pair into a specific entry in the hash table; The one or more signs include fence signs; The method further includes, after a second sub-operation of suspending the specific key-value pair operation, sending a second RDMA request to cancel setting one or more flag bits in the header information of the specific entry.

6. The method of claim 1, wherein: The specific key-value pair includes a specific value; The specific key-value pair operation is a lookup operation that retrieves the specific value from a specific entry in the hash table. The one or more flags include injection flags; The method further includes, after a second sub-operation of suspending the specific key-value pair operation and after a predetermined amount of time after one or more flags in the set of flags determined have been set, the first process performs a second lookup operation to retrieve the specific value.

7. The method of claim 1, wherein: RDMA requests are Compare and Exchange (CAS) type RDMA requests; A CAS-type RDMA request compares the header information from a specific entry with the target header information corresponding to one or more unset flag bits. The set of flags to determine one or more flags is set to include: CAS type RDMA request return information indicating that the header information from a specific entry does not match the target header information.

8. A computer-executed method, comprising: Determine the information to be migrated from the first hash table allocated in the first part of the volatile memory; In response to determining that information from the first hash table needs to be migrated: Send one or more messages to one or more clients, which are configured to use RDMA to request access to the key-value store at a first hash table. The one or more messages include information identifying a second hash table allocated in a second portion of volatile memory that is different from the first portion of the volatile memory; Relocate multiple key-value pairs in the key-value repository from multiple entries in the first hash table to multiple entries in the second hash table; While relocating the plurality of key-value pairs from the first hash table to the second hash table, both the first hash table and the second hash table are maintained. While maintaining both the first hash table and the second hash table, the following requests are served: One or more first RDMA requests to the first hash table from the one or more clients, and One or more second RDMA requests to the second hash table from the one or more clients; as well as After relocating the plurality of key-value pairs from the first hash table to the second hash table, the first hash table is released; The method is performed by one or more computing devices.

9. The method of claim 8, wherein relocating the plurality of key-value pairs from the first hash table to the second hash table comprises relocating a specific key-value pair of the plurality of key-value pairs from a first entry in the first hash table to a second entry in the second hash table by: Set a timestamp in the header information of the first entry used in the first hash table; and After setting the time stamp, the information of the specific key-value pair is written into the second entry of the second hash table.

10. The method of claim 8, wherein, During the relocation of the plurality of key-value pairs from the first hash table to the second hash table, at least in part by sending a first RDMA request of the one or more first RDMA requests to the first hash table and a second RDMA request of the one or more second RDMA requests to the second hash table, a particular client of the one or more clients performs a lookup operation on a particular key-value pair of the plurality of key-value pairs from the key-value repository.

11. The method of claim 8, further comprising, at least in part, during the relocation of the plurality of key-value pairs from the first hash table to the second hash table, a particular client among the one or more clients performs an insertion operation to insert a particular key-value pair from the plurality of key-value pairs into the key-value repository by sending a third RDMA request to write information of a particular key-value pair into the second hash table.

12. A nontransitory computer-readable medium storing one or more instructions, which, when executed by one or more processors, cause to perform the method as described in any one of claims 1-11.

13. A system including a processor, comprising: One or more processors; as well as A non-transitory computer-readable medium storing one or more instructions, which, when executed by the one or more processors, cause to perform the method as described in any one of claims 1-11.

14. A computing system comprising components for performing the method as described in any one of claims 1-11.

15. A computer program product comprising instructions that, when executed by one or more processors, cause to perform the method as described in any one of claims 1-11.

Images