System, method and computer-accessible medium for a lock free replicated state machine

Abstract

A replicated state machine can include several servers that logically process upserts (updates, inserts, or deletes) in the same order. Searches can go to any server. This exemplary design can provide high fault tolerance to server failure. Implemented in a classical way, a replicated state machine requires a single emitter of operations, atomic broadcast to send these operations in the same order everywhere, and sequential processing of these operations at every server. There are performance and fault tolerance issues with such a design: the emitter is a single point of failure and the sequential execution is a waste on modern capable machine. Parallel designs entail a coordination mechanism or locking[Esc+19]. Exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure provide a highly parallel transaction design without a single point of failure, and with the same guarantees as a replicated state machine.

Description

ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION SYSTEM, METHOD AND COMPUTER-ACCESSIBLE MEDIUM FOR A LOCK FREE REPLICATED STATE MACHINE CROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application relates to and claims priority from U.S. Patent Application No. 63 / 669,768, filed on July 11, 2024, the entire disclosure of which is incorporated herein by reference. FIELD OF THE DISCLOSURE

[0002] The present disclosure relates to systems, methods and computer-accessible medium for highly parallel transaction designs without a single point of failure, and with the same guarantees as replicated state machines. BACKGROUND INFORMATION

[0003] A database stores key-value pairs, where a key is an identifier (e.g., an employee ID), and the value is a set of values about that key (e.g., age, income, position in the company). Servers store the data on both volatile and non-volatile storage. Volatile storage (e.g., DRAM) may lose its values when power fails. Non-volatile storage does not.

[0004] A transaction is a set of upserts (e.g., inserts, deletes, and updates) of key-value pairs as well as searches that are either partially or totally ordered by the control flow of the transaction. A search on a key k reads the value associated with a key-value pair whose key value is k.

[0005] The program defining a transaction is called its transaction text. The transaction text for a relational database system might be a series of SQL statements with their associated control flow. However, the transaction text can be written in any language that the servers can either interpret or compile and execute.

[0006] A replicated state machine can include several servers that logically process upserts (updates, inserts, or deletes) in the same order. Searches can go to any server. This design can provide high fault tolerance to server failure. Implemented in a classical way, a replicated stateATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION machine requires a single emitter of operations, atomic broadcast to send these operations in the same order everywhere, and sequential processing of these operations at every server. There are, however, performance and fault tolerance issues with such a design: the emitter is a single point of failure and the sequential execution is a waste on a modern capable machine. Parallel designs entail a coordination mechanism or locking[Esc+19].

[0007] Thus, it may be beneficial to at least partially address the deficiencies existing in conventional designs, and to provide exemplary systems, methods and computer-accessible mediums, which can provide a highly parallel transaction design without a single point of failure, but with the same guarantees as a replicated state machine. SUMMARY OF EXEMPLARY EMBODIMENTS

[0008] To that end, exemplary systems, methods and computer-accessible mediums according to the exemplary embodiments of the present disclosure can be provided to make distinctions between transaction texts and transaction executions, because servers according to the exemplary embodiments of the present disclosure can store transaction texts first and then, when appropriate, execute those transaction texts resulting in transaction executions.

[0009] Exemplary systems, methods, and non-transitory computer accessible medium according to certain exemplary embodiments of the present disclosure can include at least two servers that store data in memory and one or more agents configured to send at least one transaction text to the at least two servers. These agents can be synchronized within a time difference short enough to guarantee transaction linearizability and inter-server consistency by, e.g.: (i) creating and sending, by each agent for each time point k of a schedule of increasing time points, a transaction package comprising all transactions received by the agent between the previous (k-1st) time point and the kth time point of the agent’s schedule, (ii) storing, by a first execution thread at each of the at least two servers, the transaction package from each agent sent at each time point k onto a first data structure (D1),ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION (iii) organizing by one or more second execution threads, the transactions in the transaction packages based on at least one precedence relationship between the transactions, and (iv) accessing, by a third execution thread, a transaction text S from the first data structure (D1) and, upon completion of all transactions preceding S, executing S by (1) interpreting its text, (2) adding at least one modification to a second data structure (D2), or (3) executing at least one search on D2.

[0010] Furthermore, in some exemplary embodiments of the present disclosure, each of the servers can be configured to recover from its failure by reading the package array (data structure D1) of any working server up to a time point j, executing those transactions, and then accept the agent packages for the time point immediately following j. In some exemplary embodiments of the present disclosure, time points can be or include integer values provided by a global counter. In addition, e.g., precedence relationships can be represented by a directed acyclic graph each of whose edges denotes a required precedence relationship, and this directed acyclic graph in a multicopy timestamp setting can require only a subset of the edges that exist in a single copy setting given the same arrival order of the same transactions.

[0011] In some exemplary embodiments of the present disclosure, the concurrent execution of two transactions T1 and T2 can be facilitated even if a multicopy timestamp conflict from T1 to T2 is present by means of executing T2 first. Moreover, the at least one execution thread that organizes the transactions in the packages into a directed graph whose edges denote conflicts also assigns timestamps to transactions such that (i) no two transactions have the same timestamp (ii) the timestamps are consistent with the edges of the directed graph in the sense that if T1 has an edge to T2 then T1 will have a lesser timestamp than T2.

[0012] In certain exemplary embodiments of the present disclosure, each of the servers has the ability to inspect the non-volatile memory of at least one other said server. Further, the server memory can be non-volatile, volatile, or a combination of the two. In addition, the brokers and servers can synchronize their clocks based on a beacon, and the data structure D1 can be an array. In other exemplary embodiments of the present disclosure, the data structure D2 can contain upserts with timestamps. In some exemplary embodiments of the present disclosure, the execution of a transaction and putting its upserts on data structure D2 can be non-blocking.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION

[0013] Furthermore, in some exemplary embodiments of the present disclosure, data structure D2 can use a tree data structure for fast retrieval. In addition, each non-root node of data structure D2 can have a timestamp of the maximum upsert in that node. In some exemplary embodiments of the present disclosure, a server can recover from failure by copying the package array (D1) from at least one server. Additionally, each upsert entered into data structure D2 can be given the timestamp of the transaction text S. In exemplary embodiments of the present disclosure, each search on some key k of the transaction S can return the value associated with the upsert on k whose timestamp is the greatest timestamp less than or equal to the timestamp of S, if such an upsert is present.

[0014] In some exemplary embodiments of the present disclosure, a transaction T1 can execute when all transactions having lesser timestamps than T1 and that conflict with T1 have completed, and two transactions may conflict if and only if they both reference the same data item and at least one of them changes the item. Furthermore, two transactions might conflict if and only if the one with the earlier timestamp upserts a data item that a transaction with a later timestamp reads. Additionally in some exemplary embodiments of the present disclosure, the second data structure D2 can be a list and / or a tree. In some exemplary embodiments of the present disclosure, each non-root node of the tree can have a timestamp equal to the maximum timestamp of all upserts in that node.

[0015] Additionally, in some exemplary embodiments of the present disclosure, a search on key k can avoid searching a child node whose maximum timestamp is less than the timestamp of an upsert on k that has already been found by the search. Further, each non-root node of the tree can have at least two timestamps equal to the maximum timestamp of different partitions of the upserts in that node. Moreover, a search on key k can avoid searching a child node whose maximum timestamp of the partition including k is less than the timestamp of an upsert on k that has already been found by the search.

[0016] These and other objects, features and advantages of the exemplary embodiments of the present disclosure will become apparent upon reading the following detailed description of the exemplary embodiments of the present disclosure, when taken in conjunction with the appended claims.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Further objects, features and advantages of the present disclosure will become apparent from the following detailed description taken in conjunction with the accompanying Figures showing illustrative embodiments of the present disclosure, in which:

[0018] Figure 1 is an exemplary flow chart illustrating a process in accordance with certain exemplary embodiments of the present disclosure;

[0019] Figure 2 is an exemplary illustration of two servers receiving broker messages from various brokers (e.g., specific types of agents) and the order those broker messages were received through time in accordance with certain exemplary embodiments of the present disclosure;

[0020] Figure 3(A) is an exemplary flow chart illustrating a process for coordinator actions in the case of broker message failure in accordance with certain exemplary embodiments of the present disclosure;

[0021] Figure 3(B) is an exemplary flow chart illustrating a process for server actions in the case of broker message failure in accordance with certain exemplary embodiments of the present disclosure;

[0022] Figure 4 is an exemplary illustration of a system structure for an upsert database tree in accordance with certain exemplary embodiments of the present disclosure;

[0023] Figure 5 is an exemplary flow chart illustrating a process for non-blocking prepending in a root node of an upsert database list or an upsert database tree in accordance with certain exemplary embodiments of the present disclosure;

[0024] Figure 6 is an exemplary illustration of a system structure for the top two levels of an upsert database tree in accordance with certain exemplary embodiments of the present disclosure;

[0025] Figure 7 is an exemplary flow chart illustrating a process for searching in an upsert database tree in accordance with certain exemplary embodiments of the present disclosure; and

[0026] Figure 8 is an illustration of an exemplary block diagram of an exemplary system in accordance with certain exemplary embodiments of the present disclosure;

[0027] Throughout the drawings, the same reference numerals and characters, unless otherwise stated, are used to denote like features, elements, components or portions of theATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION illustrated embodiments. Moreover, while the present disclosure will now be described in detail with reference to the figures, it is done so in connection with the illustrative embodiments and is not limited by the particular embodiments illustrated in the figures and the appended claims. DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0028] The following description of exemplary embodiments provides non-limiting representative examples referencing numerals to particularly describe features and teachings of different aspects of the present disclosure. The exemplary embodiments described should be recognized as capable of implementation separately, or in combination, with other exemplary embodiments from the description of the exemplary embodiments. A person of ordinary skill in the art reviewing the description of the exemplary embodiments should be able to learn and understand the different described aspects of the present disclosure. The description of the exemplary embodiments should facilitate understanding of the exemplary embodiments of the present disclosure to such an extent that other implementations, not specifically covered but within the knowledge of a person of skill in the art having read the description of embodiments, would be understood to be consistent with an application of the exemplary embodiments of the present disclosure.

[0029] It can be desirable for transactional databases to have the following exemplary properties: ^ Concurrency: For efficiency reasons, transactions should be able to execute concurrently (many of them should be able to overlap in time). In that way, while one transaction waits for slow memory or a network message, for example, another transaction can proceed. ^ Inter-server Consistent Linearizability: Though they execute concurrently, transactions should appear semantically to execute in an inter-server consistent, order preserving serial order. That is, (i) the concurrent execution should return the same search values as they would in some serial order O of the transactions; (ii) the final state of the database should be the same as it would be given serial order O; (iii) serial order O should have the property that if transaction T1 completes before transaction T2 begins, then T1 should appear in the serial order before T2. Systems having these three properties are sometimes called linearizable and that terminology is adopted herein. The inter-server consistencyATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION requirement is that the equivalent order-preserving serial order O1 on one server should be a prefix of the serial order O2 on every other server or O2 should be a prefix of O1. ^ Hardware fault tolerance: There should be no single hardware point of failure. Thus, if a server fails, no data will be lost and processing should continue with the same semantics (searches should return the same values and upserts should write the same values) as if no failure had occurred. ^ Non-blocking: Execution thread (whether process or process thread) failures should not cause other threads to lose access to any data or to be indefinitely delayed.

[0030] An exemplary system model and a failure model should both be described when asserting that a system is fault tolerant. ^ Non-interactive Deterministic Transactions: Transactions in the system likely may not include any user interactions. Further, they should be deterministic (e.g., no use of randomness). ^ Agents: An agent can be either a client or a broker or any other entity that interacts with a server. An exemplary system includes clients (see, e.g., Figure 1 showing exemplary clients 110, 120, and 130) that originate transactions. A given client will send zero or more transaction texts, each transaction text should go to a single broker (see, e.g., Figure 1 showing exemplary brokers 140 and 150. The ellipses of Figure 1 represent the possibility of more clients.). Each broker collects transaction texts and then sends packages of newly arrived transaction texts periodically (e.g. every millisecond) to all servers (see, e.g., Figure 1 showing exemplary servers 160 and 170). The servers execute and store transactions. In the exemplary systems, methods and computer-accessible medium according to the exemplary embodiments of the present disclosure, only servers assign timestamps to transaction texts, although other exemplary variations are possible. Any client or broker that sends (possibly through a relay) packages to a server is called an Agent. Agents can be or include a logical designation, although they can physically reside on servers or on separate computational nodes. ^ Time Synchronization: Agents and servers can be time synchronized within a very small difference (e.g. a few tens of microseconds). One exemplary way to achieve this is to useATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION a time beacon that broadcasts the time to brokers and servers with sufficient frequency that their clocks remain all within a few microseconds of one another. The U.S. National Institute of Standards broadcasts such exemplary signal. Other methods of synchronization involving agents and servers communicating with one another are also possible. ^ Remote Read: Any server can read the non-volatile memory of any other server. This can be achieved through remote disk access for disks and through Compute Express Link or some other remote memory access technology for non-volatile RAM. ^ Server Status: Any server can determine the failure status of another server: whether the server is up, down, or recovering. The failure model can facilitate or allow message loss and clean non-traitorous processor failures. ^ Message Loss from Brokers: Messages can be lost or delayed. ^ Non-traitorous clean failures: When a server fails, it will fail by stopping or slowing down, but the failure will not cause a modification to its non-volatile storage.

[0031] To make the overall protocol for exemplary systems, methods and computer- accessible medium according to exemplary embodiments of the present disclosure clear, first, a case can be considered in which every server receives all packages for every interval from all brokers. Exemplary Failure-Free Protocol

[0032] In the absence of message drops and broker failures, the exemplary protocol for the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can be as follows: 1. A client, such as elements 110, 120, and 130 of Figure 1, sends a transaction text for transaction S to a broker b. 2. In a similar way to the Calvin system of Thomson et al (see, e.g., Ref.5), at the end of each millisecond m, b (See, e.g., brokers 140 and / or 150 of Figure 1) packages all transactionsATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION that have arrived between m − 1 and m and sends them to all servers, e.g., 160 and 170 of Figure 1 (Note that the ellipses of Figure 1 represent the possibility of more servers.). This package from b for millisecond m is denoted pack(m, b). If b has received no transactions between m−1 and m, the broker nevertheless sends pack(m, b) but with an indication that it contains no transactions. The use of a millisecond interval can be merely for the sake of presenting a concrete example. In general, the interval between package sends is called the broker package interval. The length of the broker package interval can be any time that is significantly greater than the inconsistency of the broker clocks. The broker package interval can vary over time (e.g., sometimes a millisecond and sometimes 10 milliseconds), but the brokers and servers should know the schedule of interval times (e.g., 1 pm, 1 pm + 3 millisecond, 1 pm + 5 milliseconds, 1 pm + 6 milliseconds, ...). Other variants are possible (e.g., different brokers may use different schedules for sending packages). 3. Each server s has a single port through which it receives packages. When packages arrive, a thread prepends these packages onto an array, called the package array (See, e.g., Figure 2: 210, 220), also called D1 above. As evident in Figure 2, when the broker messages for each millisecond m arrive before the broker messages from m+1, the package array on each server 210 and 220 has the packages from the brokers in millisecond order, though the order of the brokers’ packages can differ from one millisecond to another on a given server as shown in each box Figure 2. Further the order among the brokers can differ for the same millisecond on the package arrays from different servers, as shown by comparing the boxes for server 1 vs. server 2.

[0033] It should be appreciated that the motivation for having each broker send a single package per millisecond (or, in general, per broker package interval) is to establish a fact denoted the No Laggards Condition as follows: If a server s has received the packages from all brokers for all periods up to time t, then s has received all transactions that have arrived before t. By contrast, if individual transaction texts were sent by brokers and one such transaction text were lost to some servers but not to others, then some servers might not receive the same transaction texts as others. Exemplary Alternative Exemplary Embodiments of the No Laggards ConditionsATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION

[0034] According to the exemplary systems, methods and computer-accessible medium of the present disclosure, as an alternative, it is possible for each broker to be known to have some schedule of sending packages. Different brokers can follow different schedules. For example, suppose there are three brokers A, B, C and broker A sends its packages on odd milliseconds and brokers B and C on even milliseconds. The No Laggard condition holds unchanged.

[0035] What does change is that the server s may have received information about transactions from times greater than some given time t as well. Consider the example setting of even and odd millisecond brokers. At millisecond 10, if the server has received messages from brokers B and C at 10 milliseconds as well as a message from broker A at 9 milliseconds, then the server can be sure that there will be no more transactions from before 9 milliseconds, but the server also has received some transactions from after 9 milliseconds..

[0036] According to the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, it is possible to use a global counter. Suppose that when a transaction text is about to be sent to a server, the transaction text is given the value of some current global counter. Such an embodiment would achieve a slight variant of the No Laggards Condition: If a server has received one transaction text for each counter value j below i, then there are no other transaction texts for any such counter value k < i. In that case, the counter serves as a timestamp. Exemplary Missing Package Recovery

[0037] In this fault tolerant protocol used with the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, each server can make use of two non-volatile data structures: (i) Answer which is a map from milliseconds and brokers to either packages or strings; and (ii) BrokerStatus which contains the "up" and "down" behavior of any broker.

[0038] When a broker is introduced to the system, its status is "up". Note that once a broker is declared down, it can never recover. If the machine on which the broker executes returns to execution, it will be given a new identity.

[0039] If server s does not receive package(m, b) from broker b soon after the end of broker package interval m, but BrokerStatus[b] is "up" for server s, then s engages in the following two phase protocol as reflected in step 310 of Figure 3(A) :ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION 1. Server s sets its non-volatile map Answer[m, b] to "not received" at step 312 and then at step 315 asks all other working servers (See, e.g., Figure 1: other server boxes) whether they have received pack(m, b). Once a server s′ receives this request at step 358 in Figure 3(B), s′ checks its non-volatile map Answer at location m, b at step 375. ^ If Answer[m, b] is defined at step 377 (meaning it has already been assigned a value), then s′ responds with the value in Answer[m, b] at step 385. ^ Otherwise, (i) if s′ has received pack(m, b) at or before the time of the request at step 376, then s′ sets Answer[m, b] := pack(m, b) at step 380; (ii) if s′ has not received pack(m, b) at step 378, then s′ sets Answer[m, b] := "not received" at step 390; Next s′ responds to s with the value of Answer[m, b] at step 385. 2. At step 355, s′ checks what s′ has received. If s′ receives pack(m, b) from b at step 359 after Answer[m, b] at step 370 is already defined or is “not received” at step 372, s′ simply drops pack(m, b) at step 395. If undefined at step 374, s′ adds pack(m, b) to the package array at step 398. On the other hand, if s′ receives pack(m, b) from another server with the instruction "b is ok, pack(m, b)" at step 357, and the current value of Answer[m, b] is "not received" or Answer[m, b] is undefined, then s′ sets Answer[m, b] := pack(m, b) and puts pack(m, b) on its package array at step 365. 3. If coordinating server s receives pack(m, b) from another server at step 325 as a response to its status request (yes at step 328), then s sends "b is ok, pack(m, b)" to all servers at step 330. In addition, s inspects the Answer map and package array of any failed server s′′ that might have been up at millisecond m − 1 or later at step 345. (For example, in this exemplary and non-limiting case, it can be assumed for the sake of concreteness, the broker package interval is one millisecond.) If s finds pack(m, b) either in the package array of s′′ or in Answer[m, b] of s′′, then s sends "b is ok, pack(m, b)" to all servers at step 330. Upon receiving that message, a server s′′′ will set Answer[m, b] to pack(m, b) at step 365. If s never finds pack(m, b), e.g., receives a no at step 332 and all unfailed servers return "not received" at steps 335, 340, 342, 345, and 348, then s sends a message "b is down, ignore pack(m, b)" to all servers at step 350. A receiving server s′′′ will receive the message at step 356 and then set Answer[m, b] to "ignore"ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION and BrokerStatus[b] to "down" (when a broker is introduced, its status is set to "up") at step 360. Servers will thereafter ignore pack(m, b) and any future messages from b at step 350.

[0040] The purpose of the above exemplary protocol according to the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure is to establish the following facts: Inter-server guarantee: Either all up servers will place pack(m, b) on their package arrays or none will. Non-blocking guarantee: Neither a single server failure nor multiple message loss leads to blocking. Exemplary Server Failure Recovery

[0041] When a server s physically recovers from failure, in the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, s can read the state of some other working server s′ and all uncompleted transactions from the package array of s′ up to the previous millisecond (or, in general, the previous broker package interval), say n, and then begins to add new packages starting at n + 1. Exemplary Executing Transactions

[0042] Executing a transaction on a server consists of reading the transaction text from the package array and then executing that text. Executing the text will entail generating searches on keys and upserts.

[0043] To achieve high transaction throughput within a server, it can help to execute transactions concurrently. Further, to avoid blocking, it can help to avoid all use of locks. Exemplary Consistency and Concurrency

[0044] Consistency is an important feature of replicated state machines.

[0045] Timestamp-respecting serializability is the guarantee that the effect of transaction execution on a server is equivalent to a serial execution of all transactions received by that server in timestamp order. That is, given a set of transactions received by the server, the final state of executing those transactions should be equivalent to the final state of executing those transactions in timestamp order and any search sQof some transaction Q will return the same value as sQ in that serial execution.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION

[0046] A sufficient condition to ensure inter-server consistency as well as linearizability consists of two parts: (i) ensure that every server receives the same transactions having the same timestamps and (ii) timestamp-respecting serializability.

[0047] Having brokers send packages at every millisecond (or, in general, at every broker package interval) along with the lost message recovery protocol ensures (i). This section shows how to ensure (ii).

[0048] The exemplary methods below assign an execution order and a timestamp based on some notion of conflict. The definition of conflict can depend on how the system stores and accesses data. ^ Single copy model: Suppose there is at most one copy of each key k whose associated value may change as updates arrive. In that case, transactions S and S′ are said to single copy conflict if both access the same key k′ and at least one of those transactions upserts k′. ^ Multicopy timestamp model: Suppose by contrast that each upsert to some key k′ is associated with a value and the timestamp of the transaction that originated the upsert. Thus, each upsert is represented as a triple. For example, to perform a delete of k′ at timestamp t, the upsert consists of the triple (k′,NULL, t) where the NULL indicates deletion and the t is the timestamp. An update or insert for a transaction with timestamp t has the format (k′, v′, t) where the v′ is the new value in the case of the update or the inserted value in the case of an insert.

[0049] In this exemplary timestamp model according to the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, a search on k′ from a transaction S having timestamp TS(S) can look for an upsert on k having the greatest timestamp less than or equal to TS(S). In such a setting, transaction S’ multicopy timestamp conflicts with S only if TS(S′) < TS(S) and S’ issues an upsert on a key k that S searches. Thus, S can begin as soon as all transactions that multicopy conflict with S complete.

[0050] A transaction can be complete (and can be marked as complete on the package array) when all upserts from that transaction have entered the upsert database list, preferably on nonvolatile memory. Once all transactions having a lesser timestamp than TS(S) and that conflict with S complete, S may execute.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION

[0051] Thus, combining the multicopy and timestamp features can reduce the number of conflicts to just those that multicopy-timestamp-conflict. Exemplary Ordering Transactions and Assigning Timestamps

[0052] How transactions should be ordered depends on whether exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure deal with single copy conflicts or multicopy conflicts. In what follows, it can be assumed that each transaction that arrives in a package for millisecond m (where a millisecond is the broker package interval) has a unique identifier (perhaps the broker identifier concatenated with an identifier assigned by the broker to that transaction).

[0053] In the case where some group of transactions must execute in a certain order, (e.g., they come from a client that insists that they execute in that order) the transactions in that group will have directed edges consistent with the order in which they must execute at the start of the two algorithms below.

[0054] Further, the following exemplary procedures up to the creation of the graph to be described below called TGraph can be performed independently and in a non-blocking way by different threads. Suppose there is a pointer PtrTGraph to the TGraph structure. It is initially set to null. The first thread that succeeds may then perform a compare and set on PtrTGraph to make it point to the TGraph that it creates. When a transaction is completed, it can be marked as completed on TGraph, so modifications of TGraph can be made non-blocking. Exemplary Single Copy Directionality

[0055] Exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can be utilized to characterize the set of all transactions that arrive as a result of millisecond m (where a millisecond is the broker package interval) as a graph called $G1$ where each node represents a transaction and there are undirected edges between two nodes if the corresponding transactions single copy conflict. Exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can represent such a conflict by Ti ←→ Tj.

[0056] An exemplary procedure of exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can convert the double- headed arrows to single-headed arrows. A node is pure if the only edges touching it areATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION undirected (double-headed arrows). A node is resolved if all edges touching it are directed either going towards the node or away from it. Exemplary Single Copy Directionality Algorithm: 1. While there are still pure nodes, (i) Find the pure node n that has the most edges touching it (denoted neighbor(n)) and such that if some other pure node n′ has |neighbor(n′)| equal to |neighbor(n)|, then the identifier of n is less than the identifier of n′. (ii) Direct all edges touching n to point to the nodes in neighbor(n). End While 2. While there are still unresolved nodes, (i) Find the unresolved node n that has the most neighbors connect to n by double- headed edges (denoted undirected(n)) and such that if some other node n′ has |undirected(n′)| equal to |undirected(n)|, then the identifier of n is less than the identifier of n′. (ii) Direct all double-headed edges touching n to point to the nodes in undirected(n). End while 3. When the while loop is done, all nodes are resolved and therefore the graph is a directed graph which we will denote TGraph. 4. Execute the transactions in accordance with TGraph. That is, execute transaction T when there are no edges directed towards T. When T completes, T and all the edges leaving T should be removed from the graph.

[0057] Example: Starting with T1 ←→ T2 ←→ T3 ←→ T4. All nodes are pure. T2 has both the largest set of neighbors and the lowest identifier, so we redirect that edges to be T1 ← T2→ T3 ←→ T4. T4 is still pure, so we redirect the edges to T1 ← T2 → T3 ← T4. Thus, T2 and T4 may execute in parallel. When T2 completes, T1 may execute. When T2 and T4 complete, T3 may execute.

[0058] Lemma: Fact: TGraph is acyclic. Exemplary Multicopy Directionality

[0059] The exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can again be utilized to characterize the set ofATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION all transactions that arrive as a result of millisecond m (in the case that the broker package interval is a millisecond) as a graph where each node represents a transaction. However, in this case, the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can create a loop arrow from T1 to T2 if T1 would conflict with T2 when T1 preceded T2 (that is, T1 has an upsert on some key k that T2 searches). Exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can represent this by Ti LOOP→ Tj. Note that if T2 precedes T1, then there may be no conflict because T2 will read from an upsert of some other transactions. The server also has a global counter value called counter which will act as a timestamp. By global, the intended meaning is that counter increases monotonically over time including when new packages arrive.

[0060] The exemplary procedure of the exemplary systems, methods and computer- accessible medium according to exemplary embodiments of the present disclosure can be utilized to convert loop arrows to single-headed arrows. The set of all nodes that have loop arrows pointing towards a given node n are denoted loopincoming(n). Similarly, the set of all nodes that have loop arrows pointing away from n are denoted loopoutgoing(n). A node is said to be ideal if loopoutgoing(n) is empty. As in the single copy case, a node n is resolved if all edges touching it are directed either going towards the node (→ n) or away from it (n →) and none are loop edges. Exemplary Multicopy Directionality Algorithm: 1. While there are still ideal nodes, (i) Find the ideal node n that has the largest loopincoming(n) set and such that if some other ideal node n′ has |loopincoming(n′)| equal to |loopincoming(n)|, then the identifier of n is less than the identifier of n′. (ii) Discard all edges from loopincoming(n). (iii) Record TS(n) := counter; on package array (iv) counter := counter + 1 End while 2. While there are still unresolved nodes, (i) Find the unresolved node n that has the most loop edges pointing towards it (denoted loopincoming(n)) and such that if some other unresolved node n′ hasATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION |loopincoming(n′)| equal to |loopincoming(n)|, then the identifier of n is less than the identifier of n′. (ii) Create outgoing normal edges (→) from n to the nodes in loopoutgoing(n) only (because those will be the only conflicts). (iii) Drop all loop edges into or out of n. (iv) Record TS(n) := counter on package array (v) counter := counter + 1 End while 3. When the while loop is done, all nodes are resolved and therefore the graph is a directed graph which, by analogy to the single copy case, we will denote TGraph. 4. Execute the transactions in accordance with TGraph. That is, execute transaction T when there are no edges directed towards T. When T completes, T and all the edges leaving T should be removed from the graph. The execution will be done in a multicopy fashion, using counter as timestamp as explained below in the section Exemplary Non-blocking Execution in the Multicopy Timestamp Model.

[0061] Example 1: The transactions from millisecond m consists of just four transactions with the following relationships: T1 LOOP→ T2, T3 LOOP→ T4 and T3 LOOP→ T2. This could arise, for example, if T1 and T3 both had upserts and T2 searched on the key or keys of those upserts. Suppose the counter value at this point is 152. For example, the only ideal node is T2 because it has incoming loop edges. So we give T2 a timestamp of 152 and increase the counter to 153. Thus, no edges can be added, because there are no conflicts when T2 runs before T1 and T3. Now T4 is ideal, so it receives counter value 154 as its timestamp. T1 is now ideal and receives a timestamp of 155 and finally T3 receives 156. So all can now execute in parallel. Because of the timestamps, they do not conflict.

[0062] Example 2: If T1 upserts k and T2 searches on k while T2 upserts on k′ and T1 search on k′, then there will be two edges: T1 LOOP→ T2 T2 LOOP→ T1

[0063] In that exemplary case, there will be no ideal nodes. Thus, the Multicopy Directionality Algorithm / Procedure can replace T1 LOOP→ T2 by T1 → T2 and drop T2ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION LOOP→ T1. So, T1 would get a lower timestamp than T2 and T1 would have to complete before T2 could begin.

[0064] Fact: TGraph will be acyclic. Exemplary Non-blocking Execution in the Multicopy Timestamp Model

[0065] The multicopy timestamp model of exemplary systems, methods and computer- accessible medium according to exemplary embodiments of the present disclosure can also enable non-blocking execution. Suppose that all the upserts that have so far occurred reside in a preferably non-volatile storage called the upsert database. Two principal forms of an upsert database are the upsert database list and the upsert database tree. For example, Figure 6 shows the top two levels of a database upsert tree. The partition function f can map each key k to a single node n such that either k belongs to the partition associated with n. For example, f(k) may yield a value associated with n (if n is a hash function) or f(k) determines that k lies within a range of key values associated with n. Each of the lower nodes here may have children nodes which may have children nodes, and so on. In this figure, each non-root node has a single maximum timestamp, which is the maximum timestamp of any upsert in that node.

[0066] In the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, the root node of any upsert database can be a linked list to enable prepending operations to be non-blocking.

[0067] When transaction S is ready to execute, each of its upserts is prepended to a list (see, e.g., Figure 4: Pointer P 410, and blocks 1, 2, and n (430, 440, and 450 respectively)). To insert one or more upserts upgroup into the upsert database, a thread can put upgroup into a small segment s of free memory at step 510 of Figure 5, thereby resulting in New Block s 420, and then copy the location in P into the next pointer of s at step 520, i.e. s.next := P. At this point P points to Block 1. Next, the thread performs a compare and set operation (which is non-blocking) having the semantics "if the value of P equals s.next, then set P to point to s" at step 530. If it succeeds at step 540, then the pointer at P 410 can point to the new block 420 (and the arrow with label “After prepending”) containing s and return “done” at step 550. On the other hand, the compare and set done by t may not satisfy its if condition (because another thread has modified P), so thread t or another thread may have to put upgroup into the upsert database list at step 520.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION If thread t stops while executing at any point, no other thread will be blocked. That is what establishes the pre-pending algorithm to be non-blocking.

[0068] In the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, a search on key k for an executing transaction S having timestamp (the counter value in the multicopy case) t works as follows on the upsert database list. The search first consults upserts already performed by transaction S on k and returns the result of the most recent such upsert. If there is no such upsert from transaction S, then the search returns the value of the last upsert on k of a transaction S’ where the timestamp of S’ is the one having the maximum timestamp of any transaction whose timestamp is less than t and that does an upsert on k. If there is no such transaction S’ and k is not upserted by S, then the search on k returns an indication that k is not in the structure.

[0069] Because the upsert database list is implemented as a single list, a search for a key k that is absent from the list could require traversing the entire list (e.g., the entire database of upserts). Second, because the multicopy temporal approach allows or allows upserts with earlier timestamps to be inserted after upserts with later timestamps, a search for a key that is present may have to search a large portion of the list.

[0070] For that reason, in the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure may not use an upsert database list. Instead, the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can use the upsert database tree illustrated in Figure 6, having (i) a root node e.g., 610, implemented as a linked list, where the most recent upserts are placed, (ii) children nodes e.g., 620, 630, and 640 whose key-value pairs are partitioned, and (iii) descendant nodes (not shown but well known in the state of the art, as for example in B+-trees) that further partition the space of keys.

[0071] One example of a partitioning method is range partitioning in which keys from some minimum key r1 up to and less than r2 are placed in the partition of one child node, then those from r2 up to and less than r3 in the partition of another child node, and so on. An alternative is hash partitioning in which key k is placed in child node i if h(k) = i where h is a function that is designed to partition keys more or less evenly. Such partitioned tree structures are well known in the state of the art of modern LSM-trees. A tree structure reduces the worst case effort to searchATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION for a key k from being proportional to the size of the database (which is the total number of upserts that are present) to approximately the logarithm of that size.

[0072] In the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, one maintenance thread can partition key- values from a node n as follows: read a tail portion of n, partition the key-value-timestamp triples based on the partitioning function on the keys, and write each partition i to the appropriate existing child of n or to a newly created child, then delete that tail portion from n. There can be a time period during which the same upsert is in both n and a child of n. This does not change the return value of any search, because any search will return the same value whether it reads the upsert once or several times.

[0073] Further improvements can include the use of Bloom filters and other filtering data structures which are well known in the state of the art.

[0074] A further exemplary optimization according to the exemplary embodiments of the present disclosure can be provided which are associated with ordering the nodes based on timestamps. To understand this issue, consider a search having timestamp 11:00 AM for key k may find that the upsert on k with the most recent timestamp in the root node has timestamp 10:30 AM, but it might be that in the child node associated with k there is another upsert on k having timestamp 10:40 AM and even further down the upsert database tree, there might be an upsert on k having timestamp 10:50 AM.

[0075] For this reason, each non-root node, e.g., 620, 630, and 640, used with the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure can be associated with a timestamp value that is the maximum timestamp of any upsert in that node. Further, the maintenance threads that create child nodes can ensure that any child node of node n can have a lesser maximum timestamp than the maximum timestamp of n. In the exemplary systems, methods and computer-accessible medium according to exemplary embodiments of the present disclosure, this can be called the timestamp ordering condition.

[0076] This node-level timestamp information can help in the example above as follows: suppose the search having timestamp 11:00 AM on k finds an upsert on k having timestamp 10:30 AM in the root node. If the child node associated with k has a maximum timestamp ofATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION 10:15 AM, the search need not look further, because any descendant node must have an earlier maximum timestamp on any upsert for k.

[0077] In general, a search(k) for a transaction having timestamp t, as illustrated in step 710 of Figure 7, finds all upserts on k in the root node U(t) having timestamps less than or equal to t as illustrated in step 720. Let t′ be the maximum time of any upsert in (U(t)). If t′ is greater than the maximum of any upsert of the child node n’ that is responsible for k at step 730, then return the upsert associated with t′ from U(t) at step 740. Otherwise proceed to n’ and if not null, then add more upserts to U(t) in step 720. If n’ is null at step 750, then the search should return an indication that no upsert on k preceded this search at step 760.

[0078] For example, the timestamp per child node scheme can be generalized so that different partitions of the keyspace of a node could have separate maximum timestamps. For example if a node n contains a key range from 2000 to 4999, there can be one maximum timestamp for keys in the 2000s, a different one for keys in the 3000s, and a third one for the keys in the 4000s. Children of n may need to preserve the timestamp ordering condition for each of these ranges. Exemplary Related Work

[0079] The history of the observation that a transaction T2 can execute as soon as all transactions T1 that conflict with T2 and that should logically precede T2 complete, going back in some form to Ref.1 for operations and for transactions in Ref.2. Escobar et al. (see, e.g., Ref. 8) makes the process of managing the directed graph of the precedence relationships among transactions lock-free. The execution of transactions might still however require locks.

[0080] Guo et al. (see, e.g., Ref.6) propose a system where one server executes transactions in some order and that order is exported to other servers which then follow that order. The primary thus achieves high concurrency, but the other servers may not. Also, until the transactions are executed on those other servers, there is limited fault tolerance.

[0081] Kapritos et al. (see, e.g., Ref.4) present a system called Eve that has a set of machines execute a set of transactions concurrently. If they agree on their state after the execution, then they go on. Otherwise they roll back. This implies that return values cannot be sent to an application until this agreement is achieved.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION

[0082] Kemme et al. (see, e.g., Ref.3) propose a relaxation of atomic broadcast that works well on a local area network because such networks have the property that messages are usually ordered in the same way at every server. By being “optimistic,” the system executes transactions as soon as they arrive but does not commit them until it is confirmed that all servers have received transactions in the same order. Because this will most often be the case on local area networks, the system works well.

[0083] Marandi and Padone (see, e.g., Ref.7) propose an algorithm that is optimistic in a different sense: it allows operations to execute concurrently even if it is not known whether they conflict. If they do conflict, then the algorithm backs up and executes them serially.

[0084] Thomson et al (see, e.g., Ref.5) describe a system called Calvin which batches transactions in time chunks at separate “sequencers” where each sequencer has the same role as a broker described here. The transactions are then interleaved in a round robin way at each server and that is the execution order. When a sequencer fails, there is an agreement protocol to determine how to recover from the failure. Calvin uses locking to execute transactions.

[0085] Figure 8 shows a block diagram of an exemplary embodiment of a system according to the present disclosure, which can be utilized either in part or completely with any one or more of the exemplary embodiments of the present disclosure. For example, exemplary procedures in accordance with the present disclosure described herein can be performed by a processing arrangement and / or a computing arrangement 802. Such processing / computing arrangement 802 can be, for example entirely or a part of, or include, but not limited to, a computer / processor 804 that can include, for example one or more microprocessors, and use instructions stored on a computer-accessible medium (e.g., RAM, ROM, hard drive, or other storage device).

[0086] As shown in Figure 8, for example a computer-accessible medium 806 (e.g., as described herein above, a storage device such as a hard disk, floppy disk, memory stick, CD- ROM, RAM, ROM, etc., or a collection thereof) can be provided (e.g., in communication with the processing arrangement 802). The computer-accessible medium 806 can contain executable instructions 808 thereon. In addition, or alternatively, a storage arrangement 810 can be provided separately from the computer-accessible medium 806, which can provide the instructions to the processing arrangement 802 so as to configure the processing arrangement to execute certain exemplary procedures, processes and methods, as described herein above, forATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION example. Further, the exemplary processing arrangement 802 can be provided with or include an input / output arrangement 814, which can include, for example a wired network, a wireless network, the internet, an intranet, a data collection probe, a sensor, etc. Figure 8 shows that the exemplary processing arrangement 802 can be in communication with an exemplary display arrangement 812, which, according to certain exemplary embodiments of the present disclosure, can be a touch-screen configured for inputting information to the processing arrangement in addition to outputting information from the processing arrangement, for example. Further, the exemplary display 812 and / or a storage arrangement 810 can be used to display and / or store data in a user-accessible format and / or user-readable format.

[0087] According to exemplary embodiments of the present disclosure, numerous specific details have been set forth. It is to be understood, however, that implementations of the disclosed technology can be practiced without these specific details. In other instances, well- known methods, structures and techniques have not been shown in detail in order not to obscure an understanding of this description. References to “some examples,” “other examples,” “one example,” “an example,” “various examples,” “one embodiment,” “an embodiment,” “some embodiments,” “example embodiment,” “various embodiments,” “one implementation,” “an implementation,” “example implementation,” “various implementations,” “some implementations,” etc., indicate that the implementation(s) of the disclosed technology so described may include a particular feature, structure, or characteristic, but not every implementation necessarily includes the particular feature, structure, or characteristic. Further, repeated use of the phrases “in one example,” “in one exemplary embodiment,” or “in one implementation” does not necessarily refer to the same example, exemplary embodiment, or implementation, although it may.

[0088] As used herein, unless otherwise specified the use of the ordinal adjectives “first,” “second,” “third,” etc., to describe a common object, merely indicate that different instances of like objects are being referred to, and are not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in ranking, or in any other manner.

[0089] While certain implementations of the disclosed technology have been described in connection with what is presently considered to be the most practical and various implementations, it is to be understood that the disclosed technology is not to be limited to theATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION disclosed implementations, but on the contrary, is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.

[0090] The foregoing merely illustrates the principles of the disclosure. Various modifications and alterations to the described embodiments will be apparent to those skilled in the art in view of the teachings herein. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, arrangements, and procedures which, although not explicitly shown or described herein, embody the principles of the disclosure and can be thus within the spirit and scope of the disclosure. Various different exemplary embodiments can be used together with one another, as well as interchangeably therewith, as should be understood by those having ordinary skill in the art. In addition, certain terms used in the present disclosure, including the specification and drawings, can be used synonymously in certain instances, including, but not limited to, for example, data and information. It should be understood that, while these words, and / or other words that can be synonymous to one another, can be used synonymously herein, that there can be instances when such words can be intended to not be used synonymously. Further, to the extent that the prior art knowledge has not been explicitly incorporated by reference herein above, it is explicitly incorporated herein in its entirety. All publications referenced are incorporated herein by reference in their entireties.

[0091] Throughout the disclosure, the following terms take at least the meanings explicitly associated herein, unless the context clearly dictates otherwise. The term “or” is intended to mean an inclusive “or.” Further, the terms “a,” “an,” and “the” are intended to mean one or more unless specified otherwise or clear from the context to be directed to a singular form.

[0092] This written description uses examples to disclose certain implementations of the disclosed technology, including the best mode, and also to enable any person skilled in the art to practice certain implementations of the disclosed technology, including making and using any devices or systems and performing any incorporated methods.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION EXEMPLARY REFERENCES

[0093] The following references are hereby incorporated by reference, in their entireties: 1. Fred B. Schneider. “Implementing fault-tolerant services using the state machine approach: a tutorial”. In: ACM Comput. Surv.22.4 (Dec.1990), pp.299–319. issn: 0360- 0300. doi: 10.1145 / 98163.98167. url: https: / / doi.org / 10.1145 / 98163.98167. 2. Arthur Whitney, Dennis Shasha, and Steve Apter. “High volume transaction processing without currency control, two phase commit, SQLor C++”. English (US). In: Seventh international workshop on high performance transaction systems, September 1997, Asimolar, California.1997, pp.211–217. 3. F. Kemme et al. “Using optimistic atomic broadcast in transaction processing systems”. In: IEEE Transactions on Knowledge and Data Engineering, vol.15, no.4, pp.1018- 1032, July-Aug.2003 (2003). doi: 10.1109 / TKDE.2003.1209016. 4. Manos Kapritsos et al. “All about Eve: execute-verify replication for multicore servers”. In: Proceedings of the 10th USENIX Conference on Operating Systems Design and Implementation. OSDI’12. Hollywood, CA, USA: USENIX Association, 2012, pp.237– 250. isbn: 9781931971966. 5. Alexander Thomson et al. “Calvin: fast distributed transactions for partitioned database systems”. In: Proceedings of the 2012 ACM SIGMOD International Conference on Management of Data. SIGMOD ’12. Scottsdale, Arizona, USA: Association for Computing Machinery, 2012, pp.1–12. isbn: 9781450312479 doi: 10.1145 / 2213836.2213838. url: https: / / doi.org / 10.1145 / 2213836.2213838. 6. Zhenyu Guo et al. “Rex: replication at the speed of multi-core”. In: Proceedings of the Ninth European Conference on Computer Systems. EuroSys ’14. Amsterdam, The Netherlands: Association for Computing Machinery, 2014. isbn: 9781450327046. doi: 10.1145 / 2592798.2592800. url: https: / / doi.org / 10.1145 / 2592798.2592800. 7. Parisa Jalili Marandi and Fernando Pedone. “Optimistic Parallel State-Machine Replication”. In: 2014 IEEE 33rd International Symposium on Reliable Distributed Systems.2014, pp.57–66. doi: 10.1109 / SRDS.2014.25. 8. Ian Aragon Escobar et al. “Boosting concurrency in Parallel State Machine Replication”. In: Proceedings of the 20th International Middleware Conference. Middleware ’19.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION Davis, CA, USA: Association for Computing Machinery, 2019, pp.228–240. isbn: 9781450370097. doi: 10.1145 / 3361525.3361549. url: https: / / doi.org / 10.1145 / 3361525.3361549

Claims

ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION WHAT IS CLAIMED IS:

1. A system comprising: at least two servers that store data in memory; a plurality of agents configured to send at least one transaction text wherein the plurality of agents and at least two servers are synchronized within a time difference short enough to guarantee transaction linearizability and inter-server consistency through the use of the steps comprising: (i) following a schedule of increasing time points, for each time point k, an agent creates a package of zero or more transactions including the transactions received by the agent between the previous (k-1st) time point and the kth time point of that agent’s schedule and sends that package; (ii) at each server, an execution thread stores the package of the transaction texts from each agent sent at each specified time point onto a first data structure D1; (iii) one or more execution threads organizes the transactions in the packages in a way that recognizes precedence relationships between the transactions; and (iv) an execution thread accesses a transaction text S from the first data structure D1 and, when all transactions that should precede S complete, said thread executes S by interpreting its text, adding modifications such as upserts to a second data structure D2 and executing searches on the second data structure D2 as necessary.

2. The system of claim 1 wherein each of said servers is able to recover from its failure by reading the first data structure D1 of any working server up to a time point j, executing those transactions, and then accept the agent packages for the time point immediately following the time point j.

3. The system of claim 1 wherein said time points are integer values provided by a global counter.

4. The system of claim 1 wherein said precedence relationships are represented by a directed acyclic graph each of whose edges denotes a required precedence relationship.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION 5. The system of claim 4 wherein the directed acyclic graph in the multicopy timestamp setting requires only a subset of the edges that exist in the single copy setting given the same arrival order of the same transactions.

6. The system of claim 1 wherein the concurrent execution of two transactions T1 and T2 is made possible even if the transaction T1 performs an upsert on a key k and the transaction T2 searches on the key k by means of assigning an earlier timestamp to the transaction T2.

7. The system of claim 1 wherein the at least one execution thread that assigns timestamps to transactions preferentially assigns an earlier timestamp to a transaction T2 and a later timestamp to a transaction T1 if the transaction T1 performs an upsert on a key k and the transaction T2 searches on the key k.

8. The system of claim 1 wherein each of said servers is able to inspect the non-volatile memory of at least one other said server.

9. The system of claim 1 wherein the server memory may be non-volatile, volatile, or a combination of the two.

10. The system of claim 1 wherein the brokers and servers synchronize their clocks based on a beacon.

11. The system of claim 1 wherein the first data structure D1 is an array.

12. The system of claim 1 wherein the second data structure D2 contains upserts with timestamps.

13. The system of claim 1 wherein executing a transaction and putting its upserts on the second data structure D2 is non-blocking.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION 14. The system of claim 1 wherein the second data structure D2 uses a tree data structure for fast retrieval.

15. The system of claim 14 wherein the partitioning of each level of the tree is based on at least one of a hash method or a range method.

16. The system of claim 1 wherein a server recovers from failure by copying the first data structure D1 from at least one server.

17. The system of claim 1 wherein each said upsert entered into the second data structure D2 is assigned the timestamp of said transaction text S.

18. The system of claim 17 wherein each search on a key k of said transaction S returns the value associated with the upsert on the key k whose timestamp is the greatest timestamp before or equal to the timestamp of S, if such an upsert is present.

19. The system of claim 1 wherein a transaction T1 is allowed to execute when all transactions having lesser timestamps than the transaction T1 and that conflict with the transaction T1 have completed.

20. The system of claim 19 wherein two transactions conflict if and only if they both reference the same data item and at least one of them changes the item.

21. The system of claim 19 wherein two transactions conflict if and only if the one with the earlier timestamp upserts a data item that a transaction with a later timestamp searches 22. The system of claim 1 in which the second data structure D2 is a list.

23. The system of claim 1 in which the second data structure D2 is a tree.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION 24. The system of claim 23 in which each non-root node of the tree has a timestamp equal to the maximum timestamp of all upserts in that node.

25. The system of claim 24 where a search on a key k avoids searching a child node whose maximum timestamp is less than the timestamp of an upsert on the key k that has already been found by the search.

26. The system of claim 23 in which each non-root node of the tree has at least two timestamps equal to the maximum timestamp of different partitions of the upserts in that node.

27. The system of claim 26 where a search on key k avoids searching a child node whose maximum timestamp of the partition including the key k is less than the timestamp of an upsert on the key k that has already been found by the search.

28. A method to ensure that a server can verify all transactions having a timestamp before a certain time are present at said server, the method comprising the steps of: (i) Synchronizing agents and servers within a small time error; (ii) Specifying a set of time intervals that are known to agents and servers; (iii) With each agent A, at the end of time t2, sending a package of transactions that A has received between times t1 and t2, wherein t1 is the end of the previous interval; and (iv) with the server, verifying that it has received all packages from all agents at all intervals up to time t2.

29. The method of claim 28 in which the servers can ensure that either all working servers receive the package of said agent A at t2 and put said package in its first data structure D1 or no server has put that package in its first data structure D1.

30. The method of claim 29 in which a server s that has not yet received said package executes a protocol to determine whether any server has received said package.ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION 31. The method of claim 30 in which server s establishes whether any other server has received said package. If any other server has, then s ensures that other servers receive said package as well.

32. A system comprising: at least two servers that store data in memory; a plurality of agents configured to send at least one transaction text to the at least two servers; wherein the plurality of agents and at least two servers are synchronized within a time difference short enough to guarantee transaction linearizability and inter-server consistency through the procedures comprising: (i) generating and sending, by each agent for each time point k of a schedule of increasing time points, a transaction package comprising all transactions received by the agent between the previous (k-1st) time point and the kth time point of the agent’s schedule; (ii) storing, by a first execution thread at each of the at least two servers, the transaction package from each agent sent at each time point k onto a first data structure (D1); (iii) organizing, by one or more second execution threads, the transactions in the transaction packages based on at least one precedence relationship between the transactions; and (iv) accessing, by a third execution thread, a transaction text S from the first data structure (D1) and, upon completion of all transactions preceding S, executing S by (a) interpreting its text, (b) adding at least one modification to a second data structure (D2), or (3) executing at least one search on the second data structure D2.

33. A method for synchronizing a plurality of agents and at least two servers within a time difference short enough to guarantee transaction linearizability and inter-server consistency, comprising: generating and sending, by each agent for each time point k of a schedule of increasing time points, a transaction package comprising all transactions received by the agent between the previous (k-1st) time point and the kth time point of the agent’s schedule;ATTORNEY REF.300719.WO.01-109197.0000183 PATENT APPLICATION storing, by a first execution thread at each of the at least two servers, the transaction package from each agent sent at each time point k onto a first data structure (D1); organizing, by one or more second execution threads, the transactions in the transaction packages based on at least one precedence relationship between the transactions; and accessing, by a third execution thread, a transaction text S from the first data structure (D1) and, upon completion of all transactions preceding S, executing S by (1) interpreting its text, (2) adding at least one modification to a second data structure (D2), or (3) executing at least one search on the second data structure D2.

34. A non-transitory computer accessible medium which includes software thereon for synchronizing a plurality of agents and at least two servers within a time difference short enough to guarantee transaction linearizability and inter-server consistency, wherein, when at least one computer processor executes the software, the computer processor is configured to perform the procedures, comprising: generating and sending, by each agent for each time point k of a schedule of increasing time points, a transaction package comprising all transactions received by the agent between the previous (k-1st) time point and the kth time point of the agent’s schedule; storing, by a first execution thread at each of the at least two servers, the transaction package from each agent sent at each time point k onto a first data structure (D1); organizing, by one or more second execution threads, the transactions in the transaction packages based on at least one precedence relationship between the transactions; and accessing, by a third execution thread, a transaction text S from the first data structure (D1) and, upon completion of all transactions preceding S, executing S by (1) interpreting its text, (2) adding at least one modification to a second data structure (D2), or (3) executing at least one search on the second data structure D2.

Images