Membership services for microsecond applications

Abstract

A data processing apparatus and method for implementing a cloud computing microsecond level group membership service, the data processing apparatus comprising one or more processors configured to: detect one or more system failures of a first plurality of members; a current system configuration comprising a representation of a plurality of members of an activity is periodically determined by configuring the one or more processors to: determine whether there is a new system configuration to be employed according to a consensus of the plurality of members of the activity; if a consensus exists among the plurality of active members, updating the current system configuration into the new system configuration; or, if the multiple members have no consensus, maintaining the current system configuration, or updating the current system configuration to be a temporary system configuration based on the current system configuration.

Description

Technical Field

[0001] The present invention relates to a device and method for implementing a microsecond group membership service in cloud computing. Background Art

[0002] Modern datacenter structures make possible microsecond distributed applications such as data stores and message queues. One challenging aspect of developing these applications is to ensure that in addition to being fast in common situations, these applications can also react quickly to changes in their membership, such as due to reconfiguration and failures. Membership is a system configuration that includes an indication of the active and therefore available system components. This is particularly important because these applications that manage this configuration (membership) form the backbone of many cloud-enabled services such as analytical and transactional systems that try to meet increasingly stringent tail latency requirements. Since microservices-oriented architectures are the industry standard for building cloud services, a single user request to perform an operation is often translated into a command that is distributed to many microservices on the cloud computing network to satisfy the user's single request. This leads to extensive fan-out of microservice interactions on the critical path. In other words, many distributed microservices will interact for a single request in order to optimally satisfy the request. The result is irreconcilable: reconfiguration and failure events, which are traditionally infrequent, are exacerbated by communication fan-out. This means that since user requests are often subject to this fan-out, the tail latency of the service will be quickly affected, causing the service to slow down. The tail of a cloud service refers to the latency of the slowest request and is therefore the maximum latency limit experienced by the end user. Despite significant efforts in both hardware and hardware-accelerated software, keeping the tail short in large-scale configurations is one of the most important challenges in the cloud computing industry.

[0003] Existing systems cannot handle system failures within a few microseconds, so the latency of the system increases. Key-value stores such as Hermes, state machine replication systems such as Mu and Hovercraft, and transaction systems such as FaRM process requests within a few microseconds in a fault-free scenario, but cannot reach microsecond time periods in the face of failures. For example, in the event of a system failure, Mu and HoverCraft applications take 0.5ms and 10ms to failover, respectively. Hermes has a failover time of 150ms, while ZooKeeper, mentioned by FaRM, is a widely used distributed coordination service that provides millisecond-level failover at best when a system failure occurs during membership management.

[0004] Therefore, currently used applications and methods cannot provide a combination of microsecond membership services without microsecond failure detection, microsecond consensus, and lease-based microsecond active membership queries. The disadvantages of applications or methods that provide each of microsecond-free failure detection, microsecond consensus, and lease-based microsecond active membership queries have the following impacts: without fast failure detection, it is impossible to achieve microsecond accuracy without compromising accuracy, such as without generating many false positives (failures); without achieving microsecond consensus, it is impossible to achieve membership consistency at the microsecond level; and finally, without a lease-based microsecond active membership algorithm, it is impossible to obtain a microsecond-level view of membership, which can impose high overhead on the client due to the increased time required to perform the task.

[0005] Therefore, the present invention is designed to solve the above-mentioned problems by providing a microsecond membership service that detects changes in application membership and allows applications to fail over in microseconds. Summary of the invention

[0006] A data processing device for implementing a microsecond group membership service for cloud computing, the data processing device comprising one or more processors, the one or more processors being used to: detect one or more system failures of a first plurality of members; periodically determine a current system configuration including representations of a plurality of active members by configuring the one or more processors to perform the following operations: determine whether there is a new system configuration to be adopted based on the consensus of the plurality of active members; if there is a consensus among the plurality of active members, update the current system configuration to the new system configuration; or, if there is no consensus among the plurality of members, maintain the current system configuration, or update the current system configuration to a temporary system configuration based on the current system configuration. This improves cloud tolerance and microsecond consistency in cloud services, and also provides strict tail latency to ensure smoothness and scalability of user interactions. The plurality of members may be all members of the system or a subset of the members of the system.

[0007] According to the data processing device described above, the one or more system faults include user space faults, which provides a method for avoiding false positives in system fault detection.

[0008] According to the data processing device described in the above device, the one or more processors are used to detect the user space failure by detecting the failure notification sent from the Linux kernel when the system process fails. This can realize the automatic transfer of control to the kernel when a system failure occurs.

[0009] According to the data processing device described above, the one or more system failures include a kernel failure. This provides a method for detecting a driver failure in a system.

[0010] According to the data processing device described above, the one or more processors are used to detect the kernel failure by determining that the heartbeat counter local to the first or second plurality of members is not incremented. This provides a method to reduce kernel jitter and user space processes sharing the capacity of the one or more processors.

[0011] The data processing apparatus according to the above apparatus, wherein the one or more processors are used to select a leader from the plurality of members based on atomically comparing and exchanging a proposed leader from the plurality of members with a current leader from the plurality of members. This allows changing the leader among the members in a short time.

[0012] According to the data processing device described above, the current system configuration is updated to the new system configuration only after the time period of the current system configuration expires. This ensures that multiple system configurations are not implemented at the same time.

[0013] According to the data processing device described above, the temporary system configuration is maintained until the multiple active members reach a consensus on the new system configuration. This ensures consistency between the system configurations maintained by the multiple members until most members agree on the new system configuration.

[0014] According to the data processing device described above, the one or more processors are further configured to send a fault notification signal to the multiple members other than the members where the fault occurs in response to one or more system faults detected by the multiple members. This provides a communication channel between the members, allowing the members to update their system configurations to delete the members that are no longer active, for example, the members where the fault occurs.

[0015] According to the data processing device of the above device, the current and / or new system configuration expires after a predetermined period of time, which provides a limit for the update delay of the system configuration.

[0016] According to the data processing device described above, the temporary system configuration is the same as the current system configuration, but has a different time period duration, which ensures that the system configuration remains consistent so that the system can work normally before the members decide on a new system configuration.

[0017] According to the data processing device described in the above device, the data processing device includes one or more caches for storing the current system configuration and / or temporary system configuration of most members of the plurality of members. This reduces the time for members to call the system configuration status and reduces the amount of communication around the system required to obtain the system configuration.

[0018] A data processing method for implementing a microsecond group membership service for cloud computing, the data processing method comprising: detecting one or more system failures of a first plurality of members; determining a current system configuration including representations of the plurality of active members by performing the following method steps: determining whether there is a new system configuration to be adopted based on the consensus of the plurality of active members; if there is a consensus among the plurality of active members, updating the current system configuration to the new system configuration; or, if there is no consensus among the plurality of members, maintaining the current system configuration, or updating the current system configuration to a temporary system configuration based on the current system configuration. This improves cloud tolerance and microsecond consistency in cloud services, and also provides strict tail latency to ensure smoothness and scalability of user interactions.

[0019] According to the data processing method described above, wherein the one or more system faults include user space faults, this provides a method for avoiding false positives in system fault detection.

[0020] According to the data processing method described in the above method, the method further includes detecting the user space failure by detecting a failure notification sent from the Linux kernel when the system process fails. This can automatically transfer control to the kernel when a system failure occurs.

[0021] According to the data processing method described above, wherein the one or more system failures include a kernel failure, this provides a method for detecting a driver failure in a system.

[0022] According to the data processing method described above, the method further comprises detecting the kernel failure by determining that the heartbeat counters local to the plurality of members are not incremented. This provides a method to reduce kernel jitter and user space processes sharing the capacity of the one or more processors.

[0023] The data processing method according to the above method, wherein the method further comprises selecting a leader from the plurality of members based on atomically comparing and exchanging the proposed leader from the plurality of members with the current leaders from the plurality of members. This allows the leader among the members to be changed in a short time.

[0024] According to the data processing method described above, the current system configuration is updated to the new system configuration only after the time period of the current system configuration expires. This ensures that multiple system configurations are not implemented at the same time.

[0025] According to the data processing method described in the above method, the method also includes maintaining the temporary system configuration until the multiple active members reach a consensus on the new system configuration. This ensures the consistency between the system configurations maintained by the multiple members until most members agree on the new system configuration.

[0026] According to the data processing method described in the above method, the method further includes sending a fault notification signal to the plurality of members other than the members where the fault occurs in response to one or more system faults detected by the plurality of members. This provides a communication channel between the members, allowing the members to update their system configurations to delete members that are no longer active, for example, members where a fault occurs.

[0027] According to the data processing method described above, the current and / or new system configuration expires after a predetermined period of time, which provides a limit for the update delay of the system configuration.

[0028] According to the data processing method described above, the temporary system configuration is the same as the current system configuration, but has a different time period duration, which ensures that the system configuration remains consistent so that the system can work normally before the members decide on a new system configuration.

[0029] According to the data processing method described in the above method, the method further includes storing the current system configuration and / or temporary system configuration of most members among the plurality of members in one or more caches. This reduces the time for members to call the system configuration state and reduces the amount of communication around the system required to obtain the system configuration.

[0030] According to the data processing method described above, one or more system failures of the first plurality of members are detected in microseconds, which ensures that the system failure will not affect the system operation by quickly detecting and updating the system configuration. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The present invention will now be described by way of example with reference to the accompanying drawings. In the accompanying drawings:

[0032] Figure 1 shows an exemplary sequence of views and how they are updated in the event of a system failure;

[0033] Figure 2 An example of a view showing a member mounted in the event of a system failure;

[0034] Figure 3 An example of a remote direct memory access configuration is shown;

[0035] Figure 4 Another example of a view mounted on a member in the case of a system failure is shown, and the state of Algorithm 2 in each case;

[0036] Figure 5a An example of the membership service uKharon of the present invention is shown;

[0037] Figure 5b An exemplary consensus engine of the present invention is shown;

[0038] Figure 6 Shows the RPC mode of Paxos and a modified version of Paxos that uses CAS-based RPC;

[0039] Figure 7 An activity membership algorithm implemented by the data processing and method of the present invention is shown;

[0040] Figure 8a An example showing normal activity membership;

[0041] Figure 8b An example showing active membership for a lease;

[0042] Fig. 9 An example of a cache implementation method in the system of the present invention is shown;

[0043] Fig.10 A comparison of the membership service of the present invention implemented by HERD with other use cases is shown. DETAILED DESCRIPTION

[0044] Therefore, based on the observation that a key step in making tail-tolerant microsecond applications is to react quickly to failures, the present invention proposes a data processing apparatus and method. The key to achieving microsecond latency is the careful design of three basic components, all of which operate at microsecond levels using Remote Direct Memory Access (RDMA) or similar technologies such as SmartNIC. RDMA is used only as an example in the following description and should be understood as an alternative to similar technologies such as SmartNIC, etc.

[0045] First, the microsecond membership service of the present apparatus and method achieves microsecond fault detection by adopting a multi-level fault detector. This is achieved by distinguishing between application-related faults (e.g., segmentation faults), kernel-related faults (e.g., driver faults), and hardware-related faults (e.g., RDMANIC faults). For each type of fault, a different fault detector can be used. Second, our mechanism uses a consensus engine to determine membership, which, in the example given below, relies only on one-sided RDMA verbs. This engine uses RDMA Compare-and-Swap (CAS) to handle leader changes within 10μs. Third, the apparatus and method of the present invention provide membership leases that add minimal overhead to the end application and last for ~20μs. Therefore, the membership service described in this article combines typical opposing forces: enabling applications to have low-overhead dynamics in fault-free scenarios and extremely fast failover in the event of system failures. Each of these aspects will be described in detail later in the present invention.

[0046] Membership Services

[0047] To achieve resilience, long-lived distributed systems must be dynamic. Many systems achieve dynamism by relying on coordination frameworks (substrate) such as ZooKeeper or etcd. Among the various services provided by these frameworks (e.g. atomic locks, registers), dynamism is fundamentally addressed through their membership service. The membership service provides dynamism both in graceful execution and in failure. In the former case, the membership service serves join and leave requests from processes that want to become part of a distributed application or exit a distributed application. In the latter case, the membership service detects process failures and reacts. All of these events are reflected through new configurations (called views or simply membership). Essentially, the membership service serves as a store of configuration information, tracks how the set of processes evolves, and exposes this information to every component of the system.

[0048] Typically, membership services rely on consensus of members to establish a completely ordered sequence of views. This does not have to be all members, but may be a subset of members (coordinating nodes), and not all nodes need to be running together. These services, including Zookeeper and etcd, provide strong semantics because all processes using the membership service transition through the same sequence of views. Typically, membership services rely on consensus to establish a completely ordered sequence of views. These services, including Zookeeper and etcd, provide strong semantics because all processes using the membership service transition through the same sequence of views. Consensus-based membership services also provide real-time semantics. In addition to understanding the membership sequence, it is also important to understand which is the (single) active membership (system configuration). To understand why this real-time property is useful, consider the following example of a cache storage that is incorrectly built solely on the membership sequence: The cache serves READ and WRITE requests. Initially, the membership M1={S1} specifies that server S1 is responsible for the cache (i.e., S1 stores the cache and serves requests). Eventually, a second membership M2={S2} replaces S1 with S2. S2 continues to serve the client's requests as part of M2 and updates the cached contents. Meanwhile, S1 is unaware of M2 and continues to serve the client's requests. Therefore, clients that are also unaware of M2 and read from S1 will get stale data. This example demonstrates a violation of consistency. The example shows that the total order of membership does not, by itself, provide any real-time guarantees. Figure 1 It shows that when the system member π i An example of changing the view / membership / system configuration of this system when it crashes, where i is the member indicator. Figure 1 As shown, in the Figure 1 In the example, four members are active, and then member 2 crashes. Therefore, the system view is updated to reflect this, as shown in the figure. Figure 2 This new view is distributed to each new member and each process in the system, so the configuration of the system is known. Figure 1 In the example, member 3 crashes, so the view is updated again to reflect the two remaining members 1 and 4. The goal is to ensure that the view sequence is the same for all members / processes so that each member / process knows the system configuration and the active members that remain. It is important to maintain monotonicity so that only one view is active at a time so that members / processes of the system are not confused. This can be illustrated by the expression that process p installs view Vj = (j, Mj) after installing Vi = (i, Mi) and then j>i and Where M is the membership, V is the view i, and j is the view number. If two processes install views V = (i, M) and V' = (i, M'), then M = M'. If a process installs view Vi = (i, Mi) and (Not equal to) Mi, then q has collapsed.

[0049] Figure 2 Use the timeline to further illustrate Figure 1 The view of the installation is shown in . Figure 2 , you can see that the view is updated periodically to reflect the Figure 1 and 2 Between and Figure 2 and 3 The crash occurred between members 2 and 3. Figure 2 It is also shown that a process can install an old view, in this case, process (member) 4 installs an old view in which the crash of member 3 is not reflected, but this is later removed when installing the view. Figure 3 Corrected in time. Figure 4 The same is shown in the results of the activity algorithm described later in the present invention. Figure 4 As shown, when the outdated view is installed in member 4, the active algorithm returns "false" until the correct view is installed.

[0050] The examples described in this invention use Remote Direct Memory Access (RDMA), a network technology that allows a process to access the memory of a remote computer without involving the remote computer's CPU. By implementing several layers of the network stack in hardware and relying on kernel bypass, RDMA enables microsecond machine-to-machine communication. Examples of this can be found at Figure 3 As seen in . This example allows applications within a data center to communicate at speeds as low as 0.9μs. This technology is supported by different structures, such as InfiniBand and commodity Ethernet via RoCE. Applications rely on primitives called verbs to communicate through RDMA. There are single-sided verbs, including READ, WRITE, and Compare and Swap (CAS) verbs, as well as double-sided verbs, such as SEND and RECV verbs. Single-sided verbs enable processes to read, write, and apply atomic transformations to the memory of a remote machine without involving its CPU. This reduces the latency of the communication system. Double-sided verbs are similar to message passing, involving two communication sides of each other. Double-sided verbs enable processes to send and receive memory buffers. Communication in RDMA can obviously occur on an established reliable connection (Reliable Connection, RC) or an unreliable datagram (UnreliableDatagram, UD). The former provides FIFO semantics, and the latter trades reliability for better performance and support for message multicast. The membership service of the present invention is designed for data centers, is safe under asynchronous conditions, and is effective under partial synchronization.

[0051] The apparatus and method for implementing the microsecond membership service of the present invention may include three components: (1) microsecond fault detection, (2) microsecond consensus engine, and (3) microsecond active membership method, which checks / verifies whether the membership is still valid / active. These issues are described in detail below. Throughout the disclosure, the terms membership, view, and system configuration are used interchangeably. The same is true for coordinator, process, and member.

[0052] As part of the present invention, a data processing device for implementing a microsecond group membership service in cloud computing is provided. The data processing device includes one or more processors for performing the following functions.

[0053] In a first instance, a data processing device can be used to implement multi-level fault detection by detecting one or more systematic faults of multiple members. A practical fault detector aims to be as complete and accurate as possible. A complete and accurate fault detector is one that detects all faults without false positives. Being complete but inaccurate can also cause problems because false positives can trigger new memberships, which require further actions by the distributed application (e.g., rebalancing data between nodes). This is inefficient and can overburden the service. Typically, the operation of a fault detector relies on a timeout. But timeouts are difficult to set correctly: if the timeout is too low, the fault detector may experience instability (e.g., oscillatory behavior). This explains why most systems set the timeout to a sufficiently high safety value. At the microsecond level, this problem is amplified because small execution delays (e.g., kernel jitter) can take several microseconds.

[0054] However, the data processing device of the present invention follows a pragmatic approach by avoiding timeouts when possible. To achieve this goal, several levels of failures are considered. These failures are: (1) user space failures that cause the application implemented on the device to terminate (e.g., segmentation fault, out of memory error, uncaught exception); (2) kernel failures that hinder the execution of the application (e.g., kernel loop caused by core mounted in the kernel, driver crash); (3) catastrophic failures that prevent communication with the application host (e.g., power failure, RDMANIC failure); (4) Byzantine failures that affect the application state (e.g., stack overflow, variable core). The technology implemented by the data processing device according to the present invention is used to address two of these levels: (1) user space failures and (2) kernel failures. One or more processors of the data processing device are used to detect one or more system failures of multiple members, or in other words, detect one or more user space and / or kernel failures of one or more members of a membership service such as a microservice.

[0055] Once one or more system failures are detected, the one or more processors are further configured to send a failure notification signal to a plurality of members other than the member in which the failure occurred in response to the one or more system failures detected by the plurality of members. In other words, other members of the plurality of members that are still active are notified that a system failure has occurred in a member that may not be currently active.

[0056] One or more processors may detect user space faults in the following manner. User space faults are handled by the Linux kernel. An application registers with the kernel to enable deadbeat, which is a fault notification broadcast by the kernel when a process terminates. This registration is achieved by a prctl system call that is called by the application early in execution. Here, application is another term used to describe the membership service implemented by the data processing device. The system call includes a node for the system identifier and modifies the process descriptor (task_struct in Linux) using a flag that the kernel checks during the process cleanup routine. In Linux, when a process crashes, control is transferred to the kernel, and the kernel begins executing the process cleanup routine. If the flag is set, the kernel broadcasts a fault notification including the specified identifier. To implement this functionality, the prctl system call is extended and the process cleanup routine that is part of the kernel exit system call is modified. The task of broadcasting crash notifications may be delegated to a kernel module that may be included in the data processing device of the present invention. This module may broadcast crash notifications using a kernel space RDMA driver, which are polled by all instances of the membership service. Since this fault detector for user space faults does not use a timeout, false positives may be prevented from occurring. The device may implement such a kernel module to broadcast user space failures, or the kernel module may be implemented by one or more processors. This "module" relies on the kernel space RDMA driver to broadcast crash notifications when the system fails.

[0057] One or more processors of the data processing device can also be used to detect system failures in the form of kernel failures. In order to detect kernel-induced failures, consider the way RDMA is handled in user space. One or more processors can be used to register memory to an RDMA device by issuing an input / output control system call on a file descriptor. By design, the Linux kernel destroys the file descriptor, thereby disabling remote access to the memory at the end of the process cleanup routine. If this cleanup routine runs, the failure is captured by the previous failure detector. Otherwise, when the execution of the application is suspended (and the kernel is dying), the memory will still be remotely accessible. For the operation of this failure detector, the processes executed by the components of the membership service can be arranged in a logical ring, where each process monitors its successor process. In other words, each distributed microservice monitors the adjacent part of its membership service when arranged as discussed. The data processing device described in this article adopts a local heartbeat counter. One or more processors in the membership service increment this counter to indicate that the process is active. This counter is read by the predecessor process. If the process reads the same value twice by RDMA, it will report that its successor has failed. If a process (from multiple members) fails to increment the counter between two consecutive reads, it will be falsely detected as having failed. Therefore, it is important that members always increment their counters faster than the time delay between two consecutive reads. The (single-threaded) core component can be deployed in its own dedicated physical core. The core component is a single-threaded library that contains the monitoring functions of the membership service. This includes detecting failures of member nodes, listening for failures and new memberships, and renewals. This core component runs on all members to bootstrap the membership service. In this installation, a custom kernel compiled with the NO_HZ_FULL option can be used, which disables regular timer interrupts on dedicated cores. This reduces kernel jitter towards the core component. In addition, this kernel is booted with the isolcpus parameter, which prevents other userspace processes from sharing the dedicated core with the core component. In experiments, the interval between two counter increments observed under heavy load was typically 5μs and never exceeded 15μs. In order to account for unexpected jitter (e.g. thermal throttling), one or more processors of the data processing device can be used to make the process wait 30μs after the RDMAREAD completes before issuing the next read command. Since RDMAREADs are issued in order, network latency does not negatively impact the accuracy of this failure detector.

[0058] In these ways, one or more processors of a data processing device may detect a user space and / or kernel fault in any one of a plurality of members of a system, wherein the members represent distributed microservices.

[0059] In addition to detecting these failures at the microsecond level in order to maintain an accurate picture of the members that are active within the system so that members can communicate effectively when receiving requests from users, a microscale consensus engine can also be provided. Figure 6 The algorithm 1 shown on the right side is described and may form part of the data processing apparatus.

[0060] The consensus engine of the present invention is a slightly modified version of Paxos, based on the observation that the original algorithm contains remote procedure calls (RPCs) that can be simulated through RDMA compare and exchange operations.

[0061] Processes (system members who wish to join or be part of a membership) are divided into two groups: proposers and acceptors. Proposers propose a value for a decision and acceptors accept some proposed values. Once a value is accepted by a majority of acceptors, the value is decided by its proposer. Paxos is divided into two phases: prepare phase and accept phase. In these phases, messages from proposers are identified with a unique proposal number. The prepare phase serves two purposes. First, the proposer obtains a commitment from the majority of acceptors that another proposer with a lower proposal number will not be able to make a decision. Second, the proposer updates the value proposed by the proposer with the accepted value stored in the acceptor. This way, if a value has been decided, the proposer will adopt that value. The prepare phase can also be aborted if any of the acceptors in the majority of acceptors previously committed to a higher proposal number. If a proposer manages to complete the prepare phase without aborting, the proposer enters the accept phase. In this phase, the proposer attempts to store its value in the majority of acceptors. If successful (i.e., the majority accepts the value), the proposer decides on the value.

[0062] An example of a one-sided Paxos is now described, which is used to generate the consensus engine of the present invention and is shown in Algorithm 1. Paxos uses RPC in a very specific form. Paxos's RPC follows the pattern seen in RPC. The state of the acceptor consists of only three variables: min_proposal, accepted_proposal, and accepted_value. In both phases, the acceptor atomically updates these values ​​based on the proposer's input and returns some of them.

[0063] Algorithm 1 proposes a barrier-free conversion to convert Paxos's RPC into a pure one-sided conditional write using RDMACAS. One or more processors of the data processing device of the present invention can be used to implement Algorithm 1. The recipient executing the RPC compares the received value x with its state (stored state). If the comparison is successful, the recipient updates its state with the provided value x (as shown in function f). Finally, the recipient unconditionally returns a portion of its state (shown by function proj). The pattern provided in CAS-RPC allows RDMA to simulate RPC while relying only on one-sided verbs. In contrast to RPC executed on the recipient side, CAS-RPC is executed on the proposer side. In order to perform a one-sided RPC, the proposer first needs to know the state stored in the recipient's memory. This value can be guessed (e.g., using a previous state value) or obtained (e.g., using RDMAREAD, as shown in line 2). The proposer then performs the comparison locally (line 3) and decides whether to continue or terminate. If the comparison is successful, the proposer continues to update the recipient's state. It is this update that utilizes CAS. In line 7, if CAS succeeds, the state of the acceptor has been successfully updated with the move_to value. Otherwise, the state remains unchanged. When RDMACAS succeeds, that is, in the absence of contention, both RPC and CAS-RPC are equivalent. However, if RDMACAS fails, CAS-RPC aborts while RPC does not. In this case, RPC and CAS-RPC are not equivalent, but this does not violate the correctness of Paxos because Paxos tolerates any number of proposer failures and aborting RPC and restarting is no different from such failures.

[0064] The consensus engine that can be used by the data processing device of the present invention is as follows. In particular, it will now be described how to implement a variant of the above-mentioned Paxos in order to produce a state-of-the-art consensus engine.

[0065] In the present consensus engine that can be implemented by the data processing apparatus of the present invention, one or more processors are used to avoid contention from multiple concurrent proposers by adopting a leader election scheme in which the process with the lowest identifier among the coordinators (members) considered active is elected as the leader. In the event of a partial network partition, this scheme can elect multiple leaders, thereby preventing the engine from being active. For example, if the coordinator (member) C 2 The only one who can't contact C 1 The coordinator will think of himself as the leader, while other coordinators will think of C 1 is their leader. If these coordinators cannot decide, the concurrent proposers use random backoff to restore liveness until the partition is resolved.

[0066] Figure 5aAn example of the membership service implemented by the data processing device of the present invention and the communication between the application node and the coordinator is shown. Figure 5a As can be seen in , the coordinator controls requests to join or leave membership and stored views (membership). These requests are transmitted to the application nodes to reflect the join / leave requests that have been processed by the device. The application nodes also include the ability to generate deadbeats, as discussed with respect to one or more processors for detecting system failures.

[0067] Figure 5b A consensus engine is shown with RDMA-exposed storage for multiple consensus instances (left) and a state machine for a single consensus instance (right). The coordinator decides on the sequence of values ​​by running consensus on a sequence of slots, such as Figure 5b Each slot requires two RDMA delays: one for the prepare phase and one for the accept phase (shown as horizontal arrows in the figure). A stable leader prepares slots in advance and only runs the accept phase to make a decision. In this case, the leader makes a decision in a single RDMA delay. The leader uses the time it spends waiting for the accept phase to complete on one slot to run the prepare phase for the next slot. Therefore, the leader always maintains a slot that is prepared in advance (e.g. Figure 5b The coordinator can prepare the next time slot in a single RDMA delay instead of two RDMA delays. The function of the coordinator can be performed by one or more processors of the present invention.

[0068] Figure 6 Algorithm 1 is shown, which shows the RPC of Paxos ( Figure 6 ) becomes a CAS-based RPC ( Figure 6 ). Algorithm 1 assumes that the consensus state fits in a single CAS. Currently RDMANIC only supports CAS of up to 8 bytes. min_proposal and accepted_proposal are set to 2 bytes each. The remaining 6 bytes are dedicated to accepted_value. The consensus engine of the present invention uses an indirect method to overcome the limited size of accepted_value and store system membership. The coordinator does not decide the membership itself, but rather decides where the membership is located in memory. First, the proposer RDMA writes the membership to a portion of the acceptor's memory dedicated to membership proposals, which can be Figure 5bAs can be seen in , the proposer has exclusive write access to this memory. The proposer then runs the acceptance phase, where it proposes its own identifier ( Figure 5b C 1 ). If the Accept phase succeeds at a majority of acceptors, the proposer makes the decision. Due to the FIFO semantics of RDMA RC, if the previous RDMA operation (i.e., Accept phase CAS) succeeds, then the previous RDMA operation (i.e., storing membership using RDMAWRITE) also succeeds. The two RDMA operations combined together will not be executed atomically, but it is impossible for the coordinator to accept an identifier without knowing its associated membership.

[0069] It can be understood from the above disclosure that one or more processors of the data processing device of the present invention can be used to implement the above consensus engine. More specifically, one or more processors can be used to implement Figure 6 Each step of Algorithm 1 is shown on the right.

[0070] In addition to the above-mentioned fault detection and consensus engine, one or more processors of the data processing device of the present invention can also be used to implement a microsecond-level active membership method. Typically, one or more processors for periodically determining the current system configuration include representations of multiple members of the activity. This is achieved by configuring one or more processors to determine whether there is a new system configuration to be adopted based on the consensus of multiple members of the activity; if there is a consensus between multiple members of the activity, the current system configuration is updated to the new system configuration. However, if there is no consensus between multiple members, one or more processors are used to maintain the current system configuration, or update the current system configuration to a temporary system configuration based on the current system configuration. This will be described in further detail below.

[0071] The membership service of the present invention enables applications to track active membership through the Active method. Although this information is critical for consistency, it should not impose a burden on the end application.

[0072] One or more handlers used to implement the active method will not be combined Figure 7The algorithm 2 shown in is described in detail. In the active method, the membership service exposes real-time properties to the end application through the Active(membership) → Boolean method. If Active(M) returns "true", then M (membership / view / system configuration) is active at some point between the call and return of the method. Active satisfies three important properties. First, there are no two overlapping active memberships. Second, after membership M is active, no membership older than M becomes active. This ensures that incorrect and outdated memberships are not used in the system. Third, active memberships aggregate to the most recently decided membership. In other words, if multiple members decide that there is a new system configuration, the active membership is updated to the new system configuration. Intuitively, the process (distributed microservice / member) uses the active method to determine the system configuration in which the process should perform operations. When the coordinator decides that a new membership M 0 When receiving M 0 The delay, process p (member) may be left in the old membership M. Calling Active(M) will eventually return "false" on p, making the process realize that it has missed the latest membership M. 0 To ensure consistency, applications (members) typically call Active once before starting an operation and a second time before committing the operation, committing only if both calls return true.

[0073] One or more processors are Active( Figure 7 A basic implementation of Algorithm 2) shown in may require communication in each call. The membership of a general lease can be configured as follows. Let M be the kth membership decided by the coordinator, and suppose that process p calls Active(M). Essentially, if it can be concluded that no new membership M has been decided 0 , then Active declares M to be active. In other words, one or more processors determine that the current system configuration is active unless there is a new system configuration to adopt. To do this, the process RDMA reads the k+1th consensus slot at the coordinator and waits for a majority of replies (to form a consensus of members). If all replies are empty, then the k+1th membership has not yet been determined, for example, no decision has been made about the new system configuration (M 0 ), which means that M is (still) active at some point between the method's invocation and its return. On the other hand, if at least one of the replies is non-null, it is uncertain whether M has been called by M. 0 If M is determined before p issues a READ 0, then at least one reply must be non-empty, but the converse is not necessarily true. For safety, if at least one READ is non-empty on the next consensus slot, Active returns "false". A lease refers to membership and has a start date and an expiration date. The lease guarantees to its holder that its associated membership will remain active until the membership expires. In the device of the present invention, the lease can be created by the uKharon core ( Figure 5a as shown), and can last for a duration δ approximately equal to 20 microseconds.

[0074] One or more processors of the present invention are used to implement the active membership algorithm 2 for leasing, which provides a lease dependent on membership to reduce the effective implementation of Active for communication. This is also shown in Figure 8a and 8b . First, the hardware timestamp t is obtained (line 4), and then it is checked whether there is already a lease on M (line 5) (current system configuration). If there is no lease (lines 6-7), the method checks the newly decided membership by contacting the majority of coordinators. In other words, the method determines whether there is a new system configuration to adopt based on the consensus of multiple active members. If no membership newer than M can be decided (i.e., all replies are empty), the method creates a lease on M (line 7), which starts from t + δ and has no expiration date. In other words, if there is no consensus among multiple members, the current system configuration is maintained if one member is active, or the current system configuration is updated to a temporary system configuration (lease membership) according to the current system configuration (M). This prevents overlapping active memberships because any lease held on the previous membership M 0 will expire before M becomes active. If a lease for M already exists, the method will try to use the lease to avoid contacting the coordinator (line 9). If the lease cannot be used, the method tries to extend the lease by checking the coordinator (line 11). This can be seen in Figure 7 , which shows that after the active membership period, the lease of the active membership M at the read point is taken out and is based on the current system configuration (active membership M). This will maintain the system configuration during the extended lease period. "True" is only returned when the lease of the previous membership expires (line 12), which takes δ at worst to occur. Therefore, the lease affects the speed of membership change, which justifies the desire for a short lease period because the membership will change when the lease expires. According to experiments, a lease with δ ≈ 20 μs is feasible in practice. Figure 8ashows how to execute a read command when issued by a querying member when using normal active membership without a lease. In this case, when a read request is received, a check is performed to determine if view V is active, and if so, the view's value is returned to the querying member. However, this imposes a latency requirement, i.e., a check must be performed that requires computation before the information is provided. Figure 8b As shown in the active membership, one or more processors are used to periodically determine the current system configuration. It is only necessary to determine whether the current lease or temporary lease is based on the current configuration for a set duration, i.e., the duration of the lease. Therefore, when a read request is received from a member, a view showing the system configuration is easily obtained, thereby reducing system latency.

[0075] Algorithm 2 and one or more processors of the present invention are used to allow the Active method to renew its lease on demand. As long as its lease is valid, the method only needs to spend tens of nanoseconds on the hardware timestamp and return immediately without contacting the coordinator (member). Therefore, Active has very low latency overhead for the application that calls it. Only when the lease expires and must be renewed is it necessary to communicate with the coordinator in the membership service of the present invention, which causes a surge in the latency of Active. In practice, one or more processors that implement the membership service are used to renew the lease regularly in the background to ensure that Active is not delayed by calls to majority_active when the membership remains unchanged. Algorithm 2 does not rely on operating leases to achieve vitality or security. Timely renewal is only one way to reduce Active latency because Algorithm 2 works even in zero-duration leases. In other words, the membership of the lease is indefinite until the members agree on a new membership and update the current membership to the new membership. The membership service of the present invention implemented by one or more processors relies on bounded clock drift to ensure security, for example to avoid using incorrect system configurations (memberships), without relying on clock synchronization. This ensures that the duration is approximately the same for all members, preventing overlapping system configurations and confusion among members about which system configuration is valid. Clock drift occurs when a lease lasts a few nanoseconds less than its nominal value. Since drift is reset each time a lease is renewed, it does not accumulate over time. Therefore, no matter how long the system is running, its operation will not be affected by clock drift.

[0076] One or more processors may also be used to implement adaptive leases as part of Algorithm 2. So far, a fixed lease period δ has been assumed. However, network delays greater than δ render the lease useless, since each time the lease is extended (line 11 of Algorithm 2), t end(The end time of the membership) is always in the past. In this case, Active always contacts the coordinator. In order to work under partial synchronization and avoid this situation, the lease mechanism is extended as follows: The coordinator of the membership service stores the lease period for a given membership as well as the membership itself. The lease can represent a temporary system configuration based on the current and / or previous system configuration. The temporary system configuration can be the same as the current system configuration, but with a different time period duration. An application node that wishes to extend the lease period contacts the coordinator leader (selected by consensus as described above). This will result in a new compatible membership that is identical to the previous membership except for the lease period (described above). Compatible memberships are specially handled by the core components of our membership service to ensure that when transitioning from one compatible membership to another, Active does not wait for the lease of the previous membership to expire. In addition, if the latest membership M is incompatible with the previous membership, a call to Active(M) will return "false" before all possible ongoing leases of the previous membership expire.

[0077] In addition to the active leases described above, the data processing apparatus may include one or more caches for storing current system configurations and / or temporary system configurations for a majority of the plurality of members. These caches may be viewed as lease caches accessible by one or more processors and / or members of the system to quickly access the latest system configuration information, such as the latest view / membership. This is accomplished by Fig. 9 It is shown that the data device is used to implement a cache layer between the coordinator (member) and the client.

[0078] The data processing device may include a lease cache because Active needs to contact the majority of members to renew its lease. This requirement expands greatly as the number of application nodes increases and the system grows. The present device that implements the membership service solves this problem by including an intermediate renewal layer in the form of a lease cache. These caches use the Active method to lease membership within the bounds of the communication and processing delay Δ (by reading from the majority of coordinators). Then, the application node uses a lease that lasts δ (a period of time) and a modified version of Active. This version is different from the version provided in the majority_check call in Algorithm 2, which is replaced by RPC to a single lease cache. In other words, only one lease cache is checked for system configuration settings, and there is no need to check the majority of members. Therefore, the application node reduces the communication cost required for renewal by at least 3 (typical number of members) times. However, the lease cache increases the failover time of the application by at least the bounds of the communication and processing delay Δ, because when the coordinator changes the membership, the cached Active method waits for an internal duration Δ before making the new membership active. At the same time, one or more processors are used to wait for another duration δ before making the new membership active when implementing the Active method of the application node directed to a lease cache. Therefore, the total time from deciding on the new membership to the application node starting to use the new membership jumps from (at least) δ to (at least) Δ+δ.

[0079] The present invention also provides a data processing method for implementing the cloud computing microsecond group membership service described above with respect to the data processing device. The steps of the method will be apparent from the above and are further defined by the claims. However, typically, the data processing method includes detecting one or more system failures of the first plurality of members. This can be implemented using the user space failures and kernel failures discussed above. The method then further includes determining a current system configuration, which includes representations of multiple active members. This can be achieved by performing the following method steps: determining whether there is a new system configuration to be adopted based on the consensus of multiple active members; if there is a consensus between the multiple active members, updating the current system configuration to the new system configuration; or, if there is no consensus between the multiple members, maintaining the current system configuration, or updating the current system configuration to a temporary system configuration based on the current system configuration.

[0080] The membership service of the present invention can be implemented in conjunction with HERD. HERD is an RDMA-based non-replicated microsecond key value cache (KVcache). This allows clients to send requests to a HERD server via RDMA writes to a dedicated buffer allocated by the server. Such requests contain an 8-byte key and can be either a PUT or a GET. The PUT also contains the value to be stored for the specified key. The server discovers new client requests by polling its local storage, executes the request locally, and then replies to the client using RDMAUD (Unreliable datagram (UD), similar to a UDP connection / transport). The membership service can also be used to build a key-value storage system, namely uKharon-KV, an extended version of HERD that supports replication. An example of comparing the membership service of the present invention implemented by HERD with other use cases can be found in Fig.10 As can be seen in the figure, the membership service (uKharon-KV) achieves an end-to-end failover of 53 microseconds, which is equivalent to a tenfold improvement over the known service. In addition, the membership service of the present invention replicates small requests in 1.26s, which is 13% faster than the state-of-the-art technology.

[0081] Therefore, the data processing device and data processing method of the present invention that implement the above-mentioned leased active membership provide a leased active membership mechanism that protects membership with a pure client lease. This helps to reduce the lease to tens of microseconds and only takes bounded clock drift instead of loosely synchronized clocks. In addition, since Algorithm 2 relies on bounded clock drift rather than clock synchronization to ensure security, this ensures that the duration of all processes is approximately the same, thereby preventing membership overlap. In addition, the microsecond membership mechanism for implementation on one or more processors provides a core building block to build high-performance, low-latency distributed cloud services such as data storage, video streaming, e-commerce, and banking platforms. The mechanism provides cloud services with strict tail latency, ensuring the smoothness and scalability of user interactions. Fault tolerance in cloud services is also improved due to microsecond fault detection, which promotes strong consistency at the microsecond level.

[0082] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. In this sense, in view of the common sense of those skilled in the art, such features or combinations can be implemented as a whole according to this specification, without considering whether such features or combinations of features can solve any problems disclosed herein, and without limiting the scope of the claims. This application shows that various aspects of the present invention can be composed of any such individual features or combinations of features. In view of the above description, it is obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A data processing device for implementing a microsecond group membership service in cloud computing, characterized in that: The data processing device comprises one or more processors, wherein the one or more processors are configured to: detecting one or more system failures of the first plurality of members; Periodically determining a current system configuration including representations of a plurality of members of an activity by configuring the one or more processors to: determining, based on a consensus of the plurality of members of the activity, whether there is a new system configuration to be adopted; If there is a consensus among the multiple members of the activity, updating the current system configuration to the new system configuration; or, If there is no consensus among the multiple members, the current system configuration is maintained, or the current system configuration is updated to a temporary system configuration based on the current system configuration.

2. The data processing device according to claim 1, characterized in that: The one or more system failures include a user space failure.

3. The data processing device according to claim 2, characterized in that: The one or more processors are configured to detect the user space failure by detecting a failure notification sent from the Linux kernel when a system process fails.

4. A data processing device according to any one of the preceding claims, characterized in that The one or more system failures include a kernel failure.

5. The data processing device according to claim 4, characterized in that: The one or more processors are configured to detect the core failure by determining that a heartbeat counter local to the first or second plurality of members is not incrementing.

6. A data processing device according to any one of the preceding claims, characterized in that The one or more processors are configured to select a leader from the plurality of members based on an atomic comparison and exchange of a proposed leader from the plurality of members with a current leader from the plurality of members.

7. A data processing device according to any one of the preceding claims, characterized in that The current system configuration is updated to the new system configuration only after a time period of the current system configuration expires.

8. A data processing device according to any one of the preceding claims, characterized in that The temporary system configuration is maintained until the plurality of members of the activity reach a consensus on a new system configuration.

9. A data processing device according to any one of the preceding claims, characterized in that The one or more processors are further configured to send a fault notification signal to the plurality of members other than the member where the fault occurred, in response to one or more system faults detected by the plurality of members.

10. The data processing device according to any one of the preceding claims, characterized in that The current and / or new system configuration expires after a predetermined period of time.

11. The data processing device according to claim 10, characterized in that: The temporary system configuration is the same as the current system configuration but has a different time period duration.

12. A data processing device according to any one of the preceding claims, characterized in that The data processing device comprises one or more caches for storing the current system configuration and / or temporary system configuration of a majority of the plurality of members.

13. A data processing method for implementing a microsecond group membership service in cloud computing, characterized in that: The data processing method comprises: detecting one or more system failures of the first plurality of members; A current system configuration including representations of a plurality of members of an activity is determined by performing the following method steps: determining, based on a consensus of the plurality of members of the activity, whether there is a new system configuration to be adopted; If there is a consensus among the multiple members of the activity, updating the current system configuration to the new system configuration; or, If there is no consensus among the multiple members, the current system configuration is maintained, or the current system configuration is updated to a temporary system configuration based on the current system configuration.

14. The data processing method according to claim 13, characterized in that: The one or more system failures include a user space failure.

15. The data processing method according to claim 14, characterized in that: The method also includes detecting the user space failure by detecting a failure notification sent from the Linux kernel when a system process fails.

16. The data processing method according to any one of claims 13 to 15, characterized in that: The one or more system failures include a kernel failure.

17. The data processing method according to claim 16, characterized in that: The method also includes detecting the kernel failure by determining that heartbeat counters local to the plurality of members are not incrementing.

18. The data processing method according to any one of claims 13 to 17, characterized in that: The method also includes selecting a leader from the plurality of members based on an atomic comparison and exchange of a proposed leader from the plurality of members with a current leader from the plurality of members.

19. The data processing method according to any one of claims 13 to 18, characterized in that: The current system configuration is updated to the new system configuration only after a time period of the current system configuration expires.

20. The data processing method according to any one of claims 13 to 19, characterized in that: The method also includes maintaining the temporary system configuration until the plurality of members of the activity reach a consensus on a new system configuration.

21. The data processing method according to any one of claims 13 to 20, characterized in that: The method further includes, in response to the detected one or more system faults of the plurality of members, sending a fault notification signal to the plurality of members other than the member where the fault occurred.

22. The data processing method according to any one of claims 13 to 21, characterized in that: The current and / or new system configuration expires after a predetermined period of time.

23. The data processing method according to claim 22, characterized in that: The temporary system configuration is the same as the current system configuration but has a different time period duration.

24. The data processing method according to any one of claims 13 to 23, characterized in that: The method also includes storing the current system configuration and / or temporary system configuration of a majority of the plurality of members in one or more caches.

25. The data processing method according to any one of claims 13 to 24, characterized in that: One or more system failures of the first plurality of members are detected at a microsecond level.

Images