Distributed components in a computing cluster

Abstract

Embodiments of the present disclosure relate to distributed components in computing clusters. The present disclosure relates to components in different server clusters, for example, including software components such as components of a distributed computing system. The components are available for use by distributed computing system applications, but are managed by the distributed computing system runtime so that only a single instance can be activated and exists within a communicating (unpartitioned) cluster. Recovery from a situation where each of the clusters that are no longer partitioned has created the same component is also described.

Description

[0001] This application is a divisional application of an invention patent application whose international application date is October 7, 2015, entered the Chinese national stage on April 7, 2017, has Chinese national application number 201580054797.6, and whose invention name is “Distributed Components in Computing Clusters”. Technical Field

[0002] Embodiments of the present disclosure relate to distributed components in a computing cluster. Background Art

[0003] A distributed computing system may have software components, such as software objects or other types of individually addressable, isolated entities, such as distributed objects, agents, actors, virtual components, etc. Typically, each such component is individually addressable and has a unique identity (such as an integer, GUID, string, or opaque data structure, etc.) in the distributed system.

[0004] In a distributed system that allows geographic distribution (including when two sets of servers arranged as a cluster are in the same data center or in different data centers), application deployment can reside in a cluster. In this case, an application whose clients are widely distributed (e.g., around the world) processes requests at a single data center, whereby clients far away from the data center may experience high latency. Summary of the invention

[0005] This Summary is provided to introduce a selection of representative concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used in any way to limit the scope of the claimed subject matter.

[0006] Briefly, one or more of the various aspects of the subject matter described herein are directed to attempting to determine whether a component has been activated in any other cluster with which the attempting cluster is able to communicate, and if so, using the already activated component. If not yet activated, a determination is made as to whether there is a partition between at least two clusters; if there is a partition, the component is activated and associated with state data indicating that the activated component is a possible duplicate, such as associated with another instance in the partitioned cluster.

[0007] Other advantages will become apparent from the following detailed description taken in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The present disclosure is illustrated by way of example and not limitation in the accompanying drawings, in which like reference numerals indicate similar elements, and in which:

[0009] Figure 1 is a block diagram representing example components of a distributed computing system having a cluster that can be used to provide and manage partitioning of components according to one or more example implementations.

[0010] Figure 2 is a block diagram representing example components of a distributed computing system having a de-partitioned cluster that can be used to provision and manage components according to one or more example implementations.

[0011] Figure 3 is a flow chart with example steps that may be used to provide different modes of operation with respect to component activation according to one or more example implementations.

[0012] Figure 4 is a block diagram illustrating an example of components in a partitioned cluster and activation and deactivation of replica components according to one or more example implementations.

[0013] Figure 5 is a block diagram representing component activation requests and responses communicated between clusters according to one or more example implementations.

[0014] Figure 6 is a representation of data maintained by a cluster (or an individual server of a cluster) according to one or more example implementations, including activation state data for components.

[0015] Figure 7 is a flow diagram with example steps that may be used to send an activation request and process a response into an action according to one or more example implementations.

[0016] Figure 8 is a flow diagram with example steps that may be taken in receiving an activation request and outputting a response according to one or more example implementations.

[0017] Fig. 9 is a flowchart having example steps that may be taken when a no longer partitioned cluster recovers from a partition by sending a set of possible replicas to other clusters and processing responses to remove any replicas selected as non-viable, according to one or more example implementations.

[0018] Fig.10 is a flowchart having example steps that may be taken in receiving and processing a set of possible replicas to determine if any replicas exist and, if so, which replicas are alive and outputting responses, according to one or more example implementations.

[0019] Fig.11 is a block diagram representing an example non-limiting networked or distributed computing environment in which one or more aspects of the various examples described herein may be implemented.

[0020] Fig.12 is a block diagram representing an example non-limiting machine / computing device in which one or more aspects of the various examples described herein may be implemented. DETAILED DESCRIPTION

[0021] Various aspects of the technology described herein are generally directed to allowing applications to span multiple server clusters, such as server clusters in different data centers, such as in a cloud environment. Such applications activate (instantiate) components or use already activated components. Components are based on the concept of always being available and having a single activation, including in a multi-cluster environment. Note that while clusters can be in multiple data centers, the technology is applicable to different server clusters arranged in a single data center, with some clusters in one data center and some clusters in other data centers, or arranged in any other suitable manner.

[0022] As used herein, a cluster is a collection of servers that typically run within a data center. A cluster has a unique identity relative to other clusters. A multicluster is a collection of clusters that can be geographically distributed, such as (but not limited to) clusters that reside in different data centers. The entire multicluster is the set of all clusters, which is statically known to each cluster. An active multicluster is a subset of the entire multicluster whose members can communicate with each other; when some clusters of the entire multicluster cannot communicate with each other, the cluster is said to be partitioned. A quorum multicluster is an active multicluster that contains a quorum of clusters in the entire multicluster (e.g., a majority of the entire cluster) or has arbitration resources. There can be at most one arbitration multicluster in the entire multicluster. Note that when there is no partition, each cluster is part of the quorum.

[0023] In a component system, applications are programmed as if any given single activated component always exists. When a component is called, the runtime system activates the component if it is not already running, or uses the activated instance of the component if it is already running.

[0024] Therefore, applications are programmed as if there is one instance of any component at any time, which is built into the programming model and distributed runtime. If the application calls a component, the runtime system looks for an activation for the component if it exists (possibly in another server cluster), or creates it if it does not.

[0025] However, when dealing with multiple computing clusters, different clusters may attempt to create the same component at the same time, i.e., a race condition exists. Furthermore, clusters may become partitioned from one another, such as by a failure of a communication link between clusters. In such cases, the concept of a single active virtualized software component that is individually addressable becomes uncertain.

[0026] Techniques including a suitable example protocol for handling race conditions and partitioning clusters are described herein. For example, server clusters may become partitioned from one another due to failures in communication links between them, which tends to be more common when the server clusters are run in different data centers, especially when the server clusters are separated by relatively large geographic distances. Nevertheless, different server clusters that are relatively close to one another (including clusters running in the same data center) may also be partitioned from one another.

[0027] In the event that clusters are partitioned from one another, the distributed system continues to operate, but the concept of only activating a single component in the entire distributed system may be overly restrictive. For example, an application may require the use of a component instance that can run on a partitioned cluster (which the runtime will activate if necessary), and the requested cluster runtime cannot communicate with the other partitioned clusters.

[0028] Various modes of operation are described herein, which generally relate to the consistency and availability of components. For example, in high consistency mode, the system ensures that there is at most one activation of a component across clusters. To this end, if any cluster becomes partitioned, new activations of components are not allowed to occur until the clusters are communicating again.

[0029] In another mode, such as high availability mode, if a cluster cannot access an activation of a component, the cluster may create an instance of the component even though another activation of the component may already exist (or be created later) on another split cluster. When the split clusters communicate again, reconciliation of the component instances (e.g., their states) is performed.

[0030] Note that the above-mentioned operation modes and other operation modes can be based on each component, or each component type. For example, component Y can be specified (e.g., by an administrator) to run in high consistency mode, while component Z can be specified to be allowed to run in high availability mode.

[0031] Other patterns are possible, some of which are exemplified herein. As an example, if one or more clusters become partitioned, one of the clusters that is considered (and known by the other clusters as) part of a quorum by, for example, a majority of clusters (or by arbitrating ownership of resources, etc. or any other suitable ownership / reservation mechanism) can create new instances of components. One or more other clusters that do not have a quorum and that are partitioned from a cluster with a quorum cannot create new instances of any component.

[0032] Yet another mode allows for the activation of a component that exists only for that cluster. As an example, if the cluster knows (or is likely) that no other cluster will need to create another instance, the cluster can optimistically create an instance of the component. As a more specific example, if the cluster needs to activate a component that includes a video game, and the game player is also in the cluster, the component can be activated independently of any other cluster, whether partitioned or not. Other clusters can also have their own independent component instances.

[0033] In addition, at least some of the modes can be combined to some extent. For example, in the video game example, if most players (but not necessarily all) are in a cluster, or perhaps in different clusters but in the same data center, multiple clusters can optimistically create their own independent game component instances. A cluster can also optimistically create a component instance even before getting a response from any or all other clusters if it is considered possible (e.g., from previous usage patterns) that no other cluster may have created the component. Other optimistic and pessimistic scenarios are feasible, such as optimistically creating a component instance if more than half of the other clusters have responded, or some other configurable threshold percentage.

[0034] It should be understood that any examples herein are non-limiting. For example, components are not limited to any particular distributed system. As another example, a cluster generally refers to two or more servers that operate in conjunction with each other so that in the event of a communication failure within any server and / or cluster, a surviving server or set of servers takes over the application of the failed / non-communicating server. Therefore, as used herein, a "cluster" even includes a single surviving server. A cluster also includes the concept of virtual machines acting as different servers, even if running on the same physical machine.

[0035] Furthermore, a "component" refers to any software component / software object / actor having properties that can benefit from the techniques described herein. Thus, the present invention is not limited to any particular implementation, aspect, concept, structure, function, or example described herein. On the contrary, any implementation, aspect, concept, structure, function, or example described herein is non-limiting, and the present invention can generally be used in various ways that provide benefits and advantages in programming, computing systems, data centers, and distributed computing systems.

[0036] Figure 1A multi-cluster configuration is shown including clusters 102(1)-102(n). Clusters 102(1)-102(3) are partitioned from clusters 102(4)-102(n) as indicated by the "X" in the communication links. Note that for purposes of illustration, two data centers A and M are shown (whereby a partition may be caused by a failure of a wide area network connection between the data centers), however, it should be understood that data centers are not required, but any practical number may be connected if present.

[0037] As shown in one server cluster 102(1), but as will be appreciated, applicable to other clusters, one or more applications 104 may request the use or activation of a component via runtime 106. The technique may use a (e.g., consensus) protocol P for clusters to track which activations exist and at which cluster each activation resides, wherein Figure 1 The trace data is represented in Component Activation Data 108. Protocol P is implemented in a runtime 106, comprising a system running on a server processor / memory that supports a multi-cluster single activation component abstraction as described herein.

[0038] In general, if there is a component in the system, the component has an always addressable identity in the component space, whether in an activated state or a deactivated state. Therefore, the application can send operations (tasks) to the component, and the components can communicate with each other, regardless of their actual current state. If the component is in a deactivated state and is sent a communication, such as an operation request and / or a message (because the activation or deactivation state of the component is unknown to the sender), the runtime can be responsible for retaining any such operation request and communication until the component can be activated and provided with an operation request and / or communication. Alternatively, the attempt to call the component to obtain a reference to the deactivated component can cause the deactivated component to be activated, after which the calling component can send operations or messages to other components. Note that there may be rare exceptions to the concept of "always" addressable and "always" existing components, for example, the runtime can be programmed to intentionally ignore requests involving one or more components / administrators can intentionally remove one or more components from memory and persistent storage. Therefore, as used herein, "always" addressable or "always" existing means "always or substantially always" to explain any such exceptions.

[0039] In this example, consider that component activation does not currently exist on clusters 102(1)-102(3), as known by component activation data 108 maintained or otherwise accessible thereon (e.g., as a replication database). However, without a way to communicate with clusters 102(4)-102(n), one of which may attempt to create its own instance before the partition is repaired.

[0040] In this case, the runtime may access policy / rule 110 to determine what action to take. That is, as described above, one of various possible modes of operation may be used with respect to the requested component, for example, as specified in policy / rule 110.

[0041] To provide high availability in the presence of network partitions, the system allows for the possibility of two or more active nodes when the clusters cannot communicate. Figure 1 As shown, if cluster 102(1) is unable to communicate with at least one of the other clusters for high availability, runtime 106 may allow cluster 102(1) to activate a component that it cannot access based on policies / rules 110. The runtime notes that the activation is a possible duplicate, as described below, for example by associating the activation with a "DOUBTFUL" state.

[0042] When communication is restored, Figure 2 As shown, the system reestablishes component consistency by having the clusters exchange lists or other data structures of possible replica component activations (e.g., 222(1) and 222(n) are shown, although each cluster sends its own list). If the activation has in fact been replicated, the protocol reestablishes consistency by retaining only one surviving activation and notifying one or more other activations that they are replicas. Any replica activations are terminated, typically including the first persistent state. The application (e.g., 104) is responsible for merging any state of the one or more other activations into the surviving activation.

[0043] As described herein, in a high consistency mode, consistency issues can be prevented entirely by not allowing activations for any given component (or potentially for any component). In a less restrictive but still consistent mode, a rule may dictate that a set of clusters are allowed to create activations (for specified components or types, or potentially for any activation), but only if the creating cluster is part of a quorum of clusters. Availability is still compromised because clusters that are not part of a quorum are not allowed to create new activations.

[0044] However, note that some applications such as geographically distributed user-facing applications (e.g., for games and social networks, client connections) are often localized to the data center closest to them. In many cases, interactions are performed with users in the same geographic location. Therefore, the probability of multiple activations of user / session components is reduced. In this case, a non-arbitrated split cluster can create activations despite the possibility of replicas.

[0045] Figure 3is a simplified example flow chart outlining some of the above concepts. If there is no split at step 302, the system operates in normal mode, where only one activation is allowed at a time (although there may be temporary exceptions as described below, such as when the runtime optimistically creates an activation as the only possibility and then checks if another activation exists).

[0046] If there is a split, e.g. the runtime running the protocol belongs to the active cluster which is a proper subset of the entire multi-cluster, then step 302 instead jumps to step 306 which operates when an activation needs to be created in response to an application request. Note that if the activation has already been instantiated (activated) in the active cluster, the runtime only uses that activation and returns information to the application so the application will use the existing activation instance.

[0047] As described above, if the component is allowed to be created (possibly as a replica) in high availability mode (whether specifically identified or by having a certain class / type), then step 308 jumps to step 314, where the activation is created, but with a DOUBTFUL (doubtful) status indicating that it is a possible replica.

[0048] Conversely, if activation creation is allowed only in high consistency mode, but a partition exists, step 308 jumps to step 310, which evaluates whether the cluster is part of a quorum. If not, the activation request is denied at step 312. Note that in an alternative most restrictive high consistency scenario, step 310 may be bypassed, i.e., the request is denied as shown in step 312, regardless of whether the cluster is part of a quorum.

[0049] It will be appreciated that a given component may be allowed to be used in high availability mode, another component in high consistency mode (regardless of quorum), while yet another component may be in high consistency mode (depending on quorum membership). As an example, if there is no practical way to reconcile (recover) the state of a component using replicas, then such a component may need to be activated only in high consistency mode. Another component may have replicas that can be reconciled, in which case that component may be a good candidate for high availability mode.

[0050] It can be easily understood that protocol P requires runtime overhead, usually communication delay. For components that are known to be accessible only in the cluster (or possibly the data center of the cluster) that activates them statically (e.g., when the application is compiled), this overhead can be avoided. For example, the component representing the game instance can only be accessed by the player component in the same data center as the game instance. In this case, it is not necessary to notify the clusters in other data centers about the activation of the game instance. In order to avoid this notification, the class definition for the game component can be annotated with a keyword, for example, the keyword tells the system not to run the protocol when the component of the class needs to be activated.

[0051] Distributed systems therefore provide a programming model that treats components as virtual entities rather than physical entities and supports distributed computing runtimes. As will be appreciated, this greatly simplifies component management from a programmer's perspective, while providing scalability and reliability, among other benefits. Note that, as used herein, "component" refers to other distributed components, objects, or actors that are similar in nature.

[0052] For example, a "virtualized distributed component" (or more simply a "component") as described herein has an existence that is independent of the lifespan of any of its in-memory instantiations and, therefore, independent of the lifespan of any particular server. For example, if there is no in-memory instance of a component, a message automatically sent to the component can cause a new instance to be created on an available server (e.g., by the runtime or another component such as the sender), which can be any available server. The runtime thus gives the developer a virtual "component space" that allows any possible component in the system to be called as if it existed in memory. Thus, even if a server crashes, the component will not fail, and the component does not need to be supervised and / or explicitly recreated by the application.

[0053] As can be readily appreciated, application code can therefore be written in an optimistic style, assuming that all legitimate requests will be serviced by the corresponding component. This removes the need to write application code that checks for the existence of a component and creates the component if it is not found, which greatly simplifies application logic and reduces the amount of code that developers need to write.

[0054] As an example, Figure 4The activation, deactivation, and persistent state of components in one or more implementations are summarized. The deactivation component in class 440 is activated by runtime 442 into server memory 444 of a selected server of a cluster, for example, activation component M can be activated from deactivation as needed. Once activated, activation component M can call state from / store state to persistent storage 446 (if present) as needed. Note that persistent storage 446 can be accessible regardless of the physical server on which the component is currently activated, for example via centralized and / or replicated storage that can be based on each cluster, for example. Alternatively, however, state can be read / stored from / to many different kinds of media, such as mobile phones, consoles, cloud storage services, etc. Such storage may not always be accessible, for example, to devices such as phones / consoles; however, it is possible that components that rely on access to a device only need to be accessible when the device is accessible.

[0055] In addition, Figure 4 As shown, component X is deactivated by runtime 442 based on one or more deactivation criteria, such as based on (lack of) usage, for example when no work has been performed for a period of time. Note that it is also possible to allow a component to deactivate itself, for example by calling a deactivation function. At any time, and as part of deactivation before deactivation is complete, component X can persist state to persistent storage 446 (if present) as needed. Component X is then deactivated.

[0056] Figure 4 Also shown is a replicated activation D of a component, e.g., the servers on the two split clusters A and B each have their own instance of the component activation D. Each runtime allows state to be persisted, but in this case, the clusters cannot communicate, and thus when the clusters communicate again will need to perform state reconciliation, with only one activation instance surviving, as described herein. Note that in this example, storage is centralized on a per-cluster basis.

[0057] As understood, the runtime activates a component when it is needed by an application or another component. Therefore, because an application can address a component at any time and does not have the concept of an activated or deactivated state of a component, for the purpose of simplicity and explanation herein, a component may be considered "activated" or "activated" even if it is temporarily in a deactivated state.

[0058] As will be appreciated, techniques are described herein that provide support for applications across multiple clusters. Specifically, a programmer does not need to explicitly specify the cluster in which a component must be activated. Thus, one or more implementations allow unmodified applications to be geographically distributed.

[0059] In a geo-distributed setting, components are distributed across multiple clusters. However, it is unlikely that every component in the system will be called by requests from every cluster. Calls to a specific component can be restricted to one of two access modes, including locally referenced components or globally referenced components.

[0060] For example, many components may only be referenced from within a single cluster. This access pattern arises naturally in several application scenarios. For example, in the context of a multi-player gaming scenario, the game typically logs its players into the same cluster or collection of clusters in the same data center.

[0061] A globally referenced component allows a component to be referenced from any cluster. For example, a globally referenced component may contain aggregate information, such as a global leaderboard or a publicly accessible summary of a user's gaming profile.

[0062] Turning to a suitable activation creation protocol, for clarity, each cluster may be considered as if it were a single node; messages that may need to be exchanged between clusters are described herein. In one or more implementations, the geographically distributed component directory protocol P is based at least in part on the component access pattern described above.

[0063] like Figure 5 As shown, when cluster C0 needs to activate component V, C0 asks each cluster with which it can communicate (e.g., clusters C1-Cn) whether the cluster has already activated the component. To this end, the request / response part of the protocol P running on each cluster is described. If there is a cluster that has already activated the component, C0 uses that activation. If no other cluster C1-Cn with which C0 can communicate has already created an activation, cluster C0 creates an activation locally.

[0064] The techniques described herein need to handle race conditions, e.g., when C0 first attempts to invoke component V, it asks each cluster with which it can communicate about V's activation; this prevents C0 from creating a duplicate activation of V. However, if another cluster C1 simultaneously attempts to invoke V for the first time, C0 and C1 may otherwise each create a different activation for V. Protocol P avoids this race condition scenario, as described below.

[0065] The techniques described herein also need to handle partitions, i.e., when not all clusters can communicate with each other. For example, since cluster C0 can only ask clusters with which it can communicate about component V, if there are any clusters that are partitioned from cluster C0, one or more of them may have activated V. These activations in the partitioned clusters are unknown to cluster C0, so as described herein, in some cases, cluster C0 may still activate V despite these activations.

[0066] One solution to the partition problem is to use a consistent component creation policy that does not allow components to be created in the presence of a partition. Another slightly less restrictive solution is to only allow clusters that can communicate and have quorum to create components in the presence of a partition. However, there is a significant opportunity for partitions, especially in wide area networks, so such a policy can make the system, or potentially large parts of the system, unavailable for creating new components.

[0067] like Figure 6 As shown in the example, each cluster has a component directory 660, which generally includes a mapping of VirtualComponentId (virtual component Id) to a type tuple<ActivationId,ActivationState> (<activationId, activationState>) dictionary; where VirtualComponentId is the component identifier, ActivationId is the component activation identifier, and ActivationState. Figure 6 As shown in the general description, ActivationState can take one of five values:

[0068] OWNED: If an activation is created without partitioning, the activation is in the OWNED state. That is, when it is created, the active multicluster of the creating cluster is equal to the entire multicluster.

[0069] DOUBTFUL: An activation is in state DOUBTFUL if it is created with one or more splits between the creating cluster and the other clusters. That is, at the time of being activated, the active multicluster of the creating cluster is a proper subset of the entire multicluster. If an activation of component V is DOUBTFUL, it may have been created despite the existence of another activation outside the active multicluster of the creating cluster.

[0070] REQUESTED_OWNERSHIP: If the cluster in which the activation of component V exists is in the process of asking other clusters in its active multi-cluster for other activations, the activation of component V is in the state REQUESTED_OWNERSHIP.

[0071] RACE_LOSER: An activation A0 of a component V in cluster C0 is in state RACE_LOSER if cluster C0 loses the race condition with another cluster (eg, C1) that is trying to activate A1 simultaneously.

[0072] CACHED: An activation is CACHED if it is a reference to an activation in a remote cluster.

[0073] Turning to how the techniques described herein handle race conditions, as an example, consider that cluster C0 wants to create activation A0 for component V. Figure 7 As shown in step 702, C0 firstly generates a component directory V=><A0,REQUESTED_OWNERSHIP> C0 then broadcasts to the clusters in its active multi-cluster (e.g., Figure 5 and Figure 7 Step 704) sends a message of type ACTIVATION_REQUEST(V,C0), where V is the VirtualComponentId of the component that C0 is attempting to activate.

[0074] When another cluster (e.g., cluster C1) receives the ACTIVATION_REQUEST(V, C0) message, cluster C1 checks its component catalog for the mapping corresponding to component V. The response cluster runs the response part of protocol P, typically as follows Figure 8 As shown, step 802 indicates receiving a request.

[0075] If the component catalog of cluster C1 does not contain a mapping for component V, as evaluated at step 804, cluster C1 returns ACTIVATION_RESPONSE(PASS, null) at step 822. Cluster C1 does not make any changes to its component catalog.

[0076] If C1's component catalog contains a mapping for component V, the mapping can be in one of the five ActivationStates described above. The action taken by C1 is based on the active ActivationState in its component catalog.

[0077] Step 806 evaluates whether C1 contains V=> in its component catalog<A1,OWNED> , where A1 is the activation created by C1 for component V. In this case, C1 has already created an activation for component V. Therefore, cluster C1 responds to cluster C0 by sending a message ACTIVATION_RESPONSE(FAILED, A1) to cluster C0 at step 810 that C1 already owns the activation for component V, where A1 represents the activation ID.

[0078] If not, step 808 evaluates whether cluster C1 contains V=> in its component catalog<A1,DOUBTFUL> This situation is similar to the case where C1 contains an activation in state OWNED. In this case, C1 has an activation (A1) for component V, but when C1 created activation A1, C1's active multi-cluster was a subset of the entire multi-cluster. If suspicious, cluster C1 similarly responds with ACTIVATION_RESPONSE(FAILED, A1) at step 810.

[0079] If the status is neither owned nor suspected, step 812 evaluates whether cluster C1 contains V=> in C1's component catalog<A1,REQUESTED_OWNERSHIP> In this case, both C0 and C1 compete to activate component V; Figure 7 Recall from step 702 that before issuing the message ACTIVATION_REQUEST, C0 inserted the entry V=><A0,REQUESTED_OWNERSHIP> . Also note that because C1 is also requesting ownership, C0 behaves similarly as Figure 8 The response portion of the illustrated protocol.

[0080] After receiving the message ACTIVATION_REQUEST from cluster C0, the state of C1 is REQUESTED_OWNERSHIP for the component, and C1 needs to decide whether it wins or loses the race condition. A tiebreaker mechanism T, such as that built into the policy / rule 110 and exemplified herein as a deterministic priority function prec_func, can be used (step 814) to decide which cluster wins the race condition. Note that cluster C0 is or will similarly run a priority function.

[0081] In one or more implementations, the function prec_func takes two parameters, the first being a VirtualComponentId V and the second being a ClusterId C. For a given component V, prec_func(V, C) defines a total order over all Cs; for a given pair of distinct clusters C and C', prec_func(V, C) is less than prec_func(V, C') or prec_func(V, C) is greater than prec_func(V, C').

[0082] If prec_func(V,C1) is greater than prec_func(V,C0), C1 wins the race condition (step 816) and sends ACTIVATION_RESPONSE(FAILED,null) at step 818. Conversely, if prec_func(V,C1) is less than prec_func(V,C0), C1 performs two steps: At step 820, cluster C1 changes the state of its component directory entry corresponding to V to V=><A1,RACE_LOSER> , and in step 822, cluster C1 sends ACTIVATION_RESPONSE(PASS,null) to cluster C0.

[0083] If the status is not REQUESTED_OWNERSHIP in step 812, then C1 may include V=><A1,RACE_LOSER> or V=><A’,CACHED> Note that in this implementation, these are the only two remaining possible states (CACHED means that C1 contains a reference to activation A' of V owned by another cluster C'). In either case, C1 responds in step 822 with ACTIVATION_RESPONSE(PASS, null).

[0084] return Figure 7 , upon receiving an ACTIVATION_RESPONSE from each cluster in its active multi-cluster, the entry for cluster C0 for component V is either in state REQUESTED_OWNERSHIP or in state RACE_LOSER (because cluster C0 may have similarly performed steps 814, 816, and 820 based on C1's request for ownership).

[0085] If the entry for C0 of V is in state RACE_LOSER as evaluated at step 706, then the other cluster won the race. C0 delays for a short time (step 720), sets its state to REQUESTED_OWNERSHIP, and reruns the protocol by sending a message ACTIVATION_REQUEST to each cluster in its active multi-cluster. Note that cluster C0 delays in order to allow the cluster that won the race condition to take ownership of the component.

[0086] If C0's component catalog for component V is in state REQUESTED_OWNERSHIP, then C0 takes the following steps based on the message ACTIVATION_RESPONSE it sees from the clusters in its active multi-cluster:

[0087] If C0 receives ACTIVATION_RESPONSE(PASS, null) from all other clusters in its active multi-cluster (step 708), and C0's active multi-cluster is also the entire multi-cluster (i.e., there is no partition at step 710), then at step 712, C0 removes its component catalog entry from V=><A0,REQUESTED_OWNERSHIP> Change to V=><A0,OWNED> If the active multicluster of C0 is not equal to the entire multicluster, i.e. there is a partition, then C0 changes its component directory entry to V=><A0,DOUBTFUL> . Activation then occurs, as shown in step 716.

[0088] If cluster C0 receives an ACTIVATION_RESPONSE(FAILED, A') from at least one of the clusters in its active multi-cluster, then step 718 jumps to step 722, where the cluster changes its component catalog entry to V=><A0,CACHED> , as shown in step 726. Note that, whether in the OWNED state or the DOUBTFUL state, the activation ID A' is returned via step 810 executed on the cluster that activated the activation. If cluster C0 receives more than one such response (step 722), cluster C0 uses prec_func (step 724) to decide which activation to cache in its component catalog at step 726. Cluster C0 then uses the existing activation, as shown in step 728.

[0089] Returning to step 718, if cluster C0 did not receive a FAILED ACTIVATION_RESPONSE message with a non-null A', but received at least one ACTIVATION_RESPONSE(FAILED, null) message, cluster C0 reruns the protocol by sending an ACTIVATION_REQUEST message to each cluster in its active multi-cluster. This may include a delay, as shown in step 720.

[0090] The cluster may, for example, after step 716, broadcast activation notifications of its recently activated components to other clusters. In one implementation, for example, a class may be annotated with ClusterLocal or ClusterShared, so that only the activation of components of the ClusterShared class needs to be broadcast. These may be set to the state "CACHED" in the component catalogs of other clusters.

[0091] With respect to detecting multiple activations of a component, consider that cluster C0 creates a DOUBTFUL activation after running the protocol described herein when the active multicluster is a subset of the entire multicluster. Due to a partition, C0 cannot communicate with clusters that are not part of its active multicluster. These partitioned clusters may have created activations of the component. When creating an activation in state DOUBTFUL, a check is performed as to whether one or more replica activations exist when repairing after a partition. Fig. 9 and 10 Involved in handling any multiple activations.

[0092] If there are one or more duplicate activations, one activation is classified as a "winner" activation and the remaining activations are classified as "losers." The state of each loser activation is merged with the state of the winner activation, and then the loser activations are removed.

[0093] Whenever the active cluster changes (i.e., a cluster joins), this part of the protocol runs, as in Fig. 9 The steps 902 of FIG. 1 and 14 generally represent the steps 902 of FIG. 14 . A consistent protocol may be used to decide changes to the active multi-cluster. Thus, each cluster in the new active multi-cluster runs the protocol and / or when one or more clusters join the active multi-cluster. If one or more clusters exit and no clusters join, then running the protocol does not make sense.

[0094] As shown in step 904, when the active multi-cluster of cluster C0 changes, cluster C0 collects any activations in state DOUBTFUL and sends RESOLVE_REQUEST(doubtful_set, C0) to other clusters in its active multi-cluster. Doubtful_set is the set of DOUBTFUL activations created by C0, as shown in Figure 2 In one or more implementations, each DOUBTFUL activation in a Double_Set is a pair of the form<V,A> , where A is the activation of component V.

[0095] When a cluster (e.g., C1) receives the message RESOLVE_REQUEST(Doubtful_Set, C0), Fig.10As shown in step 1002, cluster C1 creates an activated empty set (step 1004), for example named Loser_Set to return C0. Then, for each component in Doubtful_Set<V,A0> , (steps 1006 and 1020), if the component catalog of C1 as evaluated for component V in step 1008 contains a DOUBTFUL or OWNED activation A1, then this indicates that both C1 and C0 created activations for component V. One is the winner and the other is the loser. To decide which cluster is which, a precedence function similar to that described above may be used, as indicated by steps 1010 and 1012. If prec_func(V, C1) is greater than prec_func(V, C0), then C1 wins and in step 1014 will<V,A0> Add to Loser_Set. If prec_func(V, C1) is less than prec_func(V, C0), C1 removes A1 from its component catalog (step 1016) and terminates activation (after merging the state or at least persisting it for later merges) in step 1018. Once C1 has finished processing all Doubtful_Set<V,A0> Yes, step 1020 jumps to step 1022, where cluster C1 sends a message RESOLVE_RESPONSE(Loser_Set) to cluster C0.

[0096] return Fig. 9 , step 906, when C0 selects from each cluster C in its active multi-cluster i Receive RESOLVE_RESPONSE(Loser_Set i ), C0 will Loser_Set i Accumulate to Complete_Loser_Set (step 908). C0 removes all activations in Complete_Loser_Set from its component catalog (step 910). Step 912 represents merging and terminating those failed replicas.

[0097] Note that if C i Activation is never created for V, or if C i Create activation for V, but C0 takes precedence over C i And therefore C i Remove V from its component catalog, then cluster C i Do not add the DOUBTFUL activation of component V to the Loser_Set iIf the new active multicluster is equal to the entire multicluster (there was no split in step 914), then C0 has contacted every cluster in the system with respect to all of its DOUBTFUL components. If no cluster has activated component V, add the activation of component V to the Loser_Set i , then the activation state of C0 is converted to the state OWNED; either no cluster has created activations for V, or C0 takes precedence over all other clusters that have created activations for V. Therefore, C0 obtains the difference between Doubtful_Set and Complete_Loser_Set (step 916), and changes the state of each activation in the difference to OWNED (step 918).

[0098] Turning to aspects of individual server operation within a cluster, when creating an activation for a component, intra-cluster messaging between servers of the cluster is adapted to a geographically distributed cluster environment. Consider the presence of a server S0c in cluster C0 that attempts to create an activation A0 for component V. S0c first computes a consistent hash of component V to find out which server of the cluster contains the appropriate directory partition S0p. If S0p already contains an activation for component V, it returns the activation to the caller S0c. However, if S0p does not contain an activation for component V, it runs the activation creation protocol as described herein with reference to FIG. Figure 5 , 7 and 8 are generally described.

[0099] About running as reference Fig. 9 and Fig.10 In the generally described duplicate detection phase, one server S0g in cluster C0 may be assigned as a gateway server. The cluster's gateway server collects the set of DOUBTFUL activations from the servers in the cluster. S0g broadcasts a message to each server in C0 to ask each server for the set of DOUBTFUL activations in its component directory partition. Once the gateway server obtains the set of DOUBTFUL activations from each server, S0g then broadcasts the message RESOLVE_REQUEST(Doubtful_Set, C0) to each of the other cluster's gateway servers S1g, as described above with reference to Figure 7 Step 704 is generally described.

[0100] When receiving the message RESOLVE_REQUEST (Doubtful_Set, C0) from S0g, for example Fig.10In step 1002, S1g distributes a set of DOUBTFUL activations to the servers responsible for the appropriate component directory partitioning based on a consistent hash of each activated VirtualComponentId. The individual servers in cluster C1 resolve the activations as described herein. Each server in C1 also returns a set of failed party activations to S1g (or an empty set if the set of remote failed party activations is empty).

[0101] If S1g obtains the set of lost activations from each server in C1, S1g sends the complete set of lost activations to S0g using the message RESOLVE_RESPONSE(Loser_Set,true). If S1g does not obtain the set of lost activations from each server, it sends RESOLVE_RESPONSE(Loser_Set,false) to S0g. These two cases are distinguishable because C0 cannot convert a DOUBTFUL activation V to OWNED unless it is certain that every other cluster C1 has ensured that it does not contain activations for V, or that it contains activations for V, but according to prec_func, C0 takes precedence.

[0102] When S0g receives the message RESOLVE_RESPONSE from each cluster in its active multi-cluster, it deactivates the set of failed activations. However, S0g only transitions the set of activations in Doubtful_Set\Loser_Set to state OWNED if the following two conditions are true: 1) its active multi-cluster is equal to the entire multi-cluster, 2) all messages are of the form RESOLVE_RESPONSE(Loser_Set,true). If one or more received messages are of the form containing RESOLVE_RESPONSE(Loser_Set,false), it will not transition any DOUBTFUL activations to OWNED. If S0g receives the message RESOLVE_RESPONSE(Loser_Set,false) from gateway server S1g of remote cluster C1, S0g cannot be sure that all DOUBTFUL activations in C0 have been analyzed by the appropriate servers in cluster C1.

[0103] Note that the above description is only an example in which each cluster uses a single, distinct server as a gateway. However, in alternative implementations, multiple servers may be used for redundancy, and / or the set of DOUBTFUL activations may be divided between several gateway servers.

[0104] Regarding cluster membership, if server S0 in cluster C attempts to create activation A for component V, S0 first sets V=><A,REQUESTED_OWNERSHIP> S0 inserts an entry of the form into its component catalog and then sends an ACTIVATION_REQUEST message to each cluster in its active multicluster. If new servers are added to or removed from cluster C in the meantime, the range of VirtualComponentId hashes that S0 is responsible for may change. Therefore, a subset of the data in S0's component catalog partition may need to be moved to another server S1. Changes to the range of VirtualComponentId hashes that S0 is responsible for may occur while S0 is running the activation create protocol.

[0105] A component catalog entry for component V is allowed to be moved from server S0 to server S1 even when S0 is running an activation create protocol for component V. When an activation create protocol is in progress, the component catalog entry has the form V=><A,REQUESTED_OWNERSHIP> or V=><A,RACE_LOSER> . When all clusters respond with the message ACTIVATION_RESPONSE, S0 cannot find the component catalog entry corresponding to component V (because the component catalog entry may have migrated). At this point, S0 recognizes that the component catalog entry has moved to another server and stops running the activation creation protocol. When the server S1 to which the component catalog entry has migrated receives V=><A,REQUESTED_OWNERSHIP> or V=><A,RACE_LOSER> When S1 receives a component catalog entry of the form RACE_LOSER, it reruns the activation creation protocol. In order to run the activation creation protocol, the ActivationState of the component catalog entry corresponding to the activation needs to be in state REQUESTED_OWNERSHIP. Because S1 may have received a component catalog entry in state RACE_LOSER, S1 needs to first change the state of the activation to REQUESTED_OWNERSHIP; S1 delays to allow the cluster that won the race condition to take ownership of the component, and then transitions the state of the component catalog entry to REQUESTED_OWNERSHIP.

[0106] S1 only needs to run the activation creation protocol for component catalog entries with ActivationState REQUESTED_OWNERSHIP or RACE_LOSER. Component catalog entries with an activation state of DOUBTFUL, OWNED, or CACHED do not require any special processing.

[0107] Turning to the other side, some (perhaps most) components are referenced only from within a single cluster (i.e., the cluster in which the component is activated). To ensure that there is only a single activation of a particular component globally, the protocol causes a cluster attempting to create an activation of component V to check whether any other cluster in its active multi-cluster has already created an activation for V. If V is a locally referenced component, then no cluster other than the creating cluster will reference V. For locally referenced components, such as V, cluster C checks for activation of V with all clusters in its cluster's active multi-cluster, even though C is the only cluster that has ever referenced V. The cost of querying each cluster in C's active multi-cluster about V is a single WAN round trip. The network latency cost of a WAN round trip is typically several orders of magnitude higher than a round trip between two nodes within a cluster.

[0108] To avoid paying the cost of a WAN round trip on each locally referenced component V, cluster C can optimistically create an activation for V before running the activation creation protocol. C then runs the protocol asynchronously, while local activation service requests come from servers in C. If C finds that all other clusters in its active multicluster respond with ACTIVATION_RESPONSE(PASS,null), then C's optimistic activation for V is DOUBTFUL or OWNED, as described herein. Conversely, if C finds that one of the clusters in its active multicluster has already activated V, then C terminates its local activation for V and adds a reference to the remote activation in its component catalog.

[0109] Aspects of the present disclosure are operable with any combination of the following. Examples of providing single-instance components in a multi-cluster distributed system of a geographically distributed computing environment are described herein. Components are always addressable by applications through the identity of the component, including when clusters are partitioned from one another.

[0110] When the computing environment does not have clusters that are separated from each other, only another single instance component in the multi-cluster distributed system can be activated in the cluster. When the cluster performing the activation is part of a cluster quorum, only another single instance component can be activated in the cluster.

[0111] Further described herein is eliminating any replica components to provide only a single surviving instance of the component when the partitioned cluster is no longer partitioned. Eliminating any replica components includes performing a deterministic operation of selecting a surviving single instance of the component.

[0112] Another aspect relates to preventing a race condition in which at least two undivided clusters simultaneously attempt to activate a component, including detecting a race condition state and selecting only one winning component instance for activation. Preventing the race condition includes performing a deterministic operation of selecting a winning component instance for activation.

[0113] In one or more aspects, a distributed computing system includes a plurality of clusters, each cluster having a runtime executing in at least one server memory on at least one processor. The runtime is configured to prevent race conditions in which two or more clusters attempt to activate a component simultaneously and allow duplicate components to exist when the cluster is partitioned, including associating state data with each possible replica to indicate possible replica states. The runtime evaluates the state data to remove any duplicate components when the cluster is no longer partitioned so that only one component survives for any replica that exists.

[0114] For a given component (or type), the runtime may be configured to prevent activation of another component in the event that a partition exists between clusters of the distributed computing system.

[0115] The runtime may be configured to optimistically activate a component before every other cluster has responded as to whether the component is activated on another cluster.

[0116] The runtime prevents race conditions by transmitting an activation request to each other cluster with which the requesting cluster can communicate, the activation request indicating the requesting cluster's intent to activate the component. When another cluster determines that the requesting cluster is in a race condition with the other cluster, the other cluster uses a tie-breaker mechanism to determine which activation request is to succeed.

[0117] The runtime removes any replica components by exchanging the set of possible replicas to every other cluster that the cluster can communicate with, and using a tie-breaker mechanism to determine which replica component survives and which replica or replicas are to be terminated for any replicas that exist. The runtime can be configured to change state data to indicate sole ownership of a cluster when no clusters are partitioned from one another.

[0118] One or more aspects relate to attempting to determine in an attempting cluster whether a component has been activated in any other cluster with which the attempting cluster can communicate, and if so, using the already activated component. If not yet activated, a determination is made as to whether a partition exists between at least two clusters. If a partition exists, the component is activated and associated with state data indicating that the activated component is a possible duplicate.

[0119] One or more aspects involve determining that at least one previously split cluster is no longer split and exchanging a set of possible replicas with at least one other cluster. If replica components exist, one component is selected to survive. When no clusters are split from each other, the state data may be changed from indicating possible replicas to indicating sole ownership of the cluster.

[0120] One or more aspects relate to detecting a race condition where at least two undivided clusters attempt to activate a component simultaneously, and selecting only one winning component instance for activation.

[0121] Also described is detecting a partition and allowing activation of another component only if the cluster attempting activation is part of a quorum of clusters.

[0122] In a geographically distributed computing environment, an apparatus is described for providing a single-instance component in a multi-cluster distributed system, wherein the component is always addressable by an application via the identity of the component, the component being addressable to the application including when the clusters are partitioned from one another. Also described is an apparatus for attempting to determine in an attempting cluster whether the component has been activated in any other cluster with which the attempting cluster can communicate, and if so, means for using the already activated component, and if not yet activated, means for determining whether a partition exists between at least two clusters, and if a partition exists, means for activating the component and associating the component with state data indicating that the activated component is a possible replica.

[0123] Example Networked and Distributed Environments

[0124] It will be appreciated by those skilled in the art that the various implementations and methods described herein can be implemented in conjunction with any computer or other client or server device, which can be deployed as a part of a computer network or in a distributed computing environment, and can be connected to one or more data repositories of any kind. In this regard, the various implementations described herein can be implemented in any computer system or environment with any number of memories or storage units and any number of applications and processes occurring across any number of storage units. This includes but is not limited to an environment with a server computer and a client computer with remote or local storage devices deployed in a network environment or a distributed computing environment. It should be understood that the system setting can be a local mode operation (e.g., directly on hardware) or a virtualized environment, or a combination of the two.

[0125] Distributed computing provides sharing of computer resources and services through communication exchanges between computing devices and systems. These resources and services include information exchange for objects (e.g., files), cache storage, and disk storage. These resources and services also include sharing of processing power across multiple processing units for load balancing, resource expansion, specialization of processing, etc. Distributed computing exploits network connectivity to allow clients to leverage their collective capabilities to benefit the entire enterprise. In this regard, various devices may have applications, objects, or resources that may participate in resource management mechanisms, as described for various implementations of the present disclosure.

[0126] Fig.11A schematic diagram of an example networked or distributed computing environment is provided. The distributed computing environment includes computing objects 1110, 1112, etc. and computing objects or devices 1120, 1122, 1124, 1126, 1128, etc., which may include programs, methods, data storage, programmable logic, etc. as represented by example applications 1130, 1132, 1134, 1136, 1138. It is understood that computing objects 1110, 1112, etc. and computing objects or devices 1120, 1122, 1124, 1126, 1128, etc. may include different devices, such as personal digital assistants (PDAs), audio / video devices, mobile phones, MP3 players, personal computers, laptop computers, etc.

[0127] Each computing object 1110, 1112, etc. and computing objects or devices 1120, 1122, 1124, 1126, 1128, etc. can communicate with one or more other computing objects 1110, 1112, etc. and computing objects or devices 1120, 1122, 1124, 1126, 1128, etc. directly or indirectly via a communication network 1140. Fig.11 1140 is shown as a single element. However, the communication network 1140 may include Fig.11 Other computing objects and computing devices that provide services for the system, and / or may represent multiple interconnected networks that are not shown. Each computing object 1110, 1112, etc. or computing objects or devices 1120, 1122, 1124, 1126, 1128, etc. may also include applications, such as applications 1130, 1132, 1134, 1136, 1138, which may utilize an API or other object, software, firmware and / or hardware suitable for communicating with or implementing an application provided according to various implementations of the present disclosure.

[0128] There are various systems, components, and network configurations that support distributed computing environments. For example, computing systems can be connected together by wired or wireless systems, by local networks or widely distributed networks. Currently, many networks are coupled to the Internet, which provides the infrastructure for widely distributed computing, and includes many different networks, although any network infrastructure can be used for communication that occurs in the systems described in the various examples.

[0129] Thus, hosts with network topologies and network infrastructures such as client / server, peer-to-peer, or hybrid architectures can be utilized. A "client" is a member of a class or group that uses the services of another class or group to which it is not related. A client can be a process that requests a service, such as a roughly set of instructions or tasks, provided by another program or process. The client process utilizes the requested service without having to "know" any working details about the other program or service itself.

[0130] In a client / server architecture, particularly a networked system, a client is typically a computer that accesses shared network resources provided by another computer (e.g., a server). Fig.11 In the illustration, as a non-limiting example, computing objects or devices 1120, 1122, 1124, 1126, 1128, etc. can be considered clients, and computing objects 1110, 1112, etc. can be considered servers, where computing objects 1110, 1112, etc. acting as servers provide data services, such as receiving data from client computing objects or devices 1120, 1122, 1124, 1126, 1128, etc., storing data, processing data, and transmitting data to client computing objects or devices 1120, 1122, 1124, 1126, 1128, etc., but any computer can be considered a client, a server, or both, depending on the circumstances.

[0131] A server is typically a remote computer system accessible through a remote or local network (such as the Internet or a wireless network infrastructure). A client process may be active in a first computer system, and a server process may be active in a second computer system, communicating with each other through a communication medium, thereby providing distributed functionality and allowing multiple clients to utilize the information gathering capabilities of the server.

[0132] In a network environment where the communication network 1140 or bus is the Internet, for example, computing objects 1110, 1112, etc. may be web servers that communicate with other computing objects or devices 1120, 1122, 1124, 1126, 1128, etc. via any of a variety of known protocols such as the Hypertext Transfer Protocol (HTTP). As may be characteristic of a distributed computing environment, computing objects 1110, 1112, etc. that act as servers may also function as clients, such as computing objects or devices 1120, 1122, 1124, 1126, 1128, etc.

[0133] Example computing device

[0134] As mentioned above, the techniques described herein can advantageously be applied to any device. Therefore, it will be appreciated that all kinds of handheld, portable, and other computing devices and computing objects are contemplated for use in conjunction with the various implementations. Fig.12 The general purpose remote computer described below is just one example of a computing device.

[0135] Examples may be implemented in part via an operating system for use by developers of services for devices or objects, and / or included within application software that operates to perform one or more functional aspects of the various implementations described herein. Software may be described in the general context of computer-executable instructions, such as program modules, executed by one or more computers, such as client workstations, servers, or other devices. Those skilled in the art will appreciate that computer systems have various configurations and protocols that can be used to transmit data, and therefore, any particular configuration or protocol is not considered limiting.

[0136] Fig.12 Thus, an example of a suitable computing system environment 1200 is shown in which one or more aspects of the examples described herein may be implemented, but as mentioned above, the computing system environment 1200 is only one example of a suitable computing environment and is not intended to suggest any limitation as to scope of use or functionality. Additionally, the computing system environment 1200 is not intended to be interpreted as having any dependency related to any one or combination of components shown in the example computing system environment 1200.

[0137] refer to Fig.12 An example remote device for implementing one or more implementations includes a general purpose computing device in the form of a computer 1210. Components of the computer 1210 may include, but are not limited to, a processing unit 1220, a system memory 1230, and a system bus 1222 that couples various system components including the system memory to the processing unit 1220.

[0138] Computer 1210 typically includes a variety of machine / computer readable media and may be any available media that can be accessed by computer 1210. System memory 1230 may include computer storage media in the form of volatile and / or nonvolatile memory, such as read-only memory (ROM) and / or random access memory (RAM). By way of example and not limitation, system memory 1230 may also include an operating system, application programs, other program modules, and program data.

[0139] A user can enter commands and information into the computer 1210 through input devices 1240. A monitor or other type of display device is also connected to the system bus 1222 via an interface, such as output interface 1250. In addition to the monitor, the computer may also include other peripheral output devices such as speakers and printers, which may be connected through output interface 1250.

[0140] The computer 1210 may operate in a networked or distributed environment using logical connections to one or more other remote computers, such as a remote computer 1270. The remote computer 1270 may be a personal computer, a server, a router, a network PC, a peer device or other public network node or any other remote media consumption or transmission device, and may include any or all of the elements described above with respect to the computer 1210. Fig.12 The logical connections shown include a network 1272, such as a local area network (LAN) or a wide area network (WAN), but other networks / buses may also be included. Such networking environments are common in homes, offices, enterprise-wide computer networks, intranets, and the Internet.

[0141] As noted above, although example implementations have been described in conjunction with various computing devices and network architectures, the underlying concepts may be applied to any network system and any computing device or system where it is desirable to improve resource usage efficiency.

[0142] In addition, there are many ways to implement the same or similar functionality, such as appropriate APIs, toolkits, driver code, operating systems, controls, standalone or downloadable software objects, etc. that enable applications and services to utilize the techniques described herein. Thus, the examples herein are considered from the perspective of an API (or other software object) as well as from a software or hardware object that implements one or more examples as described herein. Thus, the various examples described herein may have aspects that are fully hardware, partially hardware, partially software, and software.

[0143] As used herein, the word "exemplary" means used as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to these examples. In addition, any aspect or design described herein as "exemplary" is not necessarily to be construed as being preferred or advantageous over other aspects or designs, nor is it intended to exclude equivalent exemplary structures and techniques known to those of ordinary skill in the art. Furthermore, for the avoidance of doubt, to the extent the terms "containing," "having," "including," and other similar words are used, these terms are intended to be inclusive in a manner similar to the term "comprising," as open transition words, and do not exclude any additional or other elements when used in the claims.

[0144] As described above, the various techniques described herein can be implemented in conjunction with hardware or software or by a combination of the two when appropriate. As used herein, the terms "component", "module", "system", etc. are also intended to refer to a computer-related entity, which is hardware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process, a processor, an object, an executable file, an execution thread, a program, and / or a computer running on a processor. As an illustration, both an application running on a computer and a computer can be a component. One or more components can reside in a process and / or an execution thread, and a component can be located on one computer and / or distributed between two or more computers.

[0145] The above-described systems have been described with respect to the interaction between several components. It will be appreciated that such systems and components may include these components or specified subcomponents, some of the specified components or subcomponents, and / or additional components, and based on various permutations and combinations of the foregoing. Subcomponents may also be implemented as components that are communicatively coupled to other components rather than being included in a parent component (layering). In addition, it may be noted that one or more components may be combined into a single component that provides aggregate functionality or may be divided into several separate subcomponents, and any one or more intermediate layers (e.g., management layers) may be provided to communicatively couple to such subcomponents in order to provide integrated functionality. Any component described herein may also interact with one or more other components that are not specifically described herein but are generally known to those skilled in the art.

[0146] In view of the example system described herein, the method that can be implemented according to the described subject can also be understood with reference to the flowchart of each accompanying drawing. Although for the purpose of simplifying the description, the method is shown and described as a series of frames, it should be understood and appreciated that various examples are not limited by the order of the frames, because some frames can occur in a different order than that depicted and described herein, and / or occur simultaneously with other frames. In the case of non-sequential or branching processes shown by flow charts, it is understood that the order of various other branches, process paths and frames that achieve the same or similar results can be implemented. In addition, some illustrated frames are optional when implementing the method described below.

[0147] in conclusion

[0148] Although the present invention is susceptible to various modifications and alternative constructions, certain illustrated embodiments thereof are shown in the drawings and have been described in detail above. However, it should be understood that it is not intended to limit the invention to the specific forms disclosed, but rather, the present invention is intended to cover all modifications, alternative constructions and equivalents that fall within the spirit and scope of the present invention.

[0149] In addition to the various embodiments described herein, it should be understood that other similar embodiments may be used, or one or more of the described embodiments may be modified and added to perform the same or equivalent functions of the corresponding one or more embodiments. In addition, multiple processing chips or multiple devices may share the performance of one or more functions described herein, and similarly, storage may be implemented across multiple devices. Therefore, the present invention is not limited to any single embodiment, but is interpreted in breadth, spirit and scope according to the appended claims.

Claims

1. A method for managing components across multiple clusters, the method comprising: detecting the presence of a split between a first split cluster and other clusters in the plurality of clusters; receiving a request from an application to use or activate a component in the first partitioned cluster, the first partitioned cluster not having an in-memory instance of the component; automatically creating a new instance of the component on an available cluster in the first partitioned cluster in response to the request to use or activate the component; following the detected existence of the split between the first split cluster and the other cluster, assigning state data to the new instance of the component, the state data indicating that the new instance of the component has a possibility of being replicated on other servers outside of the split cluster; broadcasting the created new instance of the component and the allocated state data to other devices within the partition cluster; receiving a notification that another device in the split cluster created a second instance of the component and assigned the state data indicating that the second instance of the component has a possibility of being replicated on the other server outside the split cluster; as well as Based on the state data assigned to the second instance of the component, the new instance of the component is deleted.

2. The method according to claim 1, further comprising: determining that the first split cluster is part of a quorum among the plurality of clusters; as well as Based on the first split cluster being part of the quorum, the component is activated.

3. The method according to claim 1, further comprising: detecting that the segmentation is removed; determining that a third instance of the component has been instantiated on a previously segmented cluster; as well as Based on the third instance, the new instance of the component is eliminated.

4. The method of claim 3, wherein said eliminating said new instance of said component comprises: A deterministic operation is performed to select a surviving single instance of the component.

5. The method according to claim 1, further comprising: identifying that the first split cluster is operating in a high consistency mode from rules associated with the runtime; as well as The automatic creation of the new instance of the component is performed with the first partition cluster operating in the high consistency mode.

6. The method according to claim 5, further comprising: detecting a race condition with a second instance of the component when the split is removed; as well as A winning instance of the component is selected based on the state data associated with the created new instance and the second instance of the component.

7. The method according to claim 1, further comprising: detecting that the segmentation is removed; determining that a third instance of the component has been instantiated on a previously segmented cluster; as well as Based on the new instance of the component, the third instance of the component is eliminated.

8. A system for managing components across multiple clusters, comprising: server storage associated with multiple servers configured in a multi-cluster environment; as well as at least one processor, the at least one processor executing at least one runtime, the at least one runtime executing the server memory, the at least one runtime configured to: providing, via an identification of a component addressable by an application, the component; detecting the presence of a partition separating the plurality of servers in the multi-cluster environment; Allows replica components on clusters that cannot communicate due to partitioning; upon said detecting of said existence of said partition, activating said component and assigning to said component state data in a component catalog of a first partition cluster, said state data indicating that said activation of said component is a candidate replica for another component on another server within said multi-cluster environment, broadcasting the created new instance of the component and the allocated state data to other devices within the partition cluster; receiving a notification that another device in the split cluster created a second instance of the component and assigned the state data indicating that the second instance of the component has a possibility of being replicated on the other server outside the split cluster; as well as Based on the state data assigned to the second instance of the component, the new instance of the component is deleted.

9. The system of claim 8, wherein upon receiving a request to use the component, the component is not instantiated in the server memory, and based on the received request, the component automatically creates the new instance of the component in the server memory.

10. The system of claim 8, wherein the at least one runtime is configured to activate the component before other clusters have responded as to whether the component is activated on another cluster.

11. A system according to claim 8, wherein the at least one runtime prevents a race condition among different instances of the component by transmitting an activation request to other clusters, the requesting cluster being able to communicate with the other clusters, the activation request indicating the requesting cluster's intention to activate at least one of the replica components.

12. The system of claim 11, wherein the at least one runtime is further configured to perform a tie-breaker mechanism to resolve a race condition between the plurality of activations of the component after the partition between the different clusters is removed.

13. The system of claim 8, wherein the at least one runtime further removes one or more replica components by exchanging a set of candidate replicas with other clusters with which the cluster is able to communicate.

14. The system of claim 8, wherein the at least one runtime is configured to change the state data to indicate exclusive ownership of clusters when no clusters are partitioned from each other.

15. One or more machine-readable storage media devices embodied with machine-executable instructions, which, when executed by one or more processors, perform operations for reducing duplicate virtual components in a server cluster, the operations comprising: determining, by a computing device, that a component has not been instantiated on a server cluster with which the computing device is able to communicate; detecting a partition separating the server cluster from other servers in an overall multi-cluster group of servers; following said detection of said partition, directing a runtime to activate said component and assigning to said component in a component catalog of a first partition cluster state data indicating that said activation of said component is a candidate replica of another component on another server external to said partition cluster; broadcasting the created new instance of the component and the allocated state data to other devices within the partition cluster; receiving a notification that another device in the split cluster created a second instance of the component and assigned the state data indicating that the second instance of the component has a possibility of being replicated on the other server outside the split cluster; as well as Based on the state data assigned to the second instance of the component, the new instance of the component is deleted.

16. The one or more machine-readable storage media devices of claim 15, having further machine-readable instructions comprising: It is determined that at least one previously split cluster is no longer split, and a set of candidate replicas is exchanged with at least one other cluster.

17. The one or more machine-readable storage media devices of claim 15 having additional machine-readable instructions comprising: Determines that replica components exist, and selects one component to survive.

18. The one or more machine-readable storage media devices of claim 15 having additional machine-readable instructions comprising: A race condition is detected in which at least two undivided clusters simultaneously attempt to activate the component, and only one winning component instance is selected for activation.

19. The one or more machine-readable storage media devices of claim 16 having additional machine-readable instructions comprising: A partition is detected and activation of another component is allowed only if the cluster attempting the activation is part of a cluster quorum.

20. The one or more machine-readable storage media devices of claim 16 having further machine-readable instructions for: detecting that the segmentation is removed; determining that a third instance of the component has been instantiated on a previously partitioned cluster; and Based on the new instance of the component, the third instance of the component is eliminated.

Images