A cluster system and a method for cluster system

Abstract

The present disclosure relates to a cluster system and a method for a cluster system. Techniques for a cluster system that is aware of local geographic locations. Such a cluster system utilizes this awareness to determine subsets of nodes in a physical configuration that are located in different geographic locations, to optimize workload placement based on geographic location, to make failover and failback decisions based on geographic location, and to manage assigned voting and pruning nodes for a specified number based on geographic location. Such capabilities result in a cluster system that is more resilient and more efficient in resource usage than a cluster system that does not have such local geographic location awareness.

Description

[0001] Related Applications Citations

[0002] This application is a divisional application of the invention patent application with international application number PCT / US2016 / 047123, international application date August 16, 2016, date of entry into the Chinese national phase February 2, 2018, Chinese national application number 201680047877.3, and invention name “Optimal storage device and workload placement and high elasticity in geographically distributed cluster systems”. Technical Field

[0003] The present disclosure relates to a cluster system. Background Art

[0004] Customers deploy cluster systems across different geographic locations ("geo sites") for disaster recovery and higher resiliency, as well as to ensure that the geo sites are co-located with the customer (while using other geo sites for backup / DR). But today's cluster systems are not aware of geography. That is, they do not inherently identify which subset of cluster nodes is located in one physical location and which subset of nodes is located in another physical location. Instead, administrators must manually specify such node subsets, and even then, the cluster system does not intelligently use this information (or even use it at all) in cluster configuration, workload placement, or pruning dynamic quorum maintenance.

[0005] For example, when workloads are distributed across cluster nodes based on traditional factors such as CPU or memory utilization, the resulting workload distribution can be very inefficient from a geographic location perspective, especially when clients are located in one geographic location, the client's workload is placed in a remote geographic location, and the workload's storage is placed in another distant geographic location. Ideally, we would like the client, workload, and storage to be located in the same geographic location so that network traffic does not have to cross geographic locations. Bandwidth across geographic locations is typically more expensive and more prone to link and router / switch failures. Summary of the invention

[0006] The Summary of the Invention provided in this section summarizes one or more partial or complete example embodiments of the technology described herein in order to provide the reader with a basic high-level understanding. This Summary of the Invention is not an extensive description of the technology, nor does it identify key elements or aspects of the technology, nor does it describe the scope of the technology. Its sole purpose is to present various aspects of the technology in a simplified form as a preface to the specific embodiments provided below. The technology as a whole should not be limited to any specific embodiment or example or combination thereof provided herein.

[0007] The present invention encompasses techniques for a cluster system that is aware of local geographic locality. Such a cluster system utilizes such awareness to determine subsets of nodes in a physical configuration that are located at various geographic localities, to manage the placement of workloads based on geographic locality, to make failover and failback decisions based on geographic locality, and to prune nodes for quorum management based on geographic locality. Such capabilities result in a cluster system that is more resilient and more efficient in resource usage than a cluster system that does not have such local geographic locality awareness. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The detailed description provided below will be better understood when considered in conjunction with the accompanying drawings, in which:

[0009] Figure 1 is a block diagram illustrating an example computing environment 100 in which the techniques described herein may be implemented;

[0010] Figure 2 is a block diagram of an example system 200 configured to implement aspects of the present invention;

[0011] Figure 3 is a block diagram illustrating an example method 300 for deriving geographic locales for a cluster and identifying physical cluster resources (eg, nodes and storage devices) located at each geographic locale;

[0012] Figure 4 is a block diagram illustrating an example method 400 for placement and / or failover of groups (hosted workloads) in a geographically site-aware cluster system, such as cluster system 600;

[0013] Figure 5 is a block diagram illustrating an example method 500 for maintaining a prescribed population via a geographic venue-aware clustering system; and

[0014] Figure 6 is a block diagram of an example geo-venue-aware cluster system 600 including three example geo-venues A 610 , B 620 , and C 630 , and an example witness 640 . DETAILED DESCRIPTION

[0015] This section describes one or more partial or all example embodiments of the present technology in conjunction with the specific implementations provided in the accompanying drawings, but is not intended to describe all possible embodiments of the present technology. The detailed description sets forth various examples of at least some systems and / or methods of these technologies. However, the same or equivalent technologies, systems and / or methods can also be implemented according to the examples.

[0016] Computing Environment

[0017] Although the examples provided herein are described and illustrated as being implementable in a computing environment, the described environment is provided only as an example and not as a limitation. As will be appreciated by those skilled in the art, the disclosed examples are suitable for implementation in a variety of different computing environments.

[0018] Figure 1 1 is a block diagram illustrating an example computing environment 100 in which the techniques described herein may be implemented. A suitable computing environment may be implemented using any of a number of general or special purpose devices and / or systems. Examples of such devices / systems include, but are not limited to, personal digital assistants ("PDAs"), personal computers ("PCs"), handheld or laptop devices, microprocessor-based systems, multiprocessor systems, systems on a chip ("SOCs"), servers, Internet services, workstations, consumer electronic devices, cell phones, set-top boxes, and the like. In all cases, these systems are strictly limited to articles of manufacture, etc.

[0019] The computing environment 100 generally includes at least one computing device 101 coupled to various components such as peripheral devices 102, 103, 101, etc. These may include components of an input device 103, such as voice recognition technology, a touch pad, buttons, a keyboard, and / or a pointing device such as a mouse or trackball, which may be operated via one or more input / output ("I / O") interfaces 112. Components of the computing device 101 may include one or more processors (including a central processing unit ("CPU"), a graphics processing unit ("GPU"), a microprocessor ("μP"), etc.) 107, a system memory 109, and a system bus 108 that is generally coupled to the various components. The processor 107 generally processes or executes various computer-executable instructions and controls the operation of the computing device 101 based on these instructions. This may include the computing device 101 communicating with other electronic and / or computing devices, systems, or environments (not shown) via various communication technologies such as a network connection 114, etc. The system bus 108 represents any number of bus structures, including a memory bus or memory controller, a peripheral bus, a serial bus, an accelerated graphics port, a processor or local bus using any of a variety of bus architectures, and the like. The term "computing device" as used herein, unless preceded by the word "virtual", strictly refers to a physical article, etc. The terms "virtual computing device" and "virtual machine" generally refer to an emulation of a specific computing system, and are generally referred to as virtual machines ("VMs"). These terms are widely used to include operating system-level virtualization ("containers"), where in one example, the kernel of an operating system allows multiple isolated user space instances instead of just one, and where in another example, both the kernel and the user space are isolated. These operational VMs are ultimately hosted on non-virtual computing devices.

[0020] The system memory 109 may include computer-readable media memory in the form of volatile memory such as random access memory ("RAM") and / or non-volatile memory such as read-only memory ("ROM") or flash memory ("FLASH"). A basic input / output system ("BIOS") may be stored in a non-volatile or similar manner. The system memory 109 typically stores data, computer-executable instructions, and / or program modules that include computer-executable instructions that are immediately accessible and / or currently being operated on by one or more processors 107. The term "system memory" as used herein refers strictly to a physical article of manufacture or the like.

[0021] Mass storage devices 104 and 110 may be coupled to computing device 101 via coupling to a system bus or incorporated into computing device 101. Such mass storage devices 104 and 110 may include non-volatile RAM, a disk drive that reads from and / or writes to a removable, non-volatile magnetic disk (e.g., a "floppy disk") 105, and / or an optical disk drive that reads from and / or writes to a non-volatile optical disk such as a CD ROM, DVD ROM 106. Alternatively, a mass storage device such as hard disk 110 may include a non-removable storage medium. Other mass storage devices may include memory cards, memory sticks, tape storage devices, etc. As used herein, the term "mass storage device" strictly refers to a physical article of manufacture or the like.

[0022] Any number of computer programs, files, data structures, etc. may be stored in mass storage 110, other storage devices 104, 105, 106 and system memory 109 (typically limited by available space), including by way of example and not limitation, operating systems, application programs, data files, directory structures, computer executable instructions, etc.

[0023] Output components or devices such as display device 102 may be coupled to computing device 101 generally via an interface such as display adapter 111. Output device 102 may be a liquid crystal display ("LCD"). Other example output devices may include a printer, audio output, voice output, cathode ray tube ("CRT") display, tactile device or other sensory output mechanism, etc. Output devices may enable computing device 101 to interact with a human operator or other machine, system, computing environment, etc. A user may interface with computing environment 100 via any number of different I / O devices 103 such as a touchpad, buttons, keyboard, mouse, joystick, gamepad, data port, etc. These and other I / O devices may be coupled to processor 107 via an I / O interface 112, which may be coupled to system bus 108, and / or may be coupled through other interfaces and bus structures such as a parallel port, game port, universal serial bus ("USB"), firewire, infrared ("IR") port, etc.

[0024] The computing device 101 may operate in a networked environment via a communications connection to one or more remote computing devices through one or more cellular networks, wireless networks, local area networks (“LANs”), wide area networks (“WANs”), storage area networks (“SANs”), the Internet, radio links, optical links, etc. The computing device 101 may be coupled to a network via a network adapter 113 or the like, or alternatively via a modem, a digital subscriber line (“DSL”) link, an integrated services digital network (“ISDN”) link, an Internet link, a wireless link, etc.

[0025] A communication connection 114, such as a network connection, typically provides a coupling to a communication medium, such as a network. Communication media typically provide computer-readable and computer-executable instructions, data structures, files, program modules, and other data using a modulated data signal, such as a carrier wave or other transport mechanism. The term "modulated data signal" typically means a signal that has one or more of its characteristics set or changed in such a manner as to encode information into the signal. By way of example and not limitation, communication media may include: wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, radio frequency, infrared or other wireless communication mechanisms.

[0026] The power supply 190, such as a battery or a power supply, typically provides power for part or all of the computing environment 100. In the case where the computing environment 100 is a mobile device or portable device, etc., the power supply 190 may be a battery. Alternatively, in the case where the computing environment 100 is a desktop computer or server, etc., the power supply 190 may be a power supply designed to be connected to an alternating current ("AC") source, such as via a wall outlet.

[0027] Some mobile devices may only include a combination of Figure 1 Some of the components described. For example, an electronic badge may be composed of a coil or the like, together with a simple processing unit 107 or the like, the coil being configured to function as a power source 190 when in proximity to a card reader device or the like. Such a coil may also be configured to act as an antenna coupled to the processing unit 107 or the like, the coil antenna being capable of transmitting / receiving communications between the electronic badge and another device such as a card reader device. Such communications may not involve networking, but may also be general or specialized communications via telemetry, point-to-point, RF, IR, audio, or other means. The electronic card may not include a display 102, an I / O device 103, or may be combined with a display 102, an I / O device 103, or a plurality of other devices. Figure 1 Many other components described. Figure 1 Other mobile devices for many of the components described include, by way of example and not limitation, electronic bracelets, electronic tags, implantable devices, and the like.

[0028] Those skilled in the art will recognize that storage devices for providing computer-readable and computer-executable instructions and data can be distributed over a network. For example, a remote computer or storage device can store computer-readable instructions and computer-executable instructions in the form of software applications and data. A local computer can access a remote computer or storage device via a network and download some or all of the software applications or data, and can execute any computer-executable instructions. Alternatively, a local computer can download pieces of software or data as needed, or process software in a distributed manner by executing some instructions at a local computer and some instructions at a remote computer and / or device.

[0029] Those skilled in the art will also recognize that, by utilizing conventional techniques, all or a portion of the computer-executable instructions of the software may be performed by a dedicated electronic circuit such as a digital signal processor ("DSP"), a programmable logic array ("PLA"), discrete circuits, etc. The term "electronic device" may include a computing device or consumer electronic device that includes any software, firmware, etc., or an electronic device or circuit that does not include software, firmware, etc.

[0030] The term "firmware" generally refers to executable instructions, codes, data, applications, programs, program modules, etc. maintained in an electronic device such as a ROM. The term "software" generally refers to computer-executable instructions, codes, data, applications, programs, program modules, etc. maintained in or on any form or type of computer-readable medium that is configured to store computer-executable instructions, etc. in a manner accessible to a computing device. The terms "computer-readable medium," "computer-readable media," and the like as used herein are limited to strictly refer to one or more specified devices, articles of manufacture, etc. that are not themselves signals or carrier waves. The term "computing device," as used in the claims, is limited to strictly refer to one or more specified devices, articles of manufacture, etc. that are not themselves signals or carrier waves, such as a computing device 101 covering client devices, mobile devices, one or more servers, network services such as Internet services or enterprise network services based on one or more computers, and / or any combination thereof.

[0031] The terms "cluster node" or "node" and the like refer to a physical resource, such as a computing device 101, that is configured as part of a cluster system. Such nodes typically execute specific cluster node software. A cluster system typically includes multiple such physical cluster nodes, some of which may be physically located at one location, while other physical cluster nodes may be physically located at one or more other locations. Such physical locations are typically physically and geographically distinct and are referred to herein as "geographic locations." For example, several exemplary nodes AC of an example cluster are physically located at a geographic location A in Seattle, while other exemplary nodes DF of the same cluster are physically located at a geographic location B in Dallas (see, e.g., FIG. 1 ). Figure 6 ). The term "cluster system" and the like as used herein generally refers to a conventional failover cluster system and the like including the various novel and unique geographic location enhancements of the present invention. The term "cluster environment" as used herein generally refers to an environment in which a cluster system according to the present invention operates, wherein the environment includes the various elements, services, data and / or other resources required for the geographic location enhancement to function as required.

[0032] The nodes and workloads of a cluster system typically utilize at least two types of "storage devices": file shares and cluster shared volumes ("CSV"). The terms "shares" and "file shares" used herein generally refer to traditional file shares. Typically, cluster nodes (and any VMs operating on cluster nodes) can access file shares given appropriate configuration (e.g., establishing drive mappings, etc.). The term "cluster shared volumes" used herein generally refers to shared storage devices accessible to all nodes. In one example, a virtual hard disk ("VHD") of a virtual machine ("VM") is created or copied to (hosted on) a CSV. Multiple VHDs can be placed on a CSV, which in turn is associated with multiple VMs that can run on different nodes of the cluster. The term "dependent storage device" used herein refers to a specific storage device that a specific VM or other workload depends on, such as a specific CSV or file share, or a specific VHD hosted on a specific CSV or file share, etc.

[0033] The term "workload" as used herein generally refers to a client application hosted on a cluster system. Such client applications include, but are not limited to, virtual machines ("VMs"), storage devices such as CSV and file sharing, any type of hosted client application, and any combination of the foregoing, also referred to herein as "virtual resources". Non-limiting examples of some client applications that clients may wish to have hosted on a cluster system include database applications (e.g., SQL Server), email services, storage services (e.g., archive and backup services), accounting packages, websites, office suites, and any other software or services or functions that can be hosted on a cluster system. "Hosted" means that the client's workload is operated and used by the cluster system on behalf of the client and according to the client's needs. The term "group" used herein in the context of workload generally refers to a specific hosted workload, that is, a specific workload hosted by a cluster system. Workloads that rely on a specific storage device (which may be different workloads or parts of different workloads) are said to have a "storage association" with the storage. If the specific storage is hosted on a specific geographic location, the geographic location is referred to as a "storage association location" related to the workload.

[0034] Nodes of a cluster can be "online" or "offline". The term "online", when applied to a cluster node (or witness, etc.), indicates that the node is currently operating as part of the cluster. The term "offline", when applied to a cluster node (or witness, etc.), indicates that the node is not currently operating as part of the cluster. An offline node may have failed due to some failure (i.e., an "unplanned failure") and is therefore referred to as "unplanned offline", or an offline node may have been intentionally taken offline, such as according to some plan (e.g., planned maintenance, etc.), and is therefore referred to as "planned offline".

[0035] Figure 2 2 is a block diagram of an example system 200 configured to implement aspects of the present invention. The system 200 generally includes several components, including a geo-site physical configuration manager ("GPCM") 210, a resource manager ("RM") 220, a geo-site virtual configuration manager ("GVCM") 230, a geo-site provisioned quantity manager ("GQM"), a network topology manager ("NTM") 250, a storage topology manager ("STM") 260, and a directory interface manager ("DFM") 270. Each of these components (including any subcomponents) may be implemented as a system including a hardware logic device (e.g., an integrated circuit or a device including such a circuit), a computing device (such as a computer program product) configured to perform the necessary methods based on the execution of specific instructions, or a computer program product (such as a computer program product) configured to perform the necessary methods based on the execution of specific instructions. Figure 1 etc.), including a collection of devices (including such as a combination of Figure 1 The present invention may be implemented as a service of one or more computing devices described in the present invention, such as a hardware logic device or a computing device, or as some combination of the foregoing. Each such component may be implemented on or by a device such as a hardware logic device or a computing device, or across multiple such devices. For example, the components may be implemented in a distributed manner on / by multiple devices such as, for example, a server or element of a network service or a website. In addition, each such component (including any subcomponents) may cover one or more subcomponents, etc., and these components may be implemented as separate components, or any two or more may be combined in whole or in part. Unless otherwise stated, the division of components described herein is non-restrictive and is primarily intended to assist in describing aspects of the present invention.

[0036] In one example, the system 200 is configured to discover and manage the physical configuration of a cluster that is aware of its geographic locale, manage and optimize the virtual configuration of the cluster based on its geographic locale, which includes the D allocation and distribution of VMs and VHs within the cluster, and manage provisioned quantities based on the geographic locale of the cluster. One challenge that the system 200 solves in providing geographic locale awareness to cluster systems and clusters is optimizing the allocation and distribution of cluster resources and virtual resources based on knowledge of geographic locale. That is, by knowing which cluster nodes (and which physical and virtual resources) are located in which geographic locales, the cluster system can manage provisioned quantities more efficiently and can more efficiently place workloads between cluster nodes and other physical resources.

[0037] For example, a cluster may have some nodes located at a geographic location in Seattle where a client (e.g., a cluster user) is located and other nodes located at a geographic location in Dallas where another client is located. Given knowledge of these geographic locations and which node is located at which location, the cluster system can manage the placement of a specified number of workloads and optimize the client's workloads, and for the most efficient operation of physical and virtual resources. Without this innate knowledge of geographic locations, the cluster system will make decisions that tend to have a negative impact on the optimal use of resources, especially when dealing with failover situations, etc. The term "preferred location" used herein refers to one of the geographic locations of a cluster that is set as the preferred location of a cluster for hosting workloads. In another example, the geographic locations of a cluster can be listed in order of preference (e.g., on a list of preferred locations for a cluster). In this way, if the current preferred location fails, the next geographic location in the list becomes the new preferred location. Alternatively or additionally, a workload can be associated with a specific geographic location that is set as the preferred geographic location of the workload, or associated with a list of geographic locations sorted by preference. In this example, the concept of a preferred location is workload-centric, rather than cluster-centric.

[0038] In addition, each geographic site may have a "preferred node" whose nodes are preferred for hosting workloads. In another example, the nodes of a geographic site may be listed in order of preference (e.g., on a list of preferred nodes for the geographic site). In this way, if the current preferred node is down, the next node on the list becomes the new preferred node. Alternatively or additionally, a workload may be associated with a particular preferred node or a list of nodes ordered by preference. In this example, the concept of a preferred node is workload-centric rather than cluster-system-centric.

[0039] A geo-site physical configuration manager ("GPCM") 210 may be a component of the system 200 or may be a separate and / or pre-existing component of a cluster environment utilized by the system 200. In one example, the GPCM 210 is a component of the system 200. The GPCM 210 generally infers the geo-site of the cluster and identifies the physical cluster resources (e.g., nodes and storage devices) located at each geo-site. In one example, "geo-site physical metadata" includes information describing the determined geo-site and the physical cluster resources identified at each geo-site. In this example, the geo-site physical metadata is stored in a geo-site physical database 212 or the like, which may be part of the GPCM 210 or separate from the GPCM 210. The functionality of the GPCM 210 is described below in conjunction with Figure 3 Describe in more detail.

[0040] Resource manager ("RM") 220 can be a component of system 200, or can be a separate and / or pre-existing component of a cluster environment utilized by system 200. In one example, RM 220 is a separate and / or pre-existing component. RM 220 typically manages the placement of workloads in the physical resources of the cluster. For example, RM 220 places VMs and CSVs in these physical resources based on its knowledge of the various physical resources of the cluster. In the event that certain physical resources (such as nodes or storage devices) fail, RM 220 reallocates the virtual resources affected by the failure in order to maintain continuous and efficient operation of the virtual resources to the greatest extent possible. However, RM 220 itself has no knowledge of these geographic locations or the distribution of physical resources in these geographic locations.

[0041] A geo-locale virtual configuration manager ("GVCM") 230 may be a component of the system 200, or may be a separate and / or pre-existing component of a cluster environment utilized by the system 200. The GVCM 230 may be a subcomponent of the RM 220, or alternatively, may supplement the operation of the RM 220. In one example, the GVCM 230 is a component of the system 200 that supplements the operation of the RM 220. The GVCM 230 generally affects or controls the allocation of VMs and VHDs among the physical resources of the cluster based on its geographic locale for the cluster and the cluster's physical resources at each geographic locale. In addition, in the event that some physical resources (e.g., nodes or storage devices) fail, the GVCM 230 affects or controls the relocation of workloads affected by the failure in order to maintain continuous and efficient geo-locale aware operation of virtual resources to the greatest extent possible. In one example, "geo-locale virtual metadata" includes information describing the impact of information used to control the allocation and / or reallocation of virtual resources. In this example, the geo-site virtual metadata is stored in a geo-site virtual database 232 or the like, which may be part of or separate from the GPCM 210. The functionality of the GVCM 230 is described below. Figure 4 Describe in more detail.

[0042] A geographic location provisioned quantity manager ("GQM") 240 may be a component of system 200, or may be a separate and / or pre-existing component of a cluster environment utilized by system 200. In one example, GQM 240 is a component of system 200. GQM 240 may be a subcomponent of a cluster provisioned quantity manager, or alternatively may supplement the operation of a cluster provisioned quantity manager. GQM 240 generally influences or controls how and with what nodes provisioned quantities are established for a cluster based on its knowledge of the geographic locations of the clusters and the physical resources of the clusters at each geographic location. Figure 5 The functionality of the GQM 240 is described in more detail.

[0043] A network topology manager ("NTM") 250 may be a component of the system 200, or may be a separate and / or pre-existing component of a cluster environment utilized by the system 200. In one example, the NTM 250 is a separate and / or pre-existing component. In this example, the NTM 250 generally discovers and / or manages network topology metadata about the nodes of the cluster. In one example, the "network topology metadata" identifies each node of the cluster, the relationships between the nodes, the network latency between each node and each other node, the relationships between subnets of different NICs of the nodes, etc. The network topology metadata may be maintained in a database, etc. that is part of the NTM 250 or separate therefrom.

[0044] Storage topology manager ("STM") 260 may be a component of system 200, or may be a separate and / or pre-existing component of a cluster environment utilized by system 200. In one example, STM 260 is a separate and / or pre-existing component. In this example, STM 250 generally discovers and / or manages storage topology metadata about the storage devices of the cluster. In one example, this "storage topology metadata" identifies each storage device of the cluster (e.g., CSV and file shares), the relationship between each storage device and each cluster node, the network latency between each storage device and each cluster node, whether the storage device is directly connected to one or more nodes, etc. The storage topology metadata may be maintained in a file system as STM 260.

[0045] 15 or a database separate from STM260, etc.

[0046] Directory interface manager ("DIM") 270 may be a component of system 200, or may be a separate and / or pre-existing component of a cluster environment utilized by system 200. In one example, DIM 270 is a separate and / or pre-existing component. Typically, DIM 270 interfaces with an existing network directory to obtain "directory metadata" that describes the configuration of the network in which system 200 operates. Such directory metadata typically provides additional information about the network and storage resources of the cluster environment in which system 200 operates, the colocation of nodes in different geographic locations, and the like.

[0047] Figure 3 3 is a block diagram illustrating an example method 300 for deriving geographic locales for a cluster and identifying physical cluster resources (e.g., nodes and storage devices) located at each geographic locale. In one example, the method 300 is performed by a geographic locale physical configuration manager ("GPCM") 210 based at least in part on metadata obtained from various components of the system 200. In some examples, the various steps of the method 300 may be performed in an order different from that shown. The method 300 generally begins at block 310.

[0048] Box 310 of method 300 generally indicates analyzing directory information about the cluster, such as directory metadata obtained by DIM 270. In one example, the directory site can be configured on a domain controller. This site configuration can be reflected in the directory metadata and can reflect the relationship between cluster nodes, domain controllers, and directory sites. The analysis generally evaluates this relationship information. In view of such analysis, cluster nodes connected to a domain controller ("DC") belonging to a particular directory site can be considered to be co-located in the same proposed geographic site. In one example, the grouping of nodes according to the DC and directory site forms an initial division of cluster nodes into one or more proposed geographic sites. Once the directory analysis is completed, typically resulting in one or more proposed geographic site groupings of cluster nodes, method 300 generally continues at box 320.

[0049] It should be noted that directory places, including active directory places, are not the same as geographic places. For example, an active directory place can be defined as a collection of well-connected subnets. An active directory site can represent the physical structure of a network, while a domain can represent the logical structure of an organization using the network. However, such an active directory physical structure representation is not required and is not typically aligned with or exposed to the various physical geographic locations where physical network resources are located. Although it is possible for a network administrator to force directory places to cover and reflect geographic places, such mapping is typically not derived from directory places. For at least these reasons, directory places such as active directory places cannot be properly considered to disclose or reveal geographic places as described herein. Instead, in some examples, a directory place can be used as a starting point for forming an initial division of cluster nodes into one or more proposed geographic places. This optional initial division is typically further refined based on additional analysis.

[0050] Block 320 of method 300 generally indicates analyzing network topology information about the cluster, such as network topology metadata maintained by NTM 250. In one example, subnet information can be used to assign cluster nodes to geographic locations. For example, all nodes that share the same subnet can be grouped into the same proposed geographic location. Or, in another example, when starting with a proposed geographic location, such as starting from block 310, if the nodes in the proposed geographic location share more than one subnet, then because of the existence of the subnets, the proposed geographic location can be divided into many new proposed geographic locations, each of which includes nodes that share one of the subnets, thereby further refining the proposed geographic location. In one specific example, if two sets of nodes are on the 172.16.0.0 / 19 and 172.17.0.0 / 19 subnets, respectively, and a third set of nodes is on the 193.1.1.0 / 27 subnet, then instead of creating three distinct geographic locations, in this example, two such locations are created: one location with nodes on the 193... subnet and another location with nodes on the 172... subnet.

[0051] In another example, network delays from network topology metadata can be used to create or further refine the proposed geographic venue. For example, delays between pairs of nodes that are physically close together, such as at the same geographic venue, tend to be lower than delays between pairs of nodes that are not physically close together. For example, pairs of nodes that are physically in the same location can have network delays of tenths of a millisecond, while pairs of nodes that are not physically in the same location can have delays of tens and hundreds of milliseconds. In this case, pairs of nodes that share delays of similar magnitude ("common delays") are typically grouped into the same geographic venue. Such a network topology analysis can be used to further refine the proposed geographic venue. Once the network topology analysis is complete, typically resulting in one or more proposed geographic venue groupings of cluster nodes, or resulting in further refined proposed geographic venues, method 300 typically continues at box 330.

[0052] Block 330 of method 300 generally indicates analyzing storage topology information about the cluster, such as storage topology metadata maintained by STM 260. In one example, subnet information can be used to assign cluster storage devices to geographic locations. For example, all storage devices that share a subnet with one or more nodes can be grouped into the same geographic location as these nodes.

[0053] In another example, network delays from storage topology metadata can be used to create or further refine the proposed geographic location. For example, the delay between node storage pairs that are physically close together, such as physically located in the same geographic location, is often lower than the delay between node storage pairs that are not physically close together. For example, node storage pairs that are physically in the same location can have a network delay of one tenth of a millisecond, while node storage pairs that are not physically in the same location can have delays of tens and hundreds of milliseconds. In this case, node storage pairs that share delays of similar magnitude ("common delays") are typically grouped into the same geographic location. Such storage topology analysis can be used to further refine the proposed geographic location. Once the storage topology analysis is completed, typically resulting in one or more proposed geographic location groupings of nodes and storage devices of the cluster, or resulting in further refined proposed geographic locations, method 300 typically continues at box 340.

[0054] Block 340 of method 300 generally indicates completion of the proposed geographic site resulting from the previous steps of method 300. In one example, this is automatically accomplished by GPCM 210 recording the final proposed geographic site as geographic site physical metadata in geographic site physical database 212, etc. In another example, an administrator of system 200 may override or manually adjust the resulting proposed geographic site, may reassign one or more nodes or storage devices from one geographic site to another geographic site. The term "geosite" also refers to an organizational structure / object of a cluster system that is based on and associated with the geographic site physical metadata. Such a "geosite" or "geosite object" generally includes data identifying physical cluster resources associated therewith; e.g., physical resources located at a physical geographic site. Once the proposed geographic site is completed and recorded, method 300 is generally completed.

[0055] Figure 44 is a block diagram illustrating an example method 400 for placement and / or failover of a group (hosted workload) in a geo-site aware cluster system, such as cluster system 600. In one example, method 400 is performed by geo-site virtual configuration manager ("GVCM") 230 based at least in part on metadata, such as geo-site physical metadata, obtained from various components of system 200. In one example, when a new workload is brought online, it is "placed" on a particular node (i.e., hosted on a particular node). This "placement" of the workload is generally in accordance with method 400. In another example, when a node hosting the workload fails, a failover of the group (hosted workload) generally occurs in accordance with method 400. The order of steps of method 400 illustrated represents a particular ordering of the order of the steps. In some examples, the various steps of method 400 may be performed in an order different from the order shown according to a different ordering. Method 400 generally begins at block 410 in the event of a failover.

[0056] If such a preferred node is set or listed, block 410 of method 400 generally indicates group placement and / or failover of a preferred node in a geographic location hosting the workload (such as a cluster-centric or workload-centric preferred node). If not, method 400 generally continues at block 420.

[0057] If the hosted workload (such as, for example, a VM) relies on a specific storage device, block 420 of method 400 generally indicates group placement and / or failover of nodes in the storage association place of the hosted workload. If not, method 400 generally continues at block 430.

[0058] If such a preferred site is set or listed, block 430 of method 400 generally indicates group placement and / or failover of available nodes in the preferred site centered on the workload of the cluster. If not, method 400 generally continues at block 440.

[0059] If such a preferred site is set or listed, block 440 of method 400 generally indicates group placement and / or failover of available nodes in the preferred site centered on the cluster. If not, method 400 generally continues at block 450.

[0060] If such a node is available, block 450 of method 400 generally indicates group placement and / or failover of a node in the geographic location hosting the workload. If not, the group can failover to a different geographic location, as discussed further below.

[0061] In one example, if the time since the last failover of the group is more than a threshold, the group is failed over as described above according to method 400. On the other hand, if the number of failures is greater than a configured value within a threshold time, the group can be failed over according to method 400 to another available geographic site in the cluster (a non-preferred geographic site or a next preferred geographic site in the case of a cluster-centric or workload-centric preferred site list).

[0062] In one example, in the event that a group is manually moved to a non-preferred site, upon failover, the group remains in the manually assigned geographic site as long as a node in that geographic site is available to accept the failover. In the event that no node in the manually assigned geographic site is available to accept the failover, then according to method 400, the group may failover to a node in a different geographic site.

[0063] The ordering of the steps of method 400 can be changed by various sorters of the cluster system. For example, a default sorter can sort the steps of method 400 in the order shown. In another example, a failover history sorter can sort the order of the steps of method 400 to prefer nodes in a geographical location that have experienced the least number of exit failovers (i.e., leaving the group of nodes due to failover) among all other cluster nodes. In yet another example, a workload sorter can sort the order of the steps of method 400 so as to distribute the workload substantially evenly across nodes in a geographical location and / or across geographical locations. Other sorters that define other node preferences or sorting schemes can also be provided.

[0064] In one example, when a dependent storage device is moved to another node in a particular geographic location for some reason (failover, manual move, placement, etc.), the VMs and workloads that depend on the storage device are scheduled to move to a node in the same geographic location, typically after a storage stabilization period. The storage stabilization period is intended to give the dependent storage device enough time to "stabilize" its new node before the dependent workload is moved. The storage stabilization period can be established as a fixed time (e.g., 10 minutes) or can be indicated by the dependent storage device itself. For example, once the dependent storage device is ready (e.g., fully replicated and available in its new location), the dependent storage device itself or some other entity (such as a master node or GVCM 230, etc.) can indicate that the storage stabilization period has ended. At this point, typically in accordance with method 400, the workload that depends on the storage device can be moved to a node in the storage's geographic location.

[0065] In one example, when the previously failed preferred geographical place or node is back online as part of the cluster, fault recovery can be scheduled. For example, the workload that has failed over to a non-preferred or less preferred geographical place or node (relative to the nearest preferred geographical place or node that returns online) will usually fail back to the preferred geographical place or node that returns online according to scheduling. In one example, the scheduling is based on the geographical place or node stable time period. The geographical place node stable time period can be established as a fixed time (e.g., 10 minutes). In some cases, such as after recovery of power or connection, multiple geographical places or nodes can return online at the same time or almost at the same time in the cluster. This stable time period allows multiple geographical places and / or nodes that return simultaneously to return online, and the most preferred geographical places and / or nodes are determined. At that point, usually according to method 400, the group that preferably returns to the geographical place or node can fail back to the geographical place and / or node.

[0066] Figure 5 is a block diagram illustrating an example method 500 for maintaining provisioned quantities by a geo-venue-aware clustering system. In one example, the method 500 is performed by or in conjunction with a geo-venue provisioned quantity manager ("GQM") 240 based at least in part on metadata obtained from various components of the system 200, such as geo-venue-physical metadata and / or geo-venue-virtual metadata. The example method 500 illustrates two distinct and separate geo-venue-aware dynamic provisioned quantity maintenance processes: dynamic weight adjustment and node pruning.

[0067] With respect to dynamic weight adjustment, each cluster node may have a vote, except for those nodes that are scheduled to be offline. Those cluster nodes that are online or not scheduled to be offline (including any witnesses) may have a vote according to their "dynamic weight" (an attribute of each node indicating whether the node currently has a vote). In one example, the dynamic weight of each cluster node is controlled by a GCM 240, etc. For example, the vote count of the cluster is adjusted by adjusting the dynamic weights of each node to indicate that a particular node may or may not vote in a specified number of decisions. The dynamic weights of the nodes are typically adjusted to maintain an odd number of voting nodes for the cluster's specified number of maintenance processes. A "target geographic location" is a location for adjustment of the dynamic weights of one or more nodes. In one example, a dynamic weight of zero indicates that the node cannot currently vote, while a non-zero dynamic weight indicates that the node can currently vote.

[0068] With respect to node pruning, a maintenance process separate and distinct from dynamic weight adjustment may require pruning (i.e., at least temporarily removing) nodes of a cluster from the cluster. For example, a node may need to be pruned due to node failures and / or communication failures such as between nodes and / or geographic locations. Pruned nodes generally do not participate in cluster operations.

[0069] The order of the steps of the illustrated method 500 represents a specific weight adjustment or deletion order. In some examples, the steps of the method 500 can be performed in an order different from the order shown according to a different weight adjustment or deletion order. In one example, when a node needs to be weighted or deleted, the method 500 usually starts at box 510.

[0070] Block 510 of method 500 generally indicates that preference is given to nodes in preferred geographical locations when weights are adjusted or pruned. For example, with respect to node pruning, given a communication problem between two geographical locations or between nodes of two geographical locations, nodes in non-preferred (target) locations are pruned first. In this example, for geographical locations listed in order of preference, nodes of the least preferred (target) geographical location may be pruned first. In this manner, nodes in preferred geographical locations are retained.

[0071] Regarding dynamic weight adjustment, given a communication problem between two geographic places or between nodes of two geographic places, nodes in non-preferred (target) places are first weighted to zero. In the example herein, for geographic places listed in order of preference, nodes in the least preferred (target) geographic place can be weighted to zero first. In this way, nodes in preferred geographic places retain voting rights. If no preferred geographic place is specified, method 500 generally continues at box 520.

[0072] Block 520 of method 500 generally indicates that preference is given to nodes in non-degraded geographic locations when weights are adjusted or pruned. For example, with respect to node pruning, if Figure 6 If geographic sites A and B have good site-to-site communication status, but sites AC and BC do not, the nodes from the target geographic site C are pruned first. In this way, geographic sites and nodes in geographic sites that contribute to communication problems are pruned first, so as to maintain nodes in geographic sites with good communication status.

[0073] Regarding dynamic weight adjustment, if Figure 6 If geographic sites A and B have good site-to-site communication status, but sites AC and BC do not, the weights of nodes from the target geographic site C are first adjusted to zero. In this way, the weights of nodes in geographic sites and geographic sites that contribute to the communication problem are first adjusted to zero in order to retain the voting authority of nodes in geographic sites with good communication status.

[0074] Block 530 of method 500 generally indicates that preference is given to preferred nodes in the target geographic location when weights are adjusted or pruned. For example, with respect to node pruning, if nodes in the target geographic location are pruned, nodes that are not preferred are pruned first. In this example, for nodes listed in order of preference, the least preferred nodes in the target geographic location may be pruned first. In this manner, preferred nodes are retained.

[0075] Regarding dynamic weight adjustment, if the node weights of the target geographic venue are adjusted, the less preferred nodes are adjusted to zero first. In this example, for the nodes listed in the order of preference, the least preferred node in the target geographic venue can be adjusted to zero first. If the preferred node is not specified, the method 500 generally continues at block 540.

[0076] Block 540 of method 500 generally indicates that preference is given to non-low priority nodes in the target geographic location when weights are adjusted or pruned. For example, with respect to node pruning, nodes in the target geographic location that are identified as low priority with respect to voting in a specified number of decisions are pruned first. In the example herein, for nodes listed in priority order, the lowest priority nodes in the target geographic location may be pruned first. In this manner, higher priority nodes are retained.

[0077] With respect to dynamic weight adjustment, nodes in the target geographic location that are identified as having low priority with respect to voting in a specified number of decisions are first adjusted to zero. In this example, for nodes listed in priority order, the lowest priority node in the target geographic location may be first adjusted to zero. In this manner, the voting rights of higher priority nodes are preserved.

[0078] Block 550 of method 500 generally indicates that witnesses are given priority when pruned. For example, witnesses are not generally pruned. However, with respect to dynamic weight adjustment, given a choice between a witness and a node, preference may be given to a node by adjusting the witness's dynamic weight to zero. In another example, given a witness and two venues, one venue having n nodes and the other venue having n-1 nodes, the nodes in the venue with n nodes may be adjusted to zero, resulting in an odd number of voting nodes, where both venues now have n-1 voting nodes.

[0079] Figure 66 is a block diagram of an exemplary geo-site-aware cluster system 600 including three exemplary geo-sites A 610, B 620, and C 630 and an exemplary witness 640. In one example, the physical configuration of cluster system 600 is a result of method 300. Each geo-site in this example includes three nodes, such as the exemplary nodes G, H, and I in exemplary geo-site C 630, and is coupled to other geo-sites via geo-site interfaces such as the exemplary interface 632, via some inter-geographical-site network such as the Internet 650 and / or some other public and / or private network. In practice, any number of nodes may be used per geo-site. The nodes in each geo-site are typically communicatively coupled to each other through one or more local networks, such as via one or more network interface devices, such as one or more network interface cards ("NICs"), etc., and coupled to their respective geo-site interfaces, such as interface 632 of the node coupled to geo-site C 630. Typically, a geo-site interface such as interface 630 is a communications coupling between a node at one geo-site and an inter-geographic-site network that couples the geo-site to other geo-sites of cluster system 600. Such an interface may include a firewall, modem, access point, router, switch, or any other combination of devices configured to couple a node at a geo-site to a network that is communicatively coupled to nodes at other geo-sites.

[0080] In the example cluster system 600, geographic place A 610 can be a preferred place. Alternatively or additionally, in this example, from the perspective of the cluster system, the geographic places of cluster 600 can be listed in a preferred order: geographic place A 610, then geographic place B 620, and then geographic place C 630. In this example, the workload can be alternatively or additionally associated with a preferred place centered on the workload or a geographic place listed in a preferred order. The priority between the preferred geographical place centered on the cluster and the preferred geographical place centered on the workload can be defaulted to being centered on the workload, centered on the cluster, or specified by the workload client and / or cluster system administrator. In one example, priority is given to the preferred geographical place centered on the workload.

[0081] Witness 640 is typically a storage device (e.g., a "witness disk") or other type of device that is accessible to at least one online cluster node and can be used by cluster 600 to maintain a specified number. Witness 640 can alternatively be a file share accessible to at least one online cluster node. Such a witness is also referred to as a "witness node" of the cluster and is considered to be a cluster node for the purpose of maintaining a specified number. For example, if a cluster system such as cluster system 600 has an even number of online nodes, the system can establish a specified number configuration by using a witness node. In this case, as long as the witness remains online (i.e., at least one online cluster node can be accessed), the cluster can maintain the failure in its entirety except that one of its nodes continues to operate. If the witness is offline, the same cluster can maintain the failure in its entirety except for two of the nodes. Thus, the witness allows for the worst case scenario in which only one node in the cluster remains operational while the remaining nodes can access the witness. Figure 5 and corresponding written description presents a method for maintaining a specified number by a geographic location-aware cluster system.

[0082] The geo-site aware cluster system 600 typically utilizes cross-geographic site heartbeat messages to monitor communications between geographic sites. The exact format of the heartbeat message is not particularly relevant here. Heartbeat messages are typically transmitted between the geographic sites of the cluster at appropriate time intervals to facilitate proof of communication problems between geographic sites. This is different from traditional cluster heartbeat messages that are typically transmitted between subnets. In one example, the site-to-site communication status is determined based on cross-geographic site heartbeat messages. In this example, the pattern of successful and lost heartbeat messages is evaluated to determine the degree of degradation or failure of cross-geographic site communications, which is then indicated by the site-to-site communication status. This status may also be affected by the evaluation of other cluster communications such as global update messages ("GUM"). Additionally or alternatively, such a status can be determined for specific nodes between geographic sites such as nodes AD, AE, AF, BD, BE, and BF.

[0083] in conclusion

[0084] In a first example, a method is performed on a cluster system including at least one computing device, the computing device including at least one processor and a memory, the method comprising: automatically identifying, by the cluster system, a plurality of geographic location objects, each of the geographic location objects indicating a subset of physical resources of the cluster system that are physically located at the corresponding geographic location; placing, by the cluster system, a client's workload on a node of a geographic location object of the plurality of geographic location objects, wherein the geographic location object corresponds to the geographic location at which the client is located; and placing, by the cluster system, a dependent storage device of the workload on another node of the geographic location object.

[0085] In a second example, there is a cluster system comprising: a node, the node comprising at least one processor and a memory coupled to the at least one processor; a geo-site physical configuration manager ("GPCM"), the cluster system identifying a plurality of geo-site objects via the geo-site physical configuration manager ("GPCM"), each geo-site object indicating a subset of physical resources of the cluster system that are physically located in a corresponding geo-site; a geo-site virtual configuration manager ("GVCM"), the cluster system placing a client's workload on a node of a geo-site object among a plurality of geo-site objects via the geo-site virtual configuration manager (GVCM), wherein the geo-site object corresponds to the geo-site where the client is located; and the cluster system placing a dependent storage device of the workload on another node of the geo-site object via the GVCM.

[0086] In a third example, at least one computer-readable medium includes computer-executable instructions, which configure the cluster system to perform actions based on execution by at least one computing device of the cluster system, the actions including: automatically identifying, by the cluster system, a plurality of geographic-venue objects, each of which indicates a subset of physical resources of the cluster system that are physically located in a corresponding geographic venue; placing, by the cluster system, a client's workload on a node of a geographic-venue object of the plurality of geographic-venue objects, wherein the geographic-venue object corresponds to the geographic venue at which the client is located; and placing, by the cluster system, a dependent storage device of the workload on another node of the geographic-venue object.

[0087] In the first example, the second example and the third example, the other node is the same as the one node; and / or the cluster system places the workload on a third node of the geographic place object when further involving a failover of the workload; and / or the cluster system further places the workload back on the node in response to a failover event involving the workload after the workload is placed on the third node; and / or the cluster system further identifies the geographic place object as a preferred geographic place; and / or the cluster system deletes some nodes in the geographic place object instead of the geographic place object based on the fact that some nodes are not in the geographic place object; and / or the node is identified as a preferred node by the cluster system.

[0088] The detailed description provided in this section combines many possible embodiments that the present invention and the aforementioned examples can be applied to, it should be recognized that the examples described herein are merely exemplary and should not be taken as limiting the scope of the present invention. Therefore, the present invention as described herein contemplates all such embodiments that may fall within the scope of the attached claims and any equivalents thereof.

Claims

1. A system for managing physical configuration, comprising: a memory storing processor executable process steps; a processor for executing the processor-executable process steps to cause the system to: determining, based on directory metadata indicating a relationship between the two or more cluster nodes and the domain controller, a first subset of the two or more cluster nodes connected to the first domain controller; determining two or more subsets of the first subset, wherein each of the two or more subsets is associated with a different network subnet and includes cluster nodes of the first subset belonging to the associated network subnet of each subset; for each of the two or more subsets, determining the storage device based on a delay between each cluster node of the subset and the storage device, wherein the delay between each storage device of the subset and each cluster node has a similar magnitude; as well as Metadata indicating a storage device of each of the two or more subsets and the cluster nodes is stored.

2. The system of claim 1 , wherein the determining of the two or more subsets comprises: determination of a subnet for each of the two or more cluster nodes; Determination of two or more cluster nodes that are located on the same subnet.

3. The system of claim 1, wherein the directory metadata indicates relationships between cluster nodes, domain controllers, and directory locations.

4. The system of claim 3, wherein the determining of the first subset comprises: It is determined that the first subset of cluster nodes is connected to a first domain controller associated with a first directory location.

5. The system of claim 1, wherein the processor is configured to execute the processor-executable process steps so that the system: For each of the two or more subsets, a pair of cluster nodes is determined, the pair of cluster nodes sharing a delay having a similar magnitude.

6. A method for a cluster system comprising two or more cluster nodes, the method comprising: determining, based on directory metadata indicating a relationship between the two or more cluster nodes and the domain controller, a first subset of the two or more cluster nodes connected to the first domain controller; determining two or more subsets of the first subset, wherein each of the two or more subsets is associated with a different network subnet and includes cluster nodes of the first subset belonging to the associated network subnet of each subset; for each of the two or more subsets, determining the storage device based on a delay between each cluster node of the subset and the storage device, wherein the delay between each storage device of the subset and each cluster node has a similar magnitude; as well as Metadata indicating a storage device of each of the two or more subsets and the cluster nodes is stored.

7. The method of claim 6, wherein determining the two or more subsets comprises: determining a subnet for each of the two or more cluster nodes; Identify two or more cluster nodes that are on the same subnet.

8. The method of claim 6, wherein the directory metadata indicates relationships between cluster nodes, domain controllers, and directory locations.

9. The method of claim 8, wherein determining the first subset comprises: It is determined that the first subset of cluster nodes is connected to a first domain controller associated with a first directory location.

10. The method according to claim 6, further comprising: For each of the two or more subsets, a pair of cluster nodes is determined, the pair of cluster nodes sharing a delay having a similar magnitude.

11. A cluster system, comprising: Two or more cluster nodes; a directory interface manager for storing directory metadata associated with the cluster system, and the directory metadata indicates a relationship between two or more cluster nodes and a domain controller; a network topology manager for storing subnet information associated with the two or more cluster nodes of the cluster system; a storage topology manager for storing information specifying network delays between two or more pairs of cluster nodes among the two or more cluster nodes, and network delays between two or more pairs of cluster nodes and storage devices; as well as A physical configuration manager, the physical configuration manager being used to: determining, based on the directory metadata associated with the cluster system, a first subset of two or more cluster nodes connected to a first directory domain controller; determining two or more subsets of the first subset, wherein each of the two or more subsets is associated with a different network subnet and includes cluster nodes of the first subset belonging to the associated subnet of each subset; for each of the two or more subsets, determining the storage device based on a delay between each cluster node of the subset and the storage device, wherein the delay between each storage device of the subset and each cluster node has a similar magnitude; as well as Metadata indicating a storage device of each of the two or more subsets and the cluster nodes is stored.

12. The system of claim 11, wherein the determining of the two or more subsets comprises: determination of a subnet for each of the two or more cluster nodes; Determination of two or more cluster nodes that are located on the same subnet.

13. The system of claim 11, wherein the directory metadata indicates relationships between cluster nodes, domain controllers, and directory locations.

14. The system of claim 11, wherein the physical configuration manager is further configured to: For each of the two or more subsets, a pair of cluster nodes is determined, the pair of cluster nodes sharing a delay having a similar magnitude.

Images