Parallel node upgrade in HCI cluster based on fault-tolerant redundancy

Abstract

An information handling system can include at least one processor and a memory. The information handling system can be configured to upgrade a set of nodes of an information handling system cluster by, where the cluster is executing a plurality of workloads: dividing the set of nodes into an upgrade candidate group (UCG) and a non-UCG, where the UCG is determined such that each workload executing on a node in the UCG is also executing on a node in the non-UCG; upgrading the nodes in the UCG in parallel; upon completion of the upgrade, triggering a resynchronization of the cluster; and repeating the steps of dividing, upgrading, and triggering until all nodes in the set of nodes have been upgraded.

Description

Technical Field

[0001] This disclosure generally relates to information processing systems, and more specifically, to accelerating upgrade events in clustered environments such as hyperconverged infrastructure (HCI) clusters. Background Technology

[0002] As the value and use of information continue to grow, individuals and businesses seek additional ways to process and store information. One option available to users is an information processing system. Information processing systems typically process, compile, store, and / or communicate information or data for business, personal, or other purposes, allowing users to leverage the value of information. Because technologies and information processing needs and requirements vary between different users or applications, information processing systems may also vary regarding: what information is processed, how it is processed, how much information is processed, stored, or communicated, and how quickly and efficiently it can be processed, stored, or communicated. Variations in information processing systems allow them to be general-purpose or configured for specific users or purposes (such as financial transaction processing, airline ticketing, enterprise data storage, or global communications). Furthermore, information processing systems can include a variety of hardware and software components that can be configured to process, store, and communicate information, and may include one or more computer systems, data storage systems, and networking systems.

[0003] Hyperconverged infrastructure (HCI) is an IT framework that combines storage, compute, and networking into a single system in an attempt to reduce data center complexity and improve scalability. A hyperconverged platform may include hypervisors for virtualized compute, software-defined storage, and virtualized networking, and these typically run on standard off-the-shelf servers. One type of HCI solution is Dell EMC VxRail. TM Systems. Some examples of HCI systems can be found in various environments (e.g., such as...). ESXi TM This operates within an HCI management system (or any other HCI management system) in the environment. Some examples of HCI systems can be as software-defined storage (SDS) cluster systems (e.g., such as...). vSAN TM The system's SDS cluster system, or any other SDS cluster system, is operated.

[0004] In the HCI context (and other contexts), information processing systems can execute virtual machines (VMs) or containerized workloads for a variety of purposes. A VM or container can typically include any program or collection of executable instructions configured to run a guest operating system on a hypervisor or host operating system. This allows for the management and / or control of the allocation and use of hardware resources such as memory, CPU time, disk space, and input / output devices, either through the hypervisor / host operating system or a combination thereof, and provides an interface between these hardware resources and the applications hosted by the guest operating system.

[0005] Cluster node lifecycle management provides full-stack software upgrade capabilities, which reduces maintenance costs and improves system stability. During such upgrades, each node in the cluster can typically be upgraded (e.g., through upgrade components such as firmware, drivers, and application software). For system administrators, a key objective is to minimize cluster upgrade time to reduce the impact on workloads.

[0006] One strategy to reduce cluster upgrade time is to upgrade all nodes in parallel. However, this is only possible if it is feasible to shut down all running workloads until the upgrade is complete, as physical nodes may need to enter maintenance mode and restart during the upgrade.

[0007] Alternatively, a rolling node upgrade strategy can be used to upgrade nodes one by one, allowing running containers or VMs to be temporarily moved to other nodes via live migration technology without affecting the workload. However, a pure rolling upgrade strategy implies sequential node upgrades without any parallelism. Therefore, embodiments of this disclosure provide a technique for achieving a level of parallelism in node upgrades without shutting down running workloads.

[0008] It should be noted that the discussion of the technology in the background section of this disclosure does not constitute an admission of the state of the prior art. No such admission is made herein unless it is clearly and explicitly stated as such. Summary of the Invention

[0009] Based on the teachings of this disclosure, the disadvantages and problems associated with cluster upgrades can be reduced or eliminated.

[0010] According to embodiments of this disclosure, an information processing system may include at least one processor and a memory. The information processing system may be configured to upgrade a set of nodes in an information processing system cluster, wherein the cluster is executing multiple workloads, by: dividing the set of nodes into upgrade candidate groups (UCGs) and non-UCGs, wherein the UCGs are determined such that each workload being executed on a node in the UCG is also being executed on a node in the non-UCG; upgrading the nodes in the UCGs in parallel; triggering a resynchronization of the cluster after the upgrade is completed; and repeating the steps of dividing, upgrading, and triggering until all nodes in the set of nodes have been upgraded.

[0011] According to these and other embodiments of this disclosure, a method for upgrading a set of nodes in an information processing system cluster, wherein the cluster is executing multiple workloads, the method may include: the information processing system dividing the set of nodes into upgrade candidate groups (UCGs) and non-UCGs, wherein the UCGs are determined such that each workload being executed on a node in the UCG is also being executed on a node in the non-UCG; the information processing system upgrading the nodes in the UCGs in parallel; after the upgrade is completed, the information processing system triggering a resynchronization of the cluster; and the information processing system repeating the steps of dividing, upgrading, and triggering until all nodes in the set of nodes have been upgraded.

[0012] According to these and other embodiments of this disclosure, an article of manufacture may include a non-transitory computer-readable medium having computer-executable instructions that can be executed by a processor of an information processing system for: upgrading a set of nodes in an information processing system cluster, wherein the cluster is executing multiple workloads, by: dividing the set of nodes into upgrade candidate groups (UCGs) and non-UCGs, wherein the UCGs are determined such that each workload being executed on a node in the UCG is also being executed on a node in the non-UCG; upgrading the nodes in the UCGs in parallel; triggering a resynchronization of the cluster after the upgrade is completed; and repeating the steps of dividing, upgrading, and triggering until all nodes in the set of nodes have been upgraded.

[0013] The technical advantages of this disclosure will likely be apparent to those skilled in the art from the accompanying drawings, specification, and claims included herein. The objectives and advantages of the embodiments will be realized and achieved, at least by the elements, features, and combinations specifically pointed out in the claims.

[0014] It should be understood that the foregoing general description and the following detailed description are both illustrative and explanatory, and not limiting of the claims set forth in this disclosure. Attached Figure Description

[0015] A more complete understanding of the embodiments and advantages of the present invention can be obtained by referring to the following description in conjunction with the accompanying drawings, wherein like reference numerals indicate like features, and in the drawings:

[0016] Figure 1 A block diagram of an example information processing system according to an embodiment of this disclosure is shown; and

[0017] Figure 2 An example method according to an embodiment of this disclosure is shown. Detailed Implementation

[0018] refer to Figure 1 and Figure 2 To best understand the preferred embodiments and their advantages, the same reference numerals are used to indicate similar and corresponding parts.

[0019] For the purposes of this disclosure, the term "information processing system" can include any tool or set of tools capable of operating to calculate, classify, process, transmit, receive, retrieve, initiate, switch, store, display, exhibit, detect, record, reproduce, dispose of, or utilize information, intelligence, or data of any form for commercial, scientific, control, entertainment, or other purposes. For example, an information processing system can be a personal computer, a personal digital assistant (PDA), a consumer electronics device, a network storage device, or any other suitable device, and can vary in size, shape, performance, functionality, and price. An information processing system can include memory, one or more processing resources, such as a central processing unit ("CPU") or hardware or software control logic. Additional components of an information processing system can include one or more storage devices, one or more communication ports for communicating with external devices, and various input / output ("I / O") devices (such as a keyboard, mouse, and video display). An information processing system may also include one or more buses capable of operating to transmit communication between various hardware components.

[0020] For the purposes of this disclosure, when two or more elements are referred to as being “coupled” to each other, such a term indicates that the two or more elements are communicating electronically or mechanically, as applicable, whether they are directly or indirectly connected, with or without intervening elements.

[0021] When two or more components are referred to as “coupleable” to each other, such a term indicates that they are capable of being coupled together.

[0022] For the purposes of this disclosure, the term "computer-readable medium" (e.g., temporary or non-temporary computer-readable medium) can include any tool or set of tools that can retain data and / or instructions for a period of time. Computer-readable media can include, but is not limited to: storage media such as direct access storage devices (e.g., hard disk drives or floppy disks), sequential access storage devices (e.g., magnetic tape drives), optical discs, CD-ROMs, DVDs, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and / or flash memory; communication media such as wires, optical fibers, microwaves, radio waves, and other electromagnetic and / or optical carriers; and / or any combination of the foregoing.

[0023] For the purposes of this disclosure, the term "information processing resource" may be used broadly to refer to any component system, apparatus or device of an information processing system, including but not limited to processors, service processors, basic input / output systems, buses, memory, I / O devices and / or interfaces, storage resources, network interfaces, motherboards and / or any other components and / or elements of the information processing system.

[0024] For the purposes of this disclosure, the term "management controller" may be used broadly to refer to an information processing system that provides management functions (typically out-of-band management functions) to one or more other information processing systems. In some embodiments, the management controller may be a service processor, a baseboard management controller (BMC), a chassis management controller (CMC), or a remote access controller (e.g., a Dell Remote Access Controller (DRAC) or an integrated Dell Remote Access Controller (iDRAC)) (or may be an integral part thereof).

[0025] Figure 1 A block diagram of an example information processing system 102 according to an embodiment of the present disclosure is shown. In some embodiments, the information processing system 102 may include a server chassis configured to house multiple servers or "blades". In other embodiments, the information processing system 102 may include a personal computer (e.g., a desktop computer, laptop computer, mobile computer, and / or notebook computer). In still other embodiments, the information processing system 102 may include a storage rack configured to house multiple physical disk drives and / or other computer-readable media for storing data (which may generally be referred to as "physical storage resources"). Figure 1 As shown, the information processing system 102 may include a processor 103, a memory 104 communicatively coupled to the processor 103, a BIOS 105 (e.g., UEFI BIOS) communicatively coupled to the processor 103, a network interface 108 communicatively coupled to the processor 103, and a management controller 112 communicatively coupled to the processor 103.

[0026] In operation, processor 103, memory 104, BIOS 105, and network interface 108 may constitute at least a part of the host system 98 of information processing system 102. In addition to the elements explicitly shown and described, information processing system 102 may also include one or more other information processing resources.

[0027] Processor 103 may include any system, apparatus, or device configured to interpret and / or execute program instructions and / or process data, and may include, but is not limited to, a microprocessor, microcontroller, digital signal processor (DSP), application-specific integrated circuit (ASIC), or any other digital or analog circuit configured to interpret and / or execute program instructions and / or process data. In some embodiments, processor 103 may interpret and / or execute program instructions and / or process data stored in memory 104 and / or another component of information processing system 102.

[0028] Memory 104 may be communicatively coupled to processor 103 and may include any system, apparatus, or device (e.g., computer-readable medium) configured to retain program instructions and / or data for a period of time. Memory 104 may include RAM, EEPROM, PCMCIA card, flash memory, magnetic storage device, opto-magnetic storage device, or any suitable choice and / or array of volatile or non-volatile memory that retains data after power to information processing system 102 is cut off.

[0029] like Figure 1 As shown, an operating system 106 may be stored on memory 104. Operating system 106 may include any program (or a collection of programs with executable instructions) configured to manage and / or control the allocation and use of hardware resources (such as memory, processor time, disk space, and input / output devices) and provide an interface between such hardware resources and applications hosted by operating system 106. Additionally, operating system 106 may include all or part of a network stack for network communication via a network interface (e.g., network interface 108 for communication over a data network). Although operating system 106... Figure 1 The operating system 106 is shown as being stored in memory 104, but in some embodiments, the operating system 106 may be stored in a storage medium accessible to the processor 103, and active portions of the operating system 106 may be transferred from such storage medium to memory 104 for execution by the processor 103.

[0030] Network interface 108 may include one or more suitable systems, devices, or apparatuses operable as an interface between information processing system 102 and one or more other information processing systems via an in-band network. Network interface 108 enables information processing system 102 to communicate using any suitable transport protocol and / or standard. In these and other embodiments, network interface 108 may include a network interface card or "NIC". In these and other embodiments, network interface 108 may be enabled as a motherboard local area network (LAN) (LOM) card.

[0031] Management controller 112 can be configured to provide management functions for managing information processing system 102. This management can be performed by management controller 112 even when information processing system 102 and / or host system 98 are powered off or powered to standby. Management controller 112 may include processor 113, memory, and a network interface 118 that is separate from and physically isolated from network interface 108.

[0032] like Figure 1 As shown, the processor 113 of the management controller 112 can be communicatively coupled to the processor 103. This coupling can be via a Universal Serial Bus (USB), a System Management Bus (SMBus), and / or one or more other communication channels.

[0033] Network interface 118 can be coupled to a management network, which can be separate from and physically isolated from the data network, as shown. Network interface 118 of management controller 112 can include any suitable system, device, or apparatus capable of operating as an interface between management controller 112 and one or more other information processing systems via an out-of-band management network. Network interface 118 enables management controller 112 to communicate using any suitable transport protocol and / or standard. In these and other embodiments, network interface 118 can include a network interface card or "NIC". Network interface 118 can be a device of the same type as network interface 108, or in other embodiments, it can be a different type of device.

[0034] As described above, the HCI cluster may include the information processing system 102 as a node, and the cluster may require upgrades. The upgrade process may be managed by one of the nodes in the cluster or by an external information processing system.

[0035] The implementation scheme disclosed herein can be used to accelerate the upgrade process by leveraging the fault-tolerant redundancy mechanism of a cluster. Specifically, the cluster can have the property of indicating the number of faults allowed (FTT). For example, if FTT=1, then any one node can fail without loss of service. If FTT=2, then any two nodes can fail, and so on.

[0036] In most real-world deployments, FTT=1, and this is therefore the example discussed in detail in this article. With FTT=1, each workload must be hosted on (at least) two nodes. As discussed below, FTT compliance may be temporarily violated during upgrades. For example, for some VMs and at certain times, only one copy of the data may remain on the operating node (the node not undergoing the upgrade process).

[0037] The implementation can search for upgrade candidate groups (UCGs) of nodes, which is mandated by the condition that the UCG is not the only location currently hosting any particular VM. That is, ideally, when a UCG node enters maintenance mode and is upgraded, the remaining nodes (those not upgraded simultaneously) remain operational to host at least one copy of all VM data, thus preventing any VM shutdowns.

[0038] The problem of determining the maximum UCG is a variant of the classic set covering problem, which is known to be NP-hard. That is, the problem can be represented as determining the minimum set of nodes that collectively comprise all running VMs. The complement of this set is the UCG.

[0039] For simplicity and performance reasons, in some implementations, a greedy search heuristic can be used to generate candidate groups. This greedy algorithm does not guarantee finding the optimal UCG (the UCG with the maximum possible number of nodes). However, in practice, it has been found to yield satisfactory results. In some implementations, other algorithms can also be applied to balance optimization between planning time and upgrade time. For example, in small deployments, using an exhaustive search algorithm that guarantees finding the optimal UCG may be feasible, and so on.

[0040] The example greedy algorithm can be performed as follows:

[0041] 1. Start with an empty UCG.

[0042] 2. For each node, determine if a replica of all VMs on that node exists on at least one other node not in the UCG. If true, add the node to the UCG. If not, do not add the node to the UCG.

[0043] 3. Repeat step 2 for each node until there are no more candidate nodes.

[0044] 4. Upgrade all nodes in UCG in parallel.

[0045] 5. When the upgrade of the UCG node is complete, trigger cluster synchronization to restore the data copy according to FTT.

[0046] 6. Repeat these steps until all nodes are upgraded.

[0047] Therefore, the implementation scheme utilizes the cluster's fault-tolerant redundancy mechanism to upgrade nodes in parallel when possible, without affecting running VMs.

[0048] Figure 2 A flowchart of an example upgrade method 200 according to some implementation schemes is shown. As shown, the cluster consists of four nodes A, B, C, and D, and various workloads are executed on VMs on these nodes.

[0049] Step 201 is the initial state of the cluster, where VM1 executes on nodes A and C, VM2 executes on nodes A and D, and VM3 executes on nodes B and D. With each VM executing on two nodes, this configuration has FTT = 1.

[0050] The UCG is defined above, and it includes nodes A and B. The workloads on nodes A and B also exist on other nodes, so it is safe to temporarily violate the FTT and put nodes A and B into maintenance mode by shutting down their workloads. Then, nodes A and B are upgraded in parallel in step 204.

[0051] In step 206, when the upgrades of nodes A and B are complete, an FTT synchronization is triggered, causing additional copies of all VMs to be restored on nodes A and B. When the FTT resumes, the process is repeated by searching the remaining nodes for the new UCG. Nodes C and D are identified as the new UCG in step 208 and then upgraded in parallel.

[0052] All nodes have now been upgraded, and another FTT synchronization can be triggered to restore the cluster to full operation.

[0053] Those skilled in the art who will benefit from this disclosure will understand that, for Figure 2 The preferred initialization point and the order of the steps constituting the method described may depend on the chosen implementation. In these and other embodiments, the method can be implemented as hardware, firmware, software, an application, a function, a library, or other instructions. Furthermore, although... Figure 2 The disclosure specifies a particular number of steps to be taken in the disclosed method, but the method can be performed with more or fewer steps than depicted. Any of the various components disclosed herein (such as...) can be used. Figure 1 The method can be implemented by components and / or any other system capable of operating to implement the method.

[0054] This disclosure covers all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments herein that will be understood by those skilled in the art. Similarly, where appropriate, the appended claims cover all changes, substitutions, variations, alterations, and modifications to the exemplary embodiments herein that will be understood by those skilled in the art. Furthermore, references in the appended claims to a device, system, or component adapted to, arranged to, capable of, configured to, enabled to, or operable to perform a particular function cover said device, system, or component, whether or not it or said particular function is activated, turned on, or unlocked, provided that said device, system, or component is so adapted, arranged, capable of, configured, enabled, operable, or operable.

[0055] Furthermore, the statements in the appended claims that the structure is “configured” or “operable to” perform one or more tasks are not intended to invoke 35 U.S.SC §112(f) with respect to the elements of the claims. Therefore, none of the claims in this filing are intended to be construed as having a component plus a functional element. If the applicant wishes to invoke §112(f) during the course of proceedings, the applicant will use the structure “component for [performing a function]” to state the elements of the claims.

[0056] All examples and conditional language used herein are intended for pedagogical purposes to aid the reader's understanding of the invention and the concepts contributed by the inventors to advance the art, and are to be construed as not being limited to the examples and conditions specifically stated herein. Although embodiments of the invention have been described in detail, it should be understood that various changes, substitutions, and modifications may be made thereto without departing from the spirit and scope of this disclosure.

Claims

1. An information processing system, the information processing system comprising: At least one processor; as well as Memory; The information processing system is configured to upgrade a set of nodes in an information processing system cluster that is performing multiple workloads by: The set of nodes is divided into upgrade candidate groups (UCGs) and non-UCGs, wherein the UCGs are determined such that each workload being executed on a node in the UCG is also being executed on a node in the non-UCG. The nodes in the UCG are upgraded in parallel. After the upgrade is completed, the cluster is resynchronized. as well as Repeat the steps of partitioning, upgrading, and triggering until all nodes in the group of nodes have been upgraded.

2. The information processing system as described in claim 1, wherein the information processing system cluster is a hyperconverged infrastructure (HCI) system.

3. The information processing system as described in claim 1, wherein the partitioning includes using a greedy algorithm.

4. The information processing system as described in claim 1, wherein the partitioning includes using an exhaustive search algorithm.

5. The information processing system of claim 1, wherein the workload is executed in a virtual machine (VM).

6. The information processing system of claim 1, wherein the workload is executed in a container.

7. A method for upgrading a group of nodes in an information processing system cluster, wherein the cluster is performing multiple workloads, the method comprising: The information processing system divides the group of nodes into upgrade candidate groups (UCGs) and non-UCGs, wherein the UCGs are determined such that each workload being executed on a node in the UCG is also being executed on a node in the non-UCG. The information processing system upgrades the nodes in the UCG in parallel. After the upgrade is completed, the information processing system triggers a resynchronization of the cluster; and The information processing system repeats the steps of dividing, upgrading, and triggering until all nodes in the group of nodes have been upgraded.

8. The method of claim 7, wherein the information processing system cluster is a hyperconverged infrastructure (HCI) system.

9. The method of claim 7, wherein the partitioning includes using a greedy algorithm.

10. The method of claim 7, wherein the partitioning includes using an exhaustive search algorithm.

11. The method of claim 7, wherein the workload is executed in a virtual machine (VM).

12. The method of claim 7, wherein the workload is executed in a container.

13. An article of manufacture comprising a non-transitory computer-readable medium having computer-executable instructions thereon, the computer-executable instructions being executable by a processor of an information processing system for: The following upgrades are performed on a group of nodes in an information processing system cluster that is running multiple workloads: The set of nodes is divided into upgrade candidate groups (UCGs) and non-UCGs, wherein the UCGs are determined such that each workload being executed on a node in the UCG is also being executed on a node in the non-UCG. The nodes in the UCG are upgraded in parallel. After the upgrade is completed, the cluster is resynchronized. as well as Repeat the steps of partitioning, upgrading, and triggering until all nodes in the group of nodes have been upgraded.

14. The article of manufacture as claimed in claim 13, wherein the information processing system cluster is a hyperconverged infrastructure (HCI) system.

15. The article of manufacture as claimed in claim 13, wherein the partitioning includes using a greedy algorithm.

16. The article of manufacture as claimed in claim 13, wherein the partitioning includes using an exhaustive search algorithm.

17. The article of manufacture of claim 13, wherein the workload is executed in a virtual machine (VM).

18. The article of manufacture of claim 13, wherein the workload is performed in a container.

Images