High availability system with reduced provisioned secondary server

Abstract

According to an example implementation, a method includes associating a secondary computer system with a primary computer system. The secondary computer system includes a first virtual machine hosted by a set of secondary physical computer nodes. The host computer system has a primary role, and the primary role corresponds to a minimum physical resource allocation. The auxiliary computer system is associated with an auxiliary role, and the second computer system is pre-configured to have a second physical resource allocation less than the minimum physical resource allocation. The method includes preparing the secondary computer system to take over the primary role in response to the primary computer system failing and without restarting the secondary computer system. The preparing includes adding at least one physical computer node to the set of secondary physical computer nodes such that the second physical resource allocation satisfies or exceeds the minimum physical resource allocation. The method includes causing the secondary computer system to take over the primary role in response to the primary computer system failing and without restarting the secondary computer system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 356,865, entitled “High Availability and Disaster Recovery,” filed on June 29, 2022, under 35 U.S.C. §119(e), and the U.S. Provisional Application is hereby incorporated by reference in its entirety. Background Art

[0003] A distributed system includes multiple computer nodes that can run in parallel to provide increased processing throughput compared to a single-node system. The computer nodes can execute corresponding programs to perform corresponding operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a block diagram of a system having a primary server, a thinly provisioned secondary server, and a secondary server expansion engine according to an example implementation.

[0005] Figure 2 is a flow chart depicting a technique for preparing a thin-provisioned secondary server to take over for a primary server according to an example implementation.

[0006] Figure 3 is a block diagram of an auxiliary server according to an example implementation, illustrating a software-defined server-based architecture of the auxiliary server.

[0007] Figure 4 is an illustration of using a spare computer node for a software-defined server located in a data center according to an example implementation.

[0008] Figure 5 is an illustration of machine readable instructions stored on a non-transitory machine readable storage medium for causing a machine to add at least one physical computer node to a secondary server to prepare the secondary server to take over for a primary server without causing the secondary server to reboot, according to an example implementation.

[0009] Figure 6 is a block diagram of a system with a failover manager that responds to a failure of a first server and adds at least one physical node to a second server to prepare the second server to take over for the first server without causing the second server to restart, according to an example implementation.

[0010] Figure 7is a flow chart depicting a technique for responding to a failure of a primary server and preparing a secondary server to take over for the primary server without causing the secondary server to restart, according to an example implementation. DETAILED DESCRIPTION

[0011] The following detailed description refers to the accompanying drawings. Where possible, the same reference numerals are used in the accompanying drawings and in the following description to refer to the same or similar parts. However, it should be clearly understood that the drawings are only for the purpose of illustration and description. Although several examples are described in this document, modifications, adaptations and other embodiments are possible. Therefore, the following specific embodiments do not limit the disclosed examples. On the contrary, the correct scope of the disclosed examples can be limited by the attached claims. The terms used herein are only used for the purpose of describing specific examples and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to also include plural forms. As used herein, the term "multiple" is defined as two or more than two. As used herein, the term "another" is defined as at least the second or more. As used herein, unless otherwise indicated, the term "connected" is defined as connected, whether directly connected without any intervening element or indirectly connected by means of at least one intervening element. Two elements can be mechanically connected, electrically connected or linked by communication channels, paths, networks or system communications. As used herein, the term "and / or" refers to and encompasses any and all possible combinations of the associated enumerated items. It should also be understood that although the terms first, second, third, etc. may be used herein to describe various elements, these elements should not be limited by these terms because these terms are only used to distinguish one element from another element unless otherwise specified or the context indicates otherwise. As used herein, the term "includes" means including but not limited to, and the term "including" means including but not limited to. The term "based on" means "based at least in part on".

[0012] A high availability (HA) system includes features (e.g., redundant software and hardware) that avoid single points of failure, so that the HA system remains available even in the event of a failure. The HA system may include a primary server and a secondary server paired with the primary server. When the primary server and the secondary server are used as part of a database management system (DBMS), the primary server may, for example, provide application services, thereby causing updates to the database state (e.g., updates related to submitted transactions). The application services may include, for example, the primary server processing in-flight database transactions with client computer systems. The primary server may or may not save the database state in the primary server's main memory. The secondary server maintains the database state so that if the primary server fails, the secondary server can take over the primary server and quickly restore the operation of the running system.

[0013] Auxiliary servers may not participate in providing application services, and instead, the purpose of the auxiliary server may be limited (as compared to the primary server) to the scope of storing database changes (e.g., committed transactions) defined by the primary server after the primary server makes a database change decision. Database changes may be recorded in log files on both the primary server and the auxiliary server, and the database changes may be stored in a fast persistent storage device. Since the auxiliary server performs lazy updates to the database state by reading log entries, updating the database state, and deleting the log entries once the updates to the database state are committed, the currentness of the auxiliary server's copy of the database state may lag behind the primary server's copy of the database state. The state of the log and the state of the database are consistent with each other on both the primary server and the auxiliary server. If the secondary server performs all updates, the database state on the primary server and the auxiliary server is the same.

[0014] In one approach, the secondary server has the same size as the primary server. In this context, the "size" of a server refers to the physical resource allocation of the server, such as available physical memory, physical central processing unit (CPU), physical storage devices, and physical network interfaces. As an example, using this approach, the size of the primary server and the secondary server can be the same. As a more specific example, the primary server can include a specific number of powered-on computer nodes (e.g., blade servers or rack servers), the secondary server can have the same number of powered-on computer nodes, and the same software (e.g., application software, operating software, and / or any other software) can be installed on both servers.

[0015] Software used in or in conjunction with an HA system may distinguish between using a primary server to provide application services in accordance with its primary role, and sending updates to a secondary server so that the secondary server keeps its database state up to date (taking log files into account) in accordance with its secondary role. In the case where the secondary server has conceptually up-to-date information consistent with the primary server, the secondary server is able to take over the primary server (thereby assuming the primary role) and provide consistent application services without any noticeable interruption when the primary server fails. However, in its standby secondary role, the secondary server may have far fewer physical resources than the primary server because the secondary server may have more limited functionality. If the primary server never fails, the hardware and operating costs of a secondary server with the same physical resources as the primary server may be a wasted expense.

[0016] According to the example implementations described herein, the secondary servers of the HA system are thinly provisioned. In the context used herein, a "thinly provisioned" server refers to a server having an apparent physical resource allocation that is less than the actual physical resource allocation of the server. In other words, the physical resources of the thinly provisioned server are oversubscribed because the server's virtual abstraction of physical resources presents an abstract view of physical resources to the server's operating system, which does not correspond to the actual available physical resources. According to the example implementations, thin provisioning of the secondary server involves configuring the secondary server to have the minimum physical resource allocation required to enable the secondary server to perform its (multiple) secondary role functions (such as maintaining a consistent database state). From the perspective of the guest operating system of the secondary server and the executing applications, thin provisioning creates the appearance that the physical resource allocation of the secondary server is greater than the actual physical resource allocation of the secondary server. This may be particularly advantageous for installing operating systems and applications on secondary servers because they expect a larger available physical resource allocation and ignore the difference between the secondary role and the primary role. Furthermore, because thin provisioning reduces the physical resource allocation of a secondary server, the hardware and operational costs associated with maintaining a secondary server as a standby are reduced (as compared to, for example, fully provisioning a secondary server for its secondary role).

[0017] According to an example implementation, the secondary server expansion engine prepares the thinly provisioned secondary server to take over the primary server by increasing the actual physical resource allocation of the secondary server in response to a failure of the primary server. The process of increasing the actual physical resource allocation of the secondary server may be referred to as "thickening" of the secondary server. As an example of thickening, the secondary server expansion engine may make the actual physical resource allocation of the secondary server the same or nearly the same as the actual physical resource allocation of the primary server. As another example of thickening, the secondary server expansion engine may make the actual physical resource allocation of the secondary server greater than the actual physical resource allocation of the primary server. The thickened secondary server may take over the primary server using its preloaded software and the database state it maintains.

[0018] According to example embodiments, the secondary server may be a software defined server or "SDS". In the context used herein, an "SDS" is a virtualized distributed system having a collection of physical computer nodes (e.g., blade servers, rack servers, or other servers) that collectively form a virtual machine (VM) that hosts a guest operating system (OS). In other words, the VM runs across (or is "hosted by") the computer nodes.

[0019] The SDS architecture allows for dynamic reconfiguration of the computer nodes of an SDS without restarting the guest operating system. More specifically, computer nodes can be added to an SDS and computer nodes can be removed from an SDS, all without restarting or interrupting the guest operating system or any executing applications of the SDS. This allows computer nodes (e.g., fully powered on and fully booted standby computer nodes) to be added to a thinly provisioned secondary SDS to thicken the secondary SDS without restarting or interrupting the guest operating system of the SDS or any executing applications of the SDS's executing applications. Thus, when the associated primary server fails, the thinly provisioned secondary SDS can thicken and take over the primary server in a seamless and time-efficient manner.

[0020] Figure 1 1 is a block diagram of an example system 100 including a high availability (HA) server pair 116, which includes a primary server 108 and a secondary server 112. The HA server pair 116 can be used for any of a number of purposes and applications. For example, the HA server pair 116 can provide at least a portion of a DBMS and provide application services related to database management. As other examples, the HA server pair 116 can provide file sharing services, e-commerce support, and / or provide mission-critical functions. Regardless of its specific purpose or use, in general, the HA server pair 116 provides a certain level of redundancy to avoid single points of failure so that even if a failure occurs, the server can be used to take over the responsibility.

[0021] For the example purpose of providing at least a portion of a DBMS, the primary server 108, in its primary role, can be accessed by client computers 180 and provide application services, which causes, for example, the primary server 108 to process in-flight transactions and commit updates to its database state. The secondary server 112, in its secondary role, can maintain a database state that is current with the database state of the primary server, so that in the event that the primary server 108 fails, the secondary server 112 can take over for the primary server 108. The secondary server 112 may or may not pre-load the database state into its memory, depending on the particular implementation.

[0022] As in Figure 1 , according to an example implementation, primary server 108 may be located in one availability zone 117, and secondary server 112 may be located in a different availability zone 118. In this context, an "availability zone" refers to a group of one or more data centers. For example, primary server 108 may be located in a data center in availability zone 117, and secondary server 112 may be located in a different data center in availability zone 118. According to an example implementation, availability zones 117 and 118 may be in respective geographic regions that are sufficiently isolated such that an event that causes an outage in one availability zone 117, 118 data center(s) in one region is not expected to cause an outage in other availability zones 117, 118 data centers(s).

[0023] According to example embodiments, the HA server pairs 116 may be used for disaster recovery (DR) purposes. For these example embodiments, the HA server pairs 116 may be located in respective geographic regions that are sufficiently isolated such that an event that causes an outage affecting a server in one region is not expected to cause an outage affecting another server. According to other embodiments, the HA server pairs 116 may not be part of a DR system. According to some embodiments, the primary server 108 and the secondary server 112 may be located entirely in respective data centers.

[0024] As in Figure 1 As depicted in , according to example embodiments, the auxiliary server 112 may be thinly provisioned. More specifically, according to some embodiments, the auxiliary server 112 may be configured to have an apparent physical resource allocation (e.g., an allocation perceived by an operating system and applications of the auxiliary server 112) that is less than the actual physical resource allocation of the server. Further, according to example embodiments, the auxiliary server 112 may be an SDS, wherein a VM runs across multiple computer nodes 114 (i.e., a VM is hosted by multiple computer nodes 114).

[0025] In the context used herein, "computer node" refers to an electronic device having processing resources capable of executing machine-readable instructions. Examples of computer nodes may include server computers (e.g., blade servers, rack servers, or stand-alone servers), desktop computers, notebook computers, tablet computers, and other processor-based systems. As an example, the computer node 114 of the auxiliary server 112 may be a blade server or a rack server that is part of the same rack. As another example, the computer node 114 of the auxiliary server 112 may be a blade server on multiple racks in the same data center. As another example, the computer node 114 of the auxiliary server 112 may be located in racks in multiple data centers. As another variation, the computer node 114 of the auxiliary server 112 may be a rack server or a server corresponding to a category other than a blade server or a rack server.

[0026] According to an example implementation, computer nodes 114 contain actual physical resources of auxiliary server 112. Accordingly, adding a computer node to a server's set of computer nodes 114 increases the actual physical resource allocation of auxiliary server 112.

[0027] The system 100 may include a pool of available spare computer nodes 190. According to example embodiments, each spare computer node 190 may be fully powered on and fully booted. As further described herein, one or more spare computer nodes 190 may be added to the auxiliary server 112 (to thereby expand the number of computer nodes 114 of the auxiliary server 112) for thickening the auxiliary server 112. As an example, the spare computer node 190 may be located in the same data center as the existing computer nodes 114 of the auxiliary server 112. As other examples, the spare computer node 190 may be located in a different data center than the existing computer nodes 114 of the auxiliary server 112, or according to other embodiments, the spare computer node 190 may be located in multiple data centers.

[0028] The primary server 108, the secondary server 112, and the backup computer nodes 190 may be interconnected via a network fabric 160. The network fabric 160 may be associated with one or more types of communication networks such as, for example, a Fibre Channel network, a Compute Express Link (CXL) fabric, a dedicated management network, a local area network (LAN), a wide area network (WAN), a global network (e.g., the Internet), a wireless network, or any combination thereof. Figure 1As depicted in , according to an example implementation, the network structure 160 may further connect the client computer 180 to the HA server pair 116 and further connect the secondary expansion engine 120 and the primary server failure detection engine 124 to the HA server pair 116.

[0029] The primary server fault detection engine 124 monitors the primary server 108 for the purpose of detecting a failure of the primary server 108. As an example, the primary server fault detection engine 124 can be part of the secondary server 112. As another example, the primary server fault detection engine 124 can be part of a network switch. As another example, the primary server fault detection engine 124 can be part of a management server or administrative server of the system 100.

[0030] According to other embodiments, the system 100 may not have a primary server failure detection engine 124, or even if the system 100 has a primary server failure detection engine 124, another entity may provide an indication that the primary server 108 has failed and / or an indication that a takeover should occur. For example, the client computer 180 may directly or indirectly trigger a fault corrective action when the primary server 108 is unresponsive. Thus, the primary server failure detection engine 124 and the client computer 180 are examples of entities that may provide an indication that a takeover by the secondary server 112 should occur.

[0031] Regardless of its specific architecture or location, the primary server failure detection engine 124 uses a mechanism (e.g., a heartbeat mechanism or other mechanism) to detect a failure of the primary server 108. In response to detecting the failure, the primary server failure detection engine 124 can then transmit an alert (e.g., a message or other notification) indicating the detected failure of the primary server 108 to the secondary server extension engine 120.

[0032] According to an example embodiment, the secondary server expansion engine 120 is an example of a failover manager that is configured to add one or more computer nodes to a thinly provisioned secondary server 112 in response to a primary server failure alert to prepare the secondary server 112 to take over the primary server 108. In this context, the secondary server expansion engine 120 "adds" (a plurality of) computer nodes refers to the secondary server expansion engine 120 taking one or more actions to at least initiate a process of increasing the number of computer nodes of the secondary server 112. According to an example embodiment in which the secondary server 112 is an SDS, the secondary server expansion engine 120 adds the (a plurality of) computer nodes by thickening the secondary server 112 (i.e., commanding one or more computer nodes to be added to the secondary server 112) by a VM command of the SDS.

[0033] As an example, the secondary server extension engine 120 can be co-located with the primary server fault detection engine 124 (e.g., associated with a pair of processes on the same server). As an example, the secondary server extension engine 120 can be part of the secondary server 112. As another example, the secondary server extension engine 120 can be part of a network switch. As another example, the secondary server extension engine 120 can be part of a management server or an administrative server of the system 100.

[0034] According to some embodiments, the master server 108 may be an SDS with VMs running across multiple physical computer nodes 110. According to other embodiments, in addition to the SDS, the master server 108 may also be a virtualized distributed system of computer nodes. According to still other embodiments, the master server 108 may be a non-virtualized distributed system or a non-distributed system.

[0035] As used herein, an "engine" may refer to one or more circuits. For example, a circuit may be a hardware processing circuit that may include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit (e.g., a programmable logic device (PLD), such as a complex PLD (CPLD)), a programmable gate array (e.g., a field programmable gate array (FPGA)), an application specific integrated circuit (ASIC), or another hardware processing circuit. Alternatively, an "engine" may refer to a combination of one or more hardware processing circuits and machine-readable instructions (software and / or firmware) that may be executed on the one or more hardware processing circuits.

[0036] Regardless of its specific location or architecture, according to an example embodiment, the secondary server expansion engine 120 executes in response to a primary server failure indication. Figure 2 The technology described in 200. Reference Figure 2 The technique 200 includes the primary server failure detection engine 124 receiving a primary server failure indication 230, which is a notification that the primary server 108 has failed. Figure 2 In the example implementation depicted in , the primary server failure indication 230 is provided by the primary server failure detection engine 124. However, according to other implementations, the primary server failure indication 230 or more generally an indication that a takeover should occur may be provided by another entity. As an example, a client computer may provide a trigger for initiating a takeover.

[0037] In the context used herein, the failure of the master server 108 generally refers to the failure of the master server 108 to function as expected. The failure of the master server 108 may be attributed to any of a variety of different reasons. As an example, the failure may be due to a power outage and / or network interruption that particularly affects the master server 108. As another example, the failure may be attributed to a power outage and / or network interruption that affects the rack of the computer node 110 in which the master server 108 is installed. As another example, the failure may be attributed to a power outage and / or network interruption that affects the data center or availability zone of the master server. As other examples, the failure may be due to a software or hardware problem or another reason (e.g., an unexpected network partition caused by a hardware failure, a software error, or another reason) that is unique to the master server 108.

[0038] As in Figure 2 , according to an example embodiment, in response to the primary server failure alarm 230, the primary server failure detection engine 124 generates a primary server failure alarm 234. Generally speaking, according to an example embodiment, the primary server failure alarm 234 initiates an action taken by the secondary server expansion engine 120 to prepare the secondary server 112 to take over the primary server 108.

[0039] More specifically, according to an example implementation, in order to prepare the secondary server 112 to take over the primary server 108, the secondary server expansion engine 120 may cause one or more computer nodes to be added (as depicted at 238) to the secondary server's set of computer nodes 114. Such node additions expand the actual physical resource allocation of the secondary server 112 so that the allocation is commensurate with the actual physical resource allocation of the primary server 108. As an example, the server expansion engine 120 may cause computer nodes to be added to the secondary server 112 to expand the server's physical resource allocation to match (e.g., be the same or nearly the same as) the physical resource allocation of the primary server 108. As another example, the server expansion engine 120 may cause computer nodes to be added to the secondary server 112 so that the server's physical resource allocation is greater than the physical resource allocation of the primary server 108.

[0040] The computer nodes added to the secondary server 112 may correspond to spare computer nodes that are part of a spare pool. The spare computer nodes may be fully powered on and fully booted before being added to the secondary server 112. This facilitates time-efficient and seamless thickening of the secondary server 112, so that the secondary server 112 can quickly and uninterruptedly take over for a failed primary server 108. For example embodiments where the secondary server 112 is an SDS, the spare computer nodes may be located in the same data center (and possibly in the same rack) as the existing computer nodes of the SDS.

[0041] As described herein, according to example embodiments, the auxiliary server 112 may be an SDS. For these example embodiments, the hyperkernel instance (also referred to herein as a "hyperkernel") of the corresponding computer node 114 of the SDS presents a unified view of hardware resources to the guest operating system (OS) of the SDS, so that the guest OS has a view of a single larger computer that includes an aggregation of resources (e.g., processors, memory, I / O resources, network communication resources, and possibly other and / or different resources). According to example embodiments, the virtual resource map of the SDS describes the available virtual resources from the perspective of the guest OS. In addition, according to example embodiments, the physical resource map of the SDS describes the physical resources available on each of the computer nodes 114 of the SDS. According to example embodiments, for the purpose of describing the current mapping between the virtual resource map and the physical resource map from the perspective of the computer node 114, the hyperkernel instance of each computer node 114 creates and maintains a current resource map.

[0042] The VM of the SDS manages the physical resource map and presents the physical resource map to the guest OS. According to some embodiments, for a thinly provisioned SDS, this presentation creates the appearance from the guest OS's perspective that the physical resource allocation is larger than the actual physical resource allocation of the SDS. Each hyperkernel instance of a thinly provisioned SDS has an accurate view of the physical resources managed by the hyperkernel instance. This allows the operating system and application software to be installed on the SDS, thereby anticipating the size of the SDS to be equal to (or larger than) the host server 108.

[0043] According to other embodiments, the guest OS may be able to distinguish between the primary and secondary roles of the SDS.For these example embodiments, for thinly provisioned SDS, the physical resource map presented by the VM to the guest OS may accurately depict the actual physical resource allocation.

[0044] According to an example embodiment, as part of the expansion of the SDS, the secondary server expansion engine 120 causes the SDS to update its physical resource mapping information to include the additional physical resources available on the computer nodes added to the SDS. In addition, according to an example embodiment, the secondary server expansion engine 120 can cause the SDS to update the current resource mapping information between the virtual resources and the physical resources of the SDS. According to an example embodiment, these updates thicken the previously thinly provisioned SDS and correspondingly expand the physical resource allocation of the SDS to be commensurate with the physical allocation of the primary server 108.

[0045] As in Figure 2As further depicted in , after adding (multiple) computer nodes to the secondary server 112, then, as depicted at 242, the server extension engine 120 can initiate a takeover of the primary server 108 by the secondary server. In other words, the secondary server 112 is no longer associated with the secondary role as a result of the takeover, but instead becomes the primary server and correspondingly becomes associated with the primary role. In addition, in conjunction with the takeover, another server can be associated with the secondary role and thereby become the new secondary server of the HA server pair.

[0046] Figure 3 3 is a block diagram of a primary server failure response subsystem 300 according to an example embodiment. The subsystem 300 includes a secondary server expansion engine 120, and the secondary server 112 is or includes an SDS 390. The SDS 390 includes a plurality of computer nodes 114 (in Figure 3 1, 114-2, and 114-N) running VM 350. In other words, N computer nodes 114 host VM 350. Figure 3 One VM 350 is depicted, but it should be noted that there may be at least one other VM running across multiple computer nodes 114-1 to 114-N. Computer nodes 114-1 to 114-N collectively form one VM 350 that hosts a guest operating system (OS) 304 and a collection of programs 306 running in VM 350. Examples of OS include any or some combination of the following: Linux OS, Microsoft WINDOWS OS, Mac OS, FreeBSD OS, and the like.

[0047] The hyperkernel 368 (or "hyperkernel instance") on each physical computer node 114 acts as part of a distributed hypervisor. The hyperkernels 368 communicate with each other to jointly perform the tasks of the hypervisor. Each hyperkernel 368 can observe the operation of the SDS 390 in real time and optimize the system resources of the corresponding computer node 114 to match the requirements of the SDS 390 during operation. The hyperkernel 368 unifies the hardware resources of the computer nodes 114 and presents the unified set to the guest OS 304.

[0048] According to an example embodiment, the hyperkernel 368 performs mapping of guest physical memory addresses to real physical memory addresses. The guest OS 304 performs mapping of guest virtual memory addresses to guest physical memory addresses (using first-level page table mapping). From the perspective of the guest OS 304, the guest physical memory addresses appear to be real physical memory addresses, but they are not. The VM 350 can maintain a virtual resource map that describes the virtual resources available to the guest OS 304 from the perspective of the guest OS 304. According to an example embodiment, the hyperkernel 368 uses second-level page table hardware and second-level address mapping information 308 to map guest physical memory addresses to real physical memory addresses. Each hyperkernel 368 has address mapping information 308, which is the current resource mapping between the virtual resource map and the physical resources managed by the hyperkernel 368 from the perspective of the hyperkernel 368. According to an example embodiment, each hyperkernel 368 has resource mapping information 309 that describes the physical resources managed by the hyperkernel 368.

[0049] According to an example embodiment, the hardware resources of each computer node 114 include a physical processor set 312 (single processor or multiple processors). As used herein, the item "set" can refer to a single item or multiple items. The processor may include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit. As a more specific example, the processor 312 may be a CPU core, a CPU core set, a CPU semiconductor package (or "socket"), a GPU core, a GPU core set, or a GPU semiconductor package.

[0050] According to an example embodiment, the hardware resources of each computer node 114 include physical memory 324 that can be implemented using a collection of physical memory devices. In general, the memory devices that form the physical memory 324, as well as other memories and possible storage media described herein, are examples of non-transitory machine-readable storage media. According to an example embodiment, the machine-readable storage medium can be used for various storage and computing related functions, such as storing machine-readable instructions; storing data representing parameters, preliminary results, final results, and intermediate results; storing tables, storing actual physical allocation representations, storing available physical allocation representations, storing databases, storing database states, storing transaction logs, transmitting alarms, transmitting server health heartbeats, rankings; storing mapping information; storing system management related parameters, and storing other data. The memory devices may include semiconductor memory devices, flash memory devices, memristors, phase change memory devices, combinations of one or more of the foregoing storage technologies, and memory devices based on other technologies. Furthermore, unless otherwise specified herein, a memory device may be a volatile memory device (e.g., a dynamic random access memory (DRAM) device, a static random access (SRAM) device, etc.) or a non-volatile memory device (e.g., a flash memory device, a read-only memory (ROM) device, etc.).

[0051] Computer node 114 may include various other physical resources, such as network interface 342, (I / O) resources, and other categories of physical resources.

[0052] According to example embodiments, the distributed hyperkernel may include an auxiliary server extension engine. According to other example embodiments, the SDS 390 may include an auxiliary server extension engine, but the auxiliary server extension engine may be external to the distributed hyperkernel. According to other embodiments, the auxiliary server extension engine may be associated with a single hyperkernel instance 368 and associated with a particular computer node 114. According to yet other example embodiments, the auxiliary server extension engine may be external to the SDS 390 (e.g., located on a management server or a jurisdiction server).

[0053] refer to Figure 4According to an example embodiment, SDS 390 may be one of a group of SDSs (including one or more additional SDSs 404) that execute or run across a corresponding set of computer nodes located in a particular data center 402. As an example, SDS 390 may initially be a thinly provisioned secondary server in an HA server pair, but later, due to a failure of the primary server of the HA server pair, SDS 390 may be thickened and become the primary server in the HA server pair. As an example, other SDSs 404 may be thinly provisioned secondary servers, primary servers, servers not attached to the HA system, a combination of any of the foregoing entities, or used for different purposes. According to an example embodiment, SDS 390 may be part of an HA server pair, wherein another server in the HA pair may be an SDS that runs across computer nodes that are part of different data centers, availability zones, and / or regions.

[0054] As an example, a data center 402 may contain a spare pool 420 of fully powered on and fully started spare computer nodes 410 available for addition to an SDS 390 and to other SDSs 404. For example, a secondary server expansion engine 120 may add one or more spare computer nodes 410 to an SDS 390 for thickening the SDS 390 to prepare the SDS 390 to take over as a primary server.

[0055] According to some embodiments, the SDS 390 may be dynamically reconfigured, and such dynamic reconfiguration may be unrelated to thickening the SDS 390 for the purpose of taking over the primary server. As an example, the dynamic reconfiguration may be initiated automatically by the SDS 390 or via a console of a management server. According to example embodiments, regardless of the specific type of dynamic reconfiguration, the dynamic reconfiguration may involve adding one or more spare computer nodes 410 to the SDS 390. As an example, the dynamic reconfiguration may be used for the purpose of replacing a computer node of a failed SDS 390. As another example, the dynamic reconfiguration may be used for the purpose of swapping out a computer node of the SDS 390 so that a planned maintenance may be performed on the removed computer node. As another example, the SDS 390 may take over the primary server, and subsequently, due to an increased job and / or application workload, the SDS 390 (now the primary server) may be expanded to accommodate the increased workload. Such expansion may include adding one or more spare computer nodes 410 from a spare pool 420 to the SDS 390. As another example, SDS 390 may take over as a primary server, and then due to a reduced job and / or application workload, SDS 390 (now the primary server) may be scaled down to accommodate the reduced workload in a process that includes returning one or more computer nodes of SDS 390 to standby pool 420.

[0056] The number of standby computer nodes 410 in the standby pool 420 is dynamic and can therefore change over time. In this way, dynamic reconfiguration of the SDS 390 and other SDSs 304 can continually add computer nodes 410 to the standby pool 420 and continually remove computer nodes 410 from the standby pool 420. Furthermore, standby computer nodes 410 can be added to and removed from the standby pool 420 over time in addition to additions / removals caused by dynamic reconfiguration. As an example, the minimum number of standby computer nodes 410 designated for the standby pool 420 can be increased, standby computer nodes 410 can fail, new standby computer nodes 410 can be installed, and additions and removals can be made for other reasons.

[0057] Return to reference Figure 3 , now turning to more specific details of SDS 390, according to some embodiments, SDS 390 can use mapping information to transition between virtual configuration and physical configuration. For example, according to some embodiments, the mapping information can include a physical resource map, an initial resource map, and a current resource map.

[0058] The "physical resource map" is a table that describes the physical resources available on each computer node 114. For example, the table contains the number and type of processors on each node 114, devices, available memory and its physical address range, etc. According to an example embodiment, this table is read-only and is fixed at startup.

[0059] The "initial virtual resource map" is fixed before the boot of the operating system and describes the available virtual resources from the perspective of the guest OS 304. The configuration can be read by the guest OS 304. In some cases, it may be desirable to configure a system that does not match the underlying hardware resources one-to-one (from the operating system's perspective). As an example, it may be desirable for the guest OS 304 to have more memory and fewer cores. This can be accomplished by changing the ratio of memory to cores (i.e., by modifying the initial virtual resource map).

[0060] A "current resource map" is created and maintained by each hyper-kernel 368. This map describes the current mapping between the virtual resource map and the physical resource map from the perspective of each computer node 114. For each entry in the virtual resource map, the definition of the physical resources currently assigned to the virtual resource is maintained. Initially (e.g., at boot time), the current resource map is a copy of the initial virtual resource map. The hyper-kernel 368 modifies the current resource map over time because it observes the characteristics of the resource load and dynamically changes the mapping of physical resources to virtual resources (and vice versa). For example, the definition of the location of an Ethernet controller in a virtualized machine can refer to different hardware controllers at different times. The current resource map is used by the hyper-kernel 368 to dynamically modify virtual hardware resource mappings, such as the virtual memory subsystem, as needed.

[0061] The hyperkernel 368 provides a dynamic adaptive reconfiguration framework that enhances system availability in a data center context by allowing the SDS 390 to be dynamically reconfigured without downtime due to failures or allowing required (or desired) elastic capacity changes to improve efficient utilization of computing resources. That is, this additional layer of the hyperkernel 368 allows the SDS 390 to be dynamically adaptively reconfigured while the system is running without having to shut down the system.

[0062] Dynamic reconfiguration as referred to herein refers to the ability to change or modify the set of hardware components that implement or support the SDS 390 while the system continues to run the guest OS 304 and its applications. For example, using the techniques described herein, the resources of a distributed multi-server architecture can be reconfigured while the operating systems running across the multi-server architecture continue to run, without having to reboot and without any interruption to running applications.

[0063] According to example embodiments, the physical hardware components of the SDS 390 may be grouped and the groups may be associated with corresponding resilient logic modules (also referred to herein as "logic modules"). According to example embodiments, the logic modules have relatively graceful failure modes compared to the physical hardware components, rather than immediate catastrophic failures. As described herein, graceful failures involve transitioning to a degraded state such that the SDS 390 continues to function and data is not lost; depending on the degree of redundancy of the available hardware, a later failure may be catastrophic if the failed component is not replaced.

[0064] In this type of approach, the SDS 390 includes elastic logic modules that allow relatively high levels of availability (e.g., "continuous availability" with an arbitrary number of availability "9s") through dynamic reconfiguration despite physical hardware outages (e.g., due to hardware failures, physical resource scaling, etc.). The main causes of data center downtime are typically related to system capacity reduction, system capacity growth, and component failures during runtime (wherein outage times increase as existing data center systems become larger). Using the elasticity / reconfiguration mechanisms described herein, these causes do not necessarily result in any downtime for the SDS 390. This also minimizes outage costs caused by hardware (e.g., component failures, reconfiguration, repairs, etc.) by allowing in-place failures / planned repairs, hot standby, etc., as will be described in further detail below. It also reduces outage costs caused by software (e.g., software errors, operator errors, overloads, etc.) by allowing thin provisioning, which will be described in further detail below.

[0065] As described above, the dynamic reconfiguration mechanism described herein allows various enhancements to system availability in a data center context, examples of which are by way of non-limiting: allowing dynamic reconfiguration of the SDS 390 without downtime due to failures and allowing required or desired elastic capacity changes to improve efficient utilization of computing resources. Using the techniques described herein, continued operation of the SDS 390 is achieved despite hardware outages (e.g., component failures, rescaling, etc.). Further details regarding such enhancements are described in further detail below.

[0066] The ability to facilitate changes in required or desired capacity of the SDS 390 (i.e., dynamic reconfigurability of the SDS 390) is referred to herein as "thin provisioning" of the SDS 390. One advantage of the hyperkernel described above is that the hyperkernel creates a software layer between the guest OS 304 and the underlying hardware. As a result, the hardware and the guest OS 304 and its applications are more loosely coupled than in previous data centers. Thin provisioning takes advantage of this loose coupling, allowing available virtual resources to be moved between hardware nodes and dynamically remapped to physical resources (e.g., via the resource migration techniques described above). For example, storage volumes, memory, and central processing units can be instantiated from physical resources as needed, rather than being selected and fixed in capacity for the duration of the operation of the SDS 390. As an example, an SDS 390 may be started with 1TB (terabyte) of physical DRAM (dynamic random access memory) and 10 physical processors, and later upgrade its physical instantiation to have more DRAM and more CPUs while the system is running without having to restart the guest OS 304.

[0067] Because virtual memory pages, virtual CPUs, input / output (I / O) devices, and other virtual resources are movable and dynamically reconfigurable throughout the SDS 390, virtual machines supported / presented by the hyperkernel can use the techniques described herein to add and remove sufficient physical resources to support the virtual resources, and then remap those physical resources to additional or different physical resources. This provides the flexibility to scale up and down the system's hardware capacity as needed as workload requirements change, thereby making more efficient use of computing resources.

[0068] Indirectly, this provides a form of resilience not only to failures but also to utilization economies. Using the dynamic reconfiguration mechanism described herein, expansion and contraction of workloads can be easily and automatically accommodated. For example, excess resources can be used by other SDSs until needed, and the power requirements of SDS 390 can be adjusted in proportion to the needs of the workload.

[0069] Being able to dynamically reconfigure the SDS 390 without downtime due to failures allows for in-place failures and further allows, for example, repair scheduling. The dynamic reconfiguration mechanism described herein therefore has an additional benefit to resilience. Physical devices can be suspended from service for repair, replacement, or upgrade while the SDS 390 and its guest OS 304 continue to run. Replaceable devices can be removed from the SDS 390 in the event of an interruption. In addition, physical devices can remain in place for any period of time until removal can be scheduled for replacement or upgrade. This flexibility is an improvement over existing data center environments, where the techniques described herein all reduce downtime by allowing the system to continue to run while allowing appropriate repairs to be performed correctly, for example, without the need for an "emergency on-call expert."

[0070] Further details regarding handling hardware failures while allowing the SDS 390 and its guest OS 304 to continue running are described below.

[0071] As used herein, a "component" refers to any hardware portion of an SDS 390, such as a chip, board, housing, cable, etc. In contrast, as used herein, a "module" refers to a combination or collection of one or more physical components that are combined to create or provide the abstract functionality of an SDS 390 or hyperkernel. One property of an SDS 390 module is that the hardware functionality of the module can be understood without understanding the details of the module's construction, and the operating state of the module can be efficiently described.

[0072] According to an example embodiment, resilient commodity hardware components with fault tolerance are used in the SDS 390 (and modules are created), such as error correction code (ECC) memory, processor over-temperature reduction, software redundant array of independent disks (RAID), multi-path storage on the structure, combined network, etc.

[0073] Examples of module types in the SDS 390 are described in further detail below. In various embodiments, a module includes: a node (e.g., one or more processor chips, a motherboard, its power supply, and a plurality of memory DIMMs (dual in-line memory modules), etc.), an interconnection network (e.g., switches, cables, and NICs (network interface controllers) providing connections between nodes), and an addressable guest network interface (e.g., a port on a hardware NIC).

[0074] As used herein, a resilient logic module (also referred to herein as a "resilient module" or "module") fails gracefully (e.g., degrades rather than fails). A resilient module includes a device that fails into a degraded state. Although suffering from performance degradation, the resilient module continues to function so that there is no data loss. However, a failure can be catastrophic when in a degraded state. Degraded components are components that need to be replaced in a timely manner to maintain system resiliency. Examples of these components include commodity ECC memory, RAID1 / 5 volumes, combined NICs, redundant power supplies, etc.

[0075] According to an example implementation, a resiliency module in the SDS 390 is a logical component that can be described as one or more hardware components that have one of five operational states: available, operational, degraded, faulted, and disconnected. An available module is a module that is ready for use in the SDS 390 but is not actually in use. An operational module is a module that is connected to the SDS 390 and is being used by the hypercore. An operational module is functioning correctly with full performance.

[0076] A degraded module is a module that is connected to the SDS 390 and is being used by the hypercore. Although the degraded module functions correctly, its performance may not be up to specification and should be replaced promptly or it will enter a faulty operating state. A faulty module is a module that is connected to the SDS 390 and causes the hypercore to fail. A faulty module cannot function correctly.

[0077] A disconnected module is a module that is not connected to the SDS 390. If a disconnected module is connected to the SDS 390, it will cause the hypercore to fail. A disconnected module cannot function properly. The SDS 390 (whose connected modules are either operational or degraded) is functioning. The SDS 390 with any module that has failed cannot operate. Available modules and disconnected modules are not connected to the SDS 390 and have no impact on system availability.

[0078] Logic modules are designed to fail gracefully. Using resilient physical components as described above (such as error correction memory, CPU modules that reduce their clock speed when overheated, software RAID, redundant switches, combined network ports, etc.), most failures (e.g., 99.9%) will be failures that fail gracefully and degrade, and cause a transition from an operational state to a degraded state. That is, the most common data center failures can be brought into the resilient module (e.g., power supply, network connection (connector, NIC), disk drive, ECC memory, etc.). The failure of redundant components causes the module operation to degrade, but the system remains operational, which requires repair. These failures that cause a transition from a degraded state to a failed state will then cover the second failure. In contrast, only a small number of failures (e.g., 0.01% of failures) will be failures that directly transition the logic module from an operational state to a failed state.

[0079] Available modules can be added to or removed from the SDS 390.

[0080] A disconnect module can be removed from the SDS 390 or repaired in place (e.g., to rewire a network cable). A disconnect module should not be added to the SDS 390.

[0081] As used herein, reconfiguration of the SDS 390 refers to the process of binding and unbinding logical modules to physical components and binding and unbinding virtual machine components to logical modules. The difference between logical modules and physical components is a form of virtualization (although a different type of virtualization than virtualizing processors, memory, and I / O devices to create virtual machines executed by the hyper-kernel). According to an example embodiment, enhancing the hyper-kernel by adding the ability to dynamically reconfigure involves creating a separation that allows replacement of hardware components (on which the hyper-kernel operates). For example, the hyper-kernel is divided into two layers. One layer includes logical modules (described in further detail below) that manage certain physical resources in the SDS 390. The second layer is referred to as the "upper layer" of the hyper-kernel that works with any logical module on any node.

[0082] In an example embodiment, a logic module representing a hardware component or other hardware resource / component group in a hyperkernel is implemented as a data table. As will be described in further detail below, an example of a logic module is a logic module representing a physical disk drive. For example, the corresponding data table describes the location of the disk drive in the SDS 390, the transactions not processed by the disk drive, etc. Therefore, the logic module includes metadata about the physical components maintained by the hyperkernel. Since the logic module is not hardware, the logic module can be migrated. That is, the logic module implementation method freely migrates its use of the physical component, and the physical component can be moved transparently. The migration of the logic module can be performed without knowing the guest OS 304. That is, this layer of the logic module is hidden from the guest OS 304.

[0083] Thus, in the dynamic reconfiguration framework described herein, the hyperkernel operates on components of logic modules that are bound to physical components at any particular time. The hyperkernel then also provides a set of virtual resources forming a virtual machine to the guest system by binding and unbinding virtual resources to the logic modules. The dynamic reconfiguration layer described herein includes a layer across the architecture of the hyperkernel, namely the logic modules.

[0084] There is a split between logical modules and virtualized objects. The hyperkernel includes an internal layering of resilient logical modules that are degraded rather than disconnected. This allows for dynamic reconfiguration of the SDS 390 and thin provisioning of physical resources while the guest OS 304 continues to run. The dynamic reconfiguration techniques described herein also provide solutions to two resiliency aspects, where the system can continue to run despite failures of hardware components in the resilient modules, and there is no downtime due to scaling changes or preventive maintenance.

[0085] According to example implementations, hardware components (there may be various hardware components serving similar functions) may be abstracted into multiple logical module types: nodes, time bases, network ports, and storage volumes.

[0086] According to an example implementation, a node logic module (also referred to herein as a "logical node") corresponds to a hypercore node. Internally, a node logic module has a CPU and memory. A node logic module may also hold other logic components of other logic module types. Holding represents a higher level aspect of reconfigurability.

[0087] According to an example implementation, there is a time base logic module in the SDS 390. The time base logic module represents the time base of the SDS 390, which is used, for example, for the virtual timestamp counter and various virtual hardware clocks in the synthesis system. In an example implementation of the SDS 390, the time base logic module is the master clock and is logically part of a specific node module.

[0088] The bus port logic module represents a high-speed interconnect from a logical node to other logical nodes attached to an Internet switch. According to an example implementation, there is one bus port logic module in each operational node logic module.

[0089] The network port logic module represents a network interface port. For example, a Virtio network virtual device can be implemented using the network port logic module. The storage volume logic module represents a logical drive controller. For example, a Virtio block virtual device can be implemented using the storage volume logic module.

[0090] The relationship between logical modules and physical components in SDS 390 is not necessarily one-to-one. According to some embodiments, drivers are used that transition between operations on logical modules and operations on physical components. In various embodiments, this may involve mapping logical module identifiers to physical component identifiers, but may also involve more complex mappings when multiple physical components are used to create elastic and reconfigurable logical modules.

[0091] The physical components of the distributed logic module can span multiple nodes. The logic module can relocate functions across different sets of nodes at any time. According to an example embodiment, the relocation is function-specific. For example, to remove a computer node: the network interface switches the MAC address to a NIC port on another computer node; the local drive array across the nodes in the RAID5 structure simply discards the drive on one computer node and adds an empty driver on another computer node; and with respect to the time-based logic module, the main computer node moves itself to another computer node and discards the current computer node. The guest OS 304 is not aware of the relocation. The relocation process does not bring about functional interruption.

[0092] According to an example implementation, the logic module interface in the dynamic reconfiguration framework described herein is defined by a set of commands, which are issued by the hypercore using the logic module. Commands can execute their jobs / tasks immediately or asynchronously. According to an example implementation, the completion of each asynchronous command is signaled by a completion queue. As an example, when issuing a command, each completion is delivered as a kernel event (also referred to as "kernel event (kevent)" herein) on a kernel queue (kernel queue) (also referred to as "kernel queue (kqueue)") specified in the command parameter. Other types of completion queues can also be implemented.

[0093] According to an example implementation, each logical module in the SDS 390 has a unique identifier, also referred to herein as its "logical module ID" (LMI). The scope of the LMI is global to the SDS 390 during the life of the SDS 390 as an entity. A node logical module has its own LMI, which is a node identifier used in the hyperkernel. As an example, the node identifier is used in the upper half of the hyperkernel. In an example implementation, the LMI is represented as a 12-bit unsigned integer (allowing for 4096 logical modules in the SDS 390) and can be stored anywhere that a node identifier is stored (e.g., in a data structure used for resource tracking of pages). The size of the LMI can be as desired.

[0094] Logic modules may be relocated infrequently (e.g., only when reconfiguration occurs). Commands issued to logic modules on one computer node may need to be transmitted to another computer node. In an example embodiment, updates to logic module locations maintain a strictly consistent mapping of logic module locations across all computer nodes (this can be compared to resource tracking for virtual resources in a hyperkernel, where per-node knowledge of the location of virtual resources may be imperfect).

[0095] According to an example embodiment, an application programming interface (API) used by the hypercore partially implements the dynamic reconfiguration framework. The API may include commands issued to logic modules as procedure calls. According to an example embodiment, a dedicated interconnect is used to convert local procedure calls to remote procedure calls.

[0096] For example, the API for logical volumes includes multi-block commands such as readblox, writeblox, and flush. The API also includes Async for distributed completion queues. The API can be extended for guest memory transfers for block I / O, such as injecting guest pages (even if the current page at the guest physical address is remote) and linking to guest pages (e.g., copy-move on guest pages).

[0097] According to an example embodiment, a logic module is distributed across multiple computer nodes (while the computer nodes themselves are not distributed). For example, components of a logic module may span multiple computer nodes at any one time. By instantiating certain types of logic modules on multiple nodes at once, resiliency or performance may be enhanced. During reconfiguration, when the functionality of a logic module is replaced by a new physical component on a new node, the transient state of the logic module is distributed. Since the hyperkernel continues to run while the reconfiguration occurs, the operation of the logic module is managed across multiple nodes during this period. Distributed logic modules also allow for rapid removal of computer nodes by abandoning components.

[0098] One feature of the distributed logic module involves the use of internal messages between nodes to perform internal coordination of portions of the distributed logic module. Ordinary interconnect messages are addressed from a logic module implementation on one node to a logic module implementation on another node. The logic module implementations do not need to process virtual objects, so they do not need to interact directly with any hypercore threads. In an example embodiment, these logic module implementations are implemented using remote process classes that run in response to messages.

[0099] In an example implementation, a logic module command to a local module is a procedure call that returns quickly after completing its work. A command to a module that is not local to a computer node will be sent to one of the computer nodes involved in the module and will return, for example, when the remote command handler returns. According to an example implementation, a remote asynchronous command forwards a kernel event to a kernel queue specified by the issued command.

[0100] In modern hardware systems, certain components are physically embedded in computer nodes. For example, in particular, memory, processors, and I / O devices are typically embedded in computer nodes. Therefore, repairing and replacing these components typically involves removing the computer node from the system, even if most of the computer nodes continue to operate.

[0101] Therefore, a certain node logic module holds each of the non-node logic modules of the system. In this context, "holding" refers to the binding that represents that a logical device is implemented by a physical component attached to the node logic module. For example, a physical NIC found in a specific physical node can be configured to implement a logical network port module, and a physical node can implement a logical node module. In this example situation, a logical node holds a logical network port module. Then a virtual network device is bound to the logical network port module.

[0102] According to an example implementation, a computer node includes a logic module for managing physical processors in an SDS 390. For example, the logic module is implemented as a thread data structure in a host operating system such as FreeBSD. This allows, for example, standardized VCPUs to be presented to a guest OS 304. The physical processors of computer nodes across an SDS 390 may be heterogeneous, with different capabilities, not all of which are presented to a guest OS 304. The logic module corresponding to the standardized VCPU includes information defining which capabilities of the physical processor are provided / not provided. Thus, even if the hardware CPUs are different (where the virtualized CPUs are implemented on a logic module, which in turn is implemented on a physical resource), a standardized set of the same VCPUs can be presented to a guest OS 304.

[0103] As another example, memory pages in SDS 390 may also be associated with logical memory modules, where the logical module for a virtual memory page includes information about the virtual memory page, such as its GCPP entry, when the logical module is local on a node. When a virtual memory page is migrated, the corresponding logical module is also migrated. Processing each virtual memory page as a logical module may not involve additional overhead because the SDS 390 architecture uses the same data structure to handle page movement between nodes.

[0104] As another example, a physical network adapter may be associated with a virtual network adapter presented to the guest OS 304. For example, two physical network adapters may exist on different computer nodes of the SDS 390. In an example embodiment, in the hyperkernel, a logical network adapter (an example logical I / O adapter) implemented by two different physical network adapters may exist on each of the two computer nodes. Although the guest OS 304 knows the virtual network adapter (it thinks that the virtual network adapter is hardware), it does not know the logical network adapter. The logical network adapter includes information that enables two actual physical network adapters to work together to act as a single adapter. For example, this information includes information such as the location of the two physical network adapters, so that the two computer nodes (the physical network adapters are located on these two computer nodes) do not know each other. When the virtual network adapter moves, the logical element (logical network adapter) of the hyperkernel also moves in order to continue working. Therefore, in this example, the guest OS 304 running in the upper level of the hyperkernel does not know the logical module. The logic module includes information about the two physical network adapters (eg, location information) and makes decisions about which physical network adapter to use to implement requests to the virtual network adapter from the guest OS 304. That is, the internal structure of the logic module includes such information about how to apply instructions to different physical adapters.

[0105] Certain logic modules may be implemented in a distributed manner, spread across multiple nodes. Such logic modules, referred to herein as "distributed logic modules", are held by all nodes involved. As an example, certain storage volume logic modules, such as storage volume logic modules with multipath capabilities, are implemented in hardware or simulated in the dynamic reconfiguration framework / subsystem described herein.

[0106] The handling of modules held across multiple computer nodes generalizes the handling of modules held by one computer node. According to an example implementation, to remove a computer node, all modules held by the computer node are rendered inoperable. For such multi-node modules, this includes changing the distributed implementation of the module so that the removed computer node no longer holds the module. The reverse situation can also be performed, as an operational computer node can be added as a holder of a distributed logic module.

[0107] According to some embodiments, to remove a physical node, the logical node module and all logic modules it holds are removed. In one example embodiment, a logic module is removed based on meeting the following criteria: the logic module is no longer operational and is therefore not used by the hypercore. Another logic module takes over the functionality of the removed module.

[0108] The holding relationship is a difference between the node logic module and other types of logic modules, which also affects the order of the reconfiguration steps. For example, suppose there is a network port logic module New, which is available and meets the criteria for taking over from another network port logic module Old to be removed. Before removing Old in, for example, an operating or degraded state, the virtual logical binding of the virtual network port is changed from using Old to using New instead, and the state of Old is changed to available. This includes assigning the media access control (MAC) address used by Old to New, then waiting for all ongoing operations on Old to complete, while directing new operations to use New. When Old is no longer in operation, Old transitions to an available state or a disconnected state.

[0109] According to an example implementation, before a node logic module is removed from the SDS 390, the node logic module should no longer contain any guest state. Specifically, guest pages and VCPUs should not be active on the node logic module. The process of removing guest pages, VCPUs, and I / O devices is referred to herein as "evacuation." In the hyperkernel architecture described herein, evacuation includes the hyperkernel mobilizing VCPUs, pages, and I / O using a resource migration mechanism such as described above.

[0110] The problem of evacuation is to handle guest pages that are "wired" to the node logic module. In some existing systems, certain guest pages can be permanently mapped to, for example, user space pages for an unlimited period of time. As another example, certain pages can be wired into memory, where such pages will have to be moved from one memory module to another. According to an example embodiment, the hyperkernel moves all virtual pages between node logic modules (i.e., there are no "wired" pages) so that virtual pages can be moved at any time. In an example embodiment, the SDS 390 does not map guest pages to user space, and the I / O device emulation handles the moved pages by pausing when accessing a non-local page, then requesting that the page be moved to the node logic module where the emulation is running, or moving the I / O device emulation thread to the node logic module containing the virtual page.

[0111] Therefore, the dynamic reconfiguration layer described herein is supported in part by guest pages that are not "wired" to a specific node logic module. According to an example embodiment, this allows it to be possible to evacuate pages from a node logic module at all times. It is worth noting that transferring data to or from a guest page does not "wire" the page. According to some embodiments, transfers (whether by VCPU or I / O emulation) are performed as unit operations that keep the guest page in memory only long enough to complete the transfer. According to an example embodiment, two mechanisms avoid many I / O-related data copies: replacing guest virtual pages with non-guest memory pages, and creating "copy-on-write" clones of guest virtual pages. Otherwise, I / O emulations break their transfers into chunks within pages and perform transfers to / from each virtual page, one at a time.

[0112] Whenever a VCPU is paused, the VCPU can be migrated to a different logical node. To evacuate any VCPU that happens to be on a node, a pause is called, and at that pause, migration of the VCPU takes the VCPU away from the node. Similarly, a virtual page can be removed to another node at any time by an operation that moves the owner of the page. For pages that are secondary copies (e.g., read-only pages), these pages can be deleted without transferring the pages anywhere.

[0113] According to an example embodiment, the evacuation of all virtual pages is performed by looping over all pages in a logical node and sending each guest virtual page to another operating logical node in SDS 390. Similarly, in an example embodiment, the evacuation of all VCPUs is performed by looping over all VCPUs in a logical node, triggering a stall that forces each VCPU to move to another operating logical node.

[0114] Two example problems that are handled during evacuation while the SDS 390 is operating are: resource tracking and migration blocking. With respect to the resource tracking problem, according to an example implementation, all virtual resources (e.g., VCPUs, virtual pages, and virtual I / O devices) are tracked by a distributed resource tracking algorithm that tracks the location where the resource was last sent on each logical node. In an example implementation, the tracking data includes a series of node pointers to follow in order to find resources that are not on the current logical node. When resources are moved during an evacuation, other logical nodes in the system can still operate under the assumption that the resources are located on the evacuated logical node. Therefore, the evacuated logical node can receive requests for these resources that have been moved. Typically, these messages will be forwarded to the logical node to which the resources have been moved, but this may not be possible once the evacuated logical node becomes inoperable.

[0115] Regarding the migration blocking problem, according to an example implementation, in order to complete or end the evacuation of a logical node, resources should not be moved to the logical node. Therefore, other logical nodes should be notified not to send resources to the node being evacuated.

[0116] The above two example problems are solved using the following example evacuation process performed in multiple steps. First, the evacuated logical node notifies all logical nodes that it is no longer the destination for VCPU, guest virtual page or virtual I / O device migration. The evacuated logical node can still receive requests for resources, and the local node (or the hypercore on the evacuated node) processes or forwards these resources as appropriate during the evacuation phase. Next, the evacuated logical node then begins to evacuate existing pages, VCPUs and virtual I / O devices.

[0117] Each evacuated resource generates a special location update message, which is broadcasted by the evacuated logical node to all other logical nodes, indicating that the resource has been moved from the evacuated logical node to the new logical node (i.e., a message informing other nodes of the updated location of the resource). According to an example embodiment, the evacuation location updates are bundled into messages on the interconnect (e.g., of medium size or any other size as appropriate), thereby reducing the overhead on the interconnect (e.g., as small as about 16 bytes / resource).

[0118] Upon receiving an evacuation location update for a resource R moved from the evacuated logical node E to logical node N, the logical node checks where the node thinks R is, and if it is E, the logical node accepts N as the new location for the resource R. When the evacuation of logical node E is complete, node E broadcasts an evacuation complete message (indicating completion of the resource evacuation) and waits for responses from all other nodes (acknowledging receipt of the evacuation complete message).

[0119] When a logical node receives an Evacuation Complete message from logical node E, the receiving logical node scans its own local table to verify that no resources are now considered to be on logical node E, and if this is true, responds with an Evacuation OK message. If not, the receiving logical node sends one or more Evacuation Location Requests for the locations of all resources still considered to be on logical node E.

[0120] When the logical node E receives Evacuation OK messages from all logical nodes, the logical node E moves out of the operational state. While waiting for the Evacuation OK message, the logical node E processes the Evacuation Location Request(s) to which it responds with the Evacuation Location Update message.

[0121] I / O mobilization supports the ability to allow virtual I / O devices to be re-bound to different logical modules on different logical nodes. There are two bindings involved: virtual to logical and logical to physical. Virtual I / O devices are mobile in that they can be accessed from any VCPU and can access any virtual page without having to move the logical modules or physical devices used in their implementation. Thus, according to an example implementation, the evacuation process handles virtual I / O devices, whereas logical device binding is handled through a logical module binding action from a component of one physical node to a component of another physical node.

[0122] There are many kinds of storage devices. Locally attached drives, removable drives, network attached storage, and fabric attached storage are just some examples of the various options that can be used. In SDS 390, such storage devices are represented by a single type of (possibly distributed) logical module, a storage volume logical module (also referred to herein as a "logical storage volume"). In particular, according to an example embodiment, the hyperkernel virtualizes block devices (e.g., whether using Virtio or emulating a peripheral component interconnect express (PCIe) hardware interface) using a public internal storage volume API that provides, for example, queued asynchronous I / O requests within a logical node (and can be optimized, for example, for handling virtual 4K page sizes and page-aligned blocks). It is worth noting that some operating systems such as Linux, FreeBSD, and Windows are designed to try to use page granularity on their disk file systems as much as possible and to keep page-aligned storage caches as much as possible. Although these operating systems can support different disk block sizes in their file system code, they typically use their virtual memory layer to manage their disk caches, so their unified disks only cache cache page-sized, page-aligned blocks.

[0123] According to an example embodiment, a higher-level hyperkernel uses a public storage volume interface to emulate a block device. For example, according to an example embodiment, the hyperkernel implements the emulation of the Virtio block advanced host controller interface (AHCI) / AT attachment package interface (ATAPI) and NVME high-speed (NVMe) virtual device. It is worth noting that although NVMe is defined for hardware implementation, it has benefits as a guest interface for storage virtualization compared to the Virtio block interface because it uses a natural 4K page size transmission and supports a more efficient queuing protocol. In some embodiments, such emulation (e.g., running on a virtual I / O processor, also referred to as "VIOP" herein) obtains a command chain containing descriptors from a queue in the guest virtual memory, and converts these commands into block-level commands to the logical storage volume. The virtual block device emulation code does not need to know how the "driver" that the storage volume is effective implements the device, and the storage volume driver does not need to know the guest virtual memory, interrupt signaling, etc. This separation of concerns provides reconfiguration flexibility.

[0124] The logical storage volume may be distributed among multiple logical nodes (i.e., distributed logical modules), and therefore, any or all logical nodes may be involved in a particular block-level command. For example, a logical storage volume may be implemented using multiple disk drives across multiple nodes, where the drives are mirrored using a redundant array of independent disks (RAID) or are striped to allow parallel transfers to / from different logical nodes. According to an example implementation, the logical storage volume driver code is configured to route the block-level command to the right disk drive(s) on the correct node.

[0125] According to an example implementation, all virtual storage volumes may be configured to have a 4K block size. Zero copy of virtual logical data transfer may be performed through mapping. Logical volumes are independent of physical volumes, wherein RAID / multipath redundancy is managed by the hyper-kernel.

[0126] As a second example, a fabric-attached logical storage volume supporting multipath access may be connected to multiple logical nodes, again presented to the hyperkernel as a distributed logical storage volume across some or all of these logical nodes (or just to a single logical point in a logical node). In this example scenario, block-level commands issued to the distributed logical storage volume may be directed to any of the elements of the distributed storage volume.

[0127] Reconfiguration can temporarily place the logical storage volume in a distributed state as the path to the content and the content itself are transferred between logical nodes.

[0128] According to an example implementation, the interface of the logical storage volume handles page-sized blocks. To write to the logical storage volume, one or more blocks are provided to the logical storage volume, the transfer is completed, and the blocks are released back to the system.

[0129] To read from a logical storage volume, one or more blocks are provided to the logical storage volume, the transfer is completed, and the block is released back to the system if the content is updated. The completion of the transfer is signaled on a completion queue along with a status block indicating the success / failure of the transfer. In an example embodiment, the completion queue uses a kernel event mechanism (kernel queue) in the operating system FreeBSD. According to an example embodiment, a flush command is provided that allows for forcing one or more blocks into stable storage. The same interface can be used even when the transfer is smaller than a block, where, for example, only a portion of a page-sized block is read or written in a physical device.

[0130] Using page-sized or page-aligned blocks makes the most common case of I / O involve only rebinding the current instance of the virtual memory page. This is more efficient than copying the data multiple times. In most cases, I / O can be performed with zero memory-to-memory copies. Reading data from the drive into a block creates a new memory page on the node containing the device. Once the read is complete, the memory page is replaced in the guest virtual memory with the page that is the target of the guest's I / O. During the I / O, the old page still occupies space in memory. However, once the I / O operation begins, the page will eventually be erased. Therefore, once the I / O operation is queued, the hypercore can erase the page. If the target of the I / O is actually on another node, this is not necessarily the case. While the page read is in progress, the target page can be erased and the dormant instance moved to the node performing the I / O. According to an example embodiment, pages in the system that have not yet been zeroed are indicated as "dormant" pages. According to an example embodiment, these dormant pages are zeroed on an ongoing basis or until they are used for the first time (e.g., allocated and written or created). When the page read is completed, the new content is in the page that can be replaced with the dormant page. If the dormant page moves off-node during this time (which is probably unlikely since it is not normally accessed), the new contents of the page can be sent to wherever the page moved to.

[0131] Writing data from a block to a drive is similarly optimizable. In an example embodiment, the page to be written should be on the node of the logical storage volume driver (if distributed, the page can be moved to any node that can perform the write most efficiently). If it is not there yet, a copy can be used. A page may be designated as a PRIMARY page, and moved to the device as a SECONDARY copy of the page, leaving the PRIMARY page where it is (where PRIMARY and SECONDARY are examples of states that a page or page copy may be in). This forms a "copy on write" situation for guest pages during the time when a write to a block is in progress. A virtual page instance used by a logical storage volume can be moved or deleted, but the instance will retain its contents and will not be released until the write operation is completed. In an example embodiment, this feature is implemented by adding a flag to each page in memory that marks the page as in use by a logical storage volume write.

[0132] Logical storage volumes contain a large amount of guest data that must be retained and will continue to be accessed during dynamic reconfiguration. Therefore, when a logical node is removed, the data in any logical storage volumes held by the node should be retained. Different retention processes can be used depending on the way the logical storage volumes are implemented in the physical media. However, in all cases, the same command to the logical storage volume can be used to complete the retention, wherein this command to the logical storage volume is referred to as logical storage volume relocation in this article.

[0133] According to an example implementation, a logical storage volume is relocated from a set of nodes containing devices to a new set of nodes containing the same or other devices. Commands issued to the logical storage volume (which knows how the command is currently implemented and on which nodes the command is currently implemented) specify a target set of nodes to be used (which may overlap with the current set of nodes used) and any new devices that may be used on those nodes. The logical storage volume driver is configured to perform steps to implement the relocation. During the relocation, the logical storage volume is expanded to include the source set of nodes and the target set of nodes, and then shrunk or contracted to include only the target set of nodes. This expansion / contraction method allows the API of the logical storage volume to hide information about the internal implementation of the logical storage volume driver from the rest of the hyperkernel.

[0134] For fabric-attached storage with multipathing capabilities, relocation consists of updating the set of nodes from which the remote storage device can be accessed and initializing any new connections in the relocation target set, while closing any connections on nodes dropped from the relocation set. Since the content is remote, there is no content transfer.

[0135] As another more complex example, a single volume involving one or more locally attached drives on a single computer node can be relocated. An example method involves transferring content by moving the physical drive itself from a slot on one computer node to a slot on another computer node. In this method, there can be a short period of time (which can be on the order of a few seconds) during which the guest will not be able to access the drive. Further, a human operator may be required to manually move the physical drive, which, if not done correctly, may cause the system to crash. Other methods for improving system availability are described below.

[0136] As another approach, it is assumed that there are spare, empty locally attached drives, or remote fabric-attached backup storage devices, on other computer nodes (e.g., on a replacement node that takes over from a failed node). The relocation can be performed by copying the contents from one volume to another. Assuming that the removed computer node continues to operate fully, such a migration can be performed, but may take time. The logical volumes are temporarily distributed across the two logical nodes, and commands to be redirected to the devices containing the current copies of the blocks at the time the blocks were copied are managed internally.

[0137] As another example, a more complex form of local storage can avoid having to keep a degraded node running for a long time when its logical storage volume is copied away. In an example embodiment, this example form involves a RAID style distribution using a subdivision or part of a RAID volume across multiple computer nodes. For example, a logical storage volume can be implemented as a RAID5 array across four computer nodes, each of which contains a driver. Each pair of logical blocks will be represented as three physical blocks on three consecutive drivers, wherein two of these physical blocks are the contents of the logical blocks, and one of these physical blocks is a parity block, which contains the bitwise exclusive or of the contents of the logical blocks. If a failure is detected in one of the four drivers, the logical storage volume is transformed into a degraded state. However, the physical contents of the drive can be rebuilt by a bit combination of the physical blocks from the other three drives. Therefore, a single drive failure can be handled by adding a new empty drive to the RAID distributed volume on any node--this is an example of a relocation operation. However, if one of the nodes is degraded in some other way, the drive on the node can be rebuilt in the same way without the need to degrade the drive on the node at all.

[0138] Therefore, one beneficial result of using a local distributed logical storage volume with a RAID5 style structure is that a node can be replaced quickly, so that the reconstruction of the logical storage volume content on the node can be carried out leisurely after the reconfiguration is completed.

[0139] One possible cost of using a RAID5-style distributed logical storage volume across multiple computer nodes is that writes to the logical storage volume may be slower because any write involves updating both the logical block and the parity block. Instead of writing once, this requires an additional write, and if the parity block is not in memory, a read of another logical block covered by the parity block may be required. By distributing physical devices across computer nodes, the data of at least one block may also have to be transferred between nodes. However, the RAID structure also has performance advantages, where, for example, the RAID structure allows data to be loaded into memory at high speed due to doubling the bandwidth from physical storage to memory.

[0140] Block I / O is accomplished through a public API that communicates with the logical storage volume abstraction implemented by the logical storage volume driver within the hyperkernel. According to an example embodiment, the logical storage volume driver is implemented as a distributed logic module on multiple nodes. The distributed logic module can be relocated so that one or more nodes no longer hold the logical storage volume. Such relocation is a way to replace the node holding the logical storage volume.

[0141] The logical storage volumes are designed in a way that, for example, hides many details of the common storage system from the guest virtual machines supported by the hyperkernel distributed virtual machine manager, which is instead presented and sees simple block drivers. According to an example embodiment, the details of the storage devices, connections, etc. can be reflected to the guest OS 304, which can then handle these details independently even through the mapping provided by the reverse virtualization performed across nodes. However, it may be more beneficial to apply the modularity achieved by reverse virtualization to hide the complexity from the guest OS 304 while allowing better performance and higher resiliency.

[0142] The logical storage volume abstraction described above creates a strict separation between how the virtual block device gets its instructions from the guest OS 304 and how those instructions are mapped to commands issued on the physical device.

[0143] According to an example embodiment, another category of I / O devices (in addition to the block devices described above) is a network interface. Compared to logical storage volumes, network interfaces do not require management of "content". Replacing one physical NIC port with another can be performed. For example, a MAC address (an aspect of a NIC port that uniquely identifies a NIC port) can be reassigned quickly (e.g., within a few milliseconds).

[0144] SDS 390 splits the emulation of the guest network interface ports (e.g., implemented using Virtio-Net or any other standard as appropriate) into two parts - the guest-facing top half and the network port logic module (also referred to as the "logical network port") that handles all issues dealing with the physical hardware.

[0145] Compared to block devices (where guest operating systems typically perform I / O using page-granular, page-aligned buffers), existing operating system networks are typically poorly organized for zero-copy transfers organized around page-sized units. As an example, the structure of an I / O operation to a network port is as follows. Commands are issued synchronously or asynchronously to the network port logic module to read a data packet or write a data packet to a physical device currently used by the network port to implement the logical network port. According to an example implementation, when a data packet is sent, the data packet is copied from the guest virtual memory to a list of real local memory blocks by the hyper-kernel, and this copy is provided to the network port logic module. In some cases, the copy will arrive at an mbuf (an example of a memory buffer) on the node where the page resides, where the mbuf is provided to the logic module. This simplifies sending from / through a local device or remote device because mbufs are used in the hyper-kernel of both the local device and the remote device.

[0146] The network port logic module can transfer a copy across the dedicated network interconnect from which the copy is made to the node of the physical device. This copy can be smaller than a page and avoids the need to copy a large amount of other data in the page to the device, thereby reducing interconnect traffic. After completion, the status is placed in the completion queue.

[0147] According to an example embodiment, the received data packets are placed in local memory buffers, which are returned in the completion queue using, for example, the command / completion queue interface described above. According to an example embodiment, the memory buffers are provided to the device emulation code, which copies these memory buffers to the guest virtual memory. Again, according to an example embodiment, mbuf chains are used as the data form returned before completing the reception of the data packets. The mbuf chains can then be copied into the virtual page one block at a time, and if the page is remote, the mbuf chains can be transmitted via another interconnect, which is configured to process the mbuf as part of its data packet.

[0148] According to example implementation, the initialization of a computer node places the corresponding logical node module in an available state. The shutdown of a computer node can be performed after the corresponding logical node module is in an available or disconnected operational state. The details of the removal of a node have been described herein, and this places the logical node in an available or disconnected operational state. In the available and disconnected states, the hyperkernel and the logical module are still in play, and their data structures of tracking logical modules and memory states are initialized, etc. These states can have similarities. An example of the difference between available and disconnected is that a computer node belonging to a disconnected state undergoes repair before the computer node is added back to the SDS 390, and a computer node belonging to an available state is ready to use.

[0149] According to an example implementation, a computer node subject to the available state is powered on and ready to be added to the SDS 390. According to an example implementation, initialization of the computer node occurs before the computer node subjects to the available state.

[0150] According to an example embodiment, shutdown of a computer node may include various steps, where, for example, certain states may need to be saved before the node is powered off (i.e., state preservation may be performed during shutdown). According to an example embodiment, in order to implement the dynamic reconfiguration framework described herein, initialization of the hyperkernel is structured as two steps and a third step of starting the SDS 390. According to an example embodiment, each logical node is initialized individually, thereby bringing the node into a usable operational state. According to an example embodiment, this initialization does not involve interaction between logical nodes, as node initialization can occur in parallel. In an example embodiment, a recipe or process for initializing each logical node is transmitted to the logical node in text form.

[0151] Initialization of the hyperkernel may then include applying SDS-specific parameters to one or more logical nodes, thereby placing these logical nodes in an operational state. According to some embodiments, this includes two types of descriptions: SDS-wide parameters (e.g., guest address space size, number of VCPUs, etc.), and definitions of logical modules to be initially stored on each logical node.

[0152] Initialization of the hyperkernel may then include selecting a single logical node as a "boot node" and providing boot information to start the SDS 390. According to an example embodiment, all VCPUs and non-dormant memory are initialized on the boot node. However, after such initialization, there is no distinction or difference between the boot node and other logical nodes. For example, the boot node may not even be logical node 0. One characteristic of the boot node is that the VCPUs are initially found on that node.

[0153] According to example embodiments, any computer node may be removed (eg, to reduce required resources or to perform replacement due to a failure). This includes computer nodes associated with a boot node, which, as described herein, no longer differs from other logical nodes after initialization.

[0154] According to an example implementation, a management node (such as Figure 4 The management node 499 of the SDS 390 manages the SDS 390. The management node can issue service commands related to node actions during startup, reconfiguration, removal, etc. According to some embodiments, the management node may not be used to maintain the operation of the SDS 390, thereby preventing the management node from becoming a single point of failure that will cause the SDS 390 to fail. The management node can provide monitoring / management capabilities to manage the physical components in the SDS 390. For example, the management node can perform external monitoring of the health of the physical components and allow manual or automatic failure response. According to an example embodiment, the nodes of the SDS 390 maintain the configuration and health information used by the management node in the event of a management service failure. In an example embodiment, the SDS 390 provides ports (which can also be replicated) on each node, which can be used by any (authorized) node that provides a management server to manage each node. These ports provide paths for recording data about operations, collecting system health information or other information. According to an example embodiment, these ports also support protocols that allow querying the current system configuration status. Rather than storing the system configuration in an external structure that may be inconsistent or unreliable, a dynamic reconfiguration layer on each node provides introspection capabilities so that a new management can discover the information needed to manage the SDS 390 if it is the first management node.

[0155] The functionality of the introspection interface provides the ability to inventory all logical modules currently operating in the SDS 390, all nodes (whether operating or not), and the physical devices in each node and their use by the logical modules. This is also the coordination point with the management console. In an example embodiment, the management console for the SDS 390 is a program that runs outside the hyperkernel to control dynamic configuration changes by using the introspection interface. According to an example embodiment, the management console is implemented as a process running simultaneously with the hypervisor in a separate computer connected to the SDS 390 or on one of the hardware nodes of the system. If the management console is implemented on one of the nodes of the SDS, it is stopped and restarted on another computer in the event that the node is to be removed from the server.

[0156] According to an example embodiment, the time base logic module (also referred to herein as the "time base module") is responsible for ensuring invariants related to guest virtual time on all nodes of the SDS390. The design invariants of the time sources and time event sources in the hyperkernel are twofold: from the perspective of the guest operating system, virtual time does not "go back"; and virtual time maintains a constant rate relative to the external clock time, as seen by all virtual time sources and time event sources. The invariant rules about "going back" can be complex because there is a causal order between the time read from the virtual clock device and the guest virtual memory content. For example, if two VCPUs read the same clock, the ordering of their memory operations before and after the clock read should conform to the causal serialization of memory accesses performed by the VCPUs. In many guest synchronization techniques that use, for example, the timestamp counter (TSC) of the Intel instruction set and memory access ordering constraints, the causal order is considered correct.

[0157] According to an example embodiment, these invariants are implemented in the SDS 390 by using a single distributed very high resolution clock that is effectively synchronized between each pair of logical nodes (e.g., with a difference of less than microseconds between each pair of nodes) and a rule that effectively delays each guest-level transmission information between the nodes so that such a transmission never arrives at a local time earlier than the time at which the guest-level transmission information was sent. If the interconnect has a transmission delay that is greater than the difference between the logical node pairs, then no delay may be needed at all, unless time synchronization allows time to skew. This can be achieved by controlling the effective rate of the virtual clock so that the virtual clock of each node has the same rate of about, for example, a few parts per million.

[0158] With respect to time-based logic modules and dynamic reconfiguration, the maintenance of synchronization and rate is also taken into account during the addition or removal of logical nodes. For example, when a new logical node becomes operational, the new logical node is synchronized to the existing logical nodes.

[0159] According to an example embodiment, the interconnection between the computer nodes of the SDS 390 is made resilient by adding redundancy. According to an example embodiment, the dynamic reconfiguration framework achieves this by using multiple ports and multiple switches in each computer node, so that the failure of a switch or cable only degrades performance, but there is an alternative (redundant) transmission path.

[0160] In an example implementation, the hyper-kernel employs ordered messaging, and the interconnect protocol manages the selection of paths to specific node destinations within the bus port logic module, which corresponds to, for example, a network kernel implementation of the hyper-kernel. According to an example implementation, the selection of paths and the detection of failed paths are performed in the network kernel driver. Thus, multiple network ports on a computer node are represented by a single bus port logic module at the reconfiguration framework API.

[0161] Future growth of SDS 390 (such as expansion of a thinly provisioned SDS 390) may involve adding more computer nodes than the initial portion of SDS 390 at startup. This means that the set of logical nodes actually in use may change over time. In an example embodiment, in order to handle multiple switches, multiple ports, etc., the interconnection protocol is configured so that it separates the logical node identifier from the physical layer constraints. When broadcasting a hyper-kernel message, inoperable logical nodes do not need to receive or process such a hyper-kernel broadcast, because if these logical nodes do so, it may cause a failure. According to an example embodiment, a hyper-kernel broadcast is implemented so that there is a "multicast group" maintained for all operational logical nodes, and the hyper-kernel broadcast operation uses this multicast group instead of a total broadcast (which may also result in a broadcast to an inoperable logical node).

[0162] According to an example embodiment, unicast messages can be sent to logical nodes, whether operational or not. This capability can be used for messages used to manage reconfiguration of the SDS 390 (e.g., messages transmitted to add another computer node and corresponding logical node to the SDS 390). According to an example embodiment, reconfiguration also uses broadcasts to all nodes on a segment - this can be used for management functions to determine the status of all nodes, not just operational nodes, while the system is running.

[0163] According to an example implementation, the dynamic reconfiguration framework of the SDS 390 divides the hyperkernel into two layers, one layer handles the implementation of a single scalable virtual hardware system (e.g., a single scalable virtual hardware system), and a lower layer provides an abstraction for logic modules that implement all functions for allowing dynamic reconfiguration while the SDS 390 continues to operate. The logic modules can encapsulate fault tolerance mechanisms by supporting a particular logic module to operate in a "degraded" state when a hardware component associated with the logic module fails.

[0164] According to an example embodiment, an external management system (e.g., implemented in a console of a management node) is configured to detect / monitor signals indicating hardware component failures. These signals may be provided by various per-node hypercores. Each hypercore may listen for fault identification signals, e.g., by polling physical components and / or obtaining reported values ​​from physical components.

[0165] The hyper-kernel can detect faults in components by accessing status information collected by the server motherboard. For example, the server motherboard can collect and / or report the performance of various hardware components (e.g., memory chips, processors, fans, block devices, network devices, etc.) attached to the motherboard. This can include, for example, telemetry data collected in memory registers, such as temperature, voltage, fan speed, detected and corrected errors, etc., which are not seen by the guest OS 304 but can be accessed by the hyper-kernel. According to an example embodiment, each node hyper-kernel is configured to poll the motherboard of the node to obtain such telemetry information. In other embodiments, the motherboard reports such telemetry information to the hyper-kernel. The hyper-kernel can then pass the telemetry information to an external management system (wherein the hyper-kernel can also perform processing (e.g., summarizing, such as determining event counts, event rates, etc.) on the telemetry information before the telemetry information is passed to the external management system).

[0166] The hardware may periodically (e.g., based on a timer interrupt) report the state of the hardware, which may be read by the hyper-core. For example, various registers may exist throughout the system, such as on a motherboard and a processor, indicating what faults have occurred. A fault may be one that does not change the behavior of the processor. As an example, consider a detected memory fault. A detected memory fault may be recorded and stored in a register of the motherboard. In existing systems, such telemetry information collected on the motherboard is typically not reported anywhere outside of the basic registers that capture the telemetry information. In an example embodiment, the hyper-core utilizes such information to identify failed / degraded hardware devices / components.

[0167] In an example embodiment, the hyper-kernel observes the count of correctable memory errors. For example, the memory bank can use this technique to determine the correctable error rate on any time scale as appropriate (e.g., correctable errors per minute, per hour, etc.). In a typical computing system, the correctable error rate detected can be approximately zero per hour or zero per week. However, as the system ages, the correctable error rate may increase to, for example, one or two correctable errors per month. Although the memory may not be completely unusable (i.e., the memory is in a degraded state) and does not need to be completely removed, this information can be used by the hyper-kernel to initiate remedial actions to address the degraded component before it fails.

[0168] refer to Figure 5According to an example embodiment, a non-transitory machine-readable storage medium 500 stores instructions 510. The instructions 510, when executed by a machine, cause the machine to associate a secondary computer system with a primary computer system. According to some embodiments, the primary computer system may be in an availability zone that is different from an availability zone of the secondary computer system. According to some embodiments, the primary computer system and the secondary computer system may be respective servers of a high availability (HA) system. According to some embodiments, these servers may be part of a disaster recovery (DR) system. According to some embodiments, the secondary computer system may be a software defined server (SDS). According to some embodiments, the primary computer system may be an SDS.

[0169] The auxiliary computer system includes a first virtual machine hosted by the auxiliary physical computer node set. The main computer system has a primary role, and the primary role corresponds to a minimum physical resource allocation. According to some embodiments, the minimum physical resource allocation corresponds to the actual minimum physical resource size that allows the main computer system to perform a set of functions (e.g., application service-related functions and database functions). As an example, the physical resources may include a processor, memory, I / O, storage device, and network resources of the main computer system.

[0170] The secondary computer system is associated with the secondary role. The secondary computer system is provisioned with a second physical resource allocation that is less than the minimum physical resource allocation. According to some embodiments, the second computer system may be thinly provisioned, and the second physical resource allocation may be sized to support a portion (e.g., database management) but not all of the functions performed by the primary computer system (e.g., the second physical resource allocation may be insufficient for the secondary computer system to provide application services).

[0171] The instructions 510, when executed by the machine, further cause the machine to respond to the primary computer system failing and prepare the secondary computer system to take over the primary role without restarting the secondary computer system. The preparing includes adding at least one physical computer node to the set of secondary physical computer nodes so that the second physical resource allocation meets or exceeds the minimum physical resource allocation. According to some embodiments, preparing the secondary computer system to take over the primary role includes thickening the thinly provisioned secondary computer system. Thickening the thinly provisioned secondary computer system may include updating resource mapping information of the SDS.

[0172] The instructions 510, when executed by the machine, may further cause the machine to cause the secondary computer system to take over the primary role. According to an example implementation, in conjunction with the secondary computer system taking over the primary role, another computer system (eg, a thin-provisioned SDS) is designated as having a secondary role.

[0173] refer to Figure 6According to an example embodiment, the system 600 includes a pair of servers, the pair of servers including a first server 608 and a second server 609. The first server 608 is associated with a primary role of the pair, and the second server 609 is associated with a secondary role of the pair. According to an example embodiment, the secondary server may be a thinly provisioned SDS. According to some embodiments, the first server 608 may be an SDS. According to some embodiments, the pair of servers may be part of a HA system. According to some embodiments, the pair of servers may be part of a DR system. According to some embodiments, the first server 608 may provide application services and database management for a database management system (DBMS). According to some embodiments, the second server 609 may be sized to maintain a database state that is the same as that of the first server 608.

[0174] System 600 includes virtual machine 616; computer node 612; operating system 620; and hyperkernel 634. According to some embodiments, operating system 620 may be a guest operating system of virtual machine 616. Computer node 612 hosts virtual machine 616 and corresponds to actual physical resource allocation of second server 609.

[0175] The hyperkernel 634 provides an abstraction of the available physical resource allocation of the second server 609 to the operating system. The available physical resource allocation is greater than the actual physical resource allocation of the first server 608. According to some embodiments, the hyperkernel 634 may be a hypervisor. According to some embodiments, for the purpose of providing abstraction, the hyperkernel 634 may use a resource map. According to an example embodiment, the resource map may provide a physical resource map, an initial virtual resource map, and a current resource map. The physical resource map is a table describing the physical resources available for each node 612. The initial virtual resource map may be fixed before starting the operating system 620 and describes the available virtual resources from the perspective of the operating system 620. The current resource map may be created and maintained by the hyperkernel 634. The current resource map may describe the current mapping between the virtual resource map and the physical resource map from the perspective of each computer node 612. According to some embodiments, the computer node 612 may be a blade server or a rack server.

[0176] The system 600 includes a failover manager 630. The failover manager 630 responds to the failure of the first server 608 and adds at least one physical computer node to the computer node 612 to make the actual physical resource allocation correspond to the available physical resource allocation without interrupting the operating system 620. According to some embodiments, adding (multiple) physical computer nodes corresponds to the second server 609 of thickening thin provisioning. According to some embodiments, the failover manager 630 can add (multiple) physical computer nodes by changing or updating the resource mapping of the second server 609. According to some embodiments, the (multiple) added physical computer nodes can correspond to one or more computer nodes added from a spare computer node pool. The computer nodes of the spare pool can be fully started and fully powered on before being added to the computer node 612. According to some embodiments, the second server 609 can include at least one additional virtual machine 616 hosted by the computer node 612 or running across these computer nodes.

[0177] refer to Figure 7 According to an example implementation, technique 700 includes providing (block 704) a secondary server as a backup for a primary server. The secondary server includes a virtual machine and a computer node that hosts the virtual machine.

[0178] According to some embodiments, the secondary server may be an SDS. According to some embodiments, the secondary server may be a thinly provisioned SDS. According to some embodiments, the primary server may be an SDS. According to some embodiments, the primary server and the secondary server may be a HA server pair and may be provided in different respective availability zones or regions. According to some embodiments, the primary server and the secondary server may be part of a DR system.

[0179] Technique 700 includes thin provisioning physical resources of a secondary server (block 708). According to example implementations, thin provisioning the secondary server includes oversubscribing physical resources of the secondary server. According to some implementations, thin provisioning of the secondary server includes sizing the secondary server to allow the secondary server to maintain a consistent database state with the primary server while maintaining a size below that required to provide application services provided by the primary server.

[0180] Technique 700 includes: expanding physical resources of a secondary server in response to a failure of the primary server, wherein the expansion includes adding at least one physical computer node to the secondary server without restarting the secondary server, and designating the secondary server as the primary server, according to block 712. According to some embodiments, the expansion of the physical resources includes updating resource mapping information of the SDS to include physical resources of one or more computer nodes previously designated as spares.

[0181] According to an example implementation, the primary server may include a second virtual machine hosted by a primary set of physical computer nodes. The number of secondary sets of physical computer nodes of the secondary computer system in a secondary role is less than the number of primary sets of physical computer nodes of the primary computer system in a primary role. The number of secondary sets of physical computer nodes of the secondary computer system in the secondary role corresponds to the second physical resource allocation. Among potential advantages, costs associated with providing and maintaining the secondary server may be reduced while still providing a seamless transition in the event of a primary server failure.

[0182] According to an example implementation, in preparation of the second server, the number of nodes added may be equal to or greater than the number of nodes of the primary server. Among potential advantages, the costs associated with providing and maintaining secondary servers may be reduced while still providing seamless transition in the event of a primary server failure.

[0183] According to some embodiments, the second server may be a software defined server. Provisioning of the secondary server may include updating a resource mapping of the software defined server. Among potential advantages, costs associated with provisioning and maintaining secondary servers may be reduced while still providing a seamless transition in the event of a primary server failure.

[0184] According to an example implementation, updating the resource mapping includes updating the physical resource map of the software-defined server to include the physical resources of the added (multiple) computer nodes. Among the potential advantages, the costs associated with providing and maintaining secondary servers can be reduced while still providing seamless transition in the event of a primary server failure.

[0185] According to some embodiments, updating the resource mapping includes updating a mapping between a virtual resource map of the software-defined server and a physical resource map of the software-defined server. The virtual resource map describes the virtual resources of the software-defined server from the perspective of a guest operating system of the software-defined server. Among potential advantages, costs associated with provisioning and maintaining secondary servers can be reduced while still providing a seamless transition in the event of a primary server failure.

[0186] According to example implementations, the preparation of the secondary server occurs without interrupting the operating system of the secondary server. Among potential advantages, the costs associated with providing and maintaining secondary servers can be reduced while still providing a seamless transition in the event of a primary server failure.

[0187] According to example implementations, the preparation of the secondary server occurs without interrupting the application of the secondary server. Among potential advantages, the costs associated with providing and maintaining the secondary server can be reduced while still providing a seamless transition in the event of a primary server failure.

[0188] According to an example implementation, the primary server and the secondary server are at least one of a pair of high availability (HA) servers or a pair of disaster recovery (DR) servers. In potential advantages, the costs associated with providing and maintaining the secondary server can be reduced while still providing seamless transition in the event of a primary server failure.

[0189] While the present disclosure has been described with respect to a limited number of embodiments, numerous modifications and variations of the present disclosure will become apparent to those skilled in the art having the benefit of this disclosure. The appended claims are intended to cover all such modifications and variations.

Claims

1. A non-transitory computer-readable storage medium for storing instructions that, when executed by a machine, cause the machine to: Associating a secondary computer system with a primary computer system, wherein The secondary computer system includes a first virtual machine hosted by a secondary set of physical computer nodes, the primary computer system having a primary role, the primary role corresponding to a minimum physical resource allocation, the secondary computer system being associated with the secondary role, and the secondary computer system being provisioned with a second physical resource allocation that is less than the minimum physical resource allocation; In response to a failure of the primary computer system and without restarting the secondary computer system: preparing the secondary computer system to take over the primary role, wherein the preparing comprises adding at least one physical computer node to the set of secondary physical computer nodes so that the second physical resource allocation meets or exceeds the minimum physical resource allocation; and Having the secondary computer system take over the primary role.

2. The non-transitory computer-readable storage medium of claim 1, wherein: The primary computer system includes, in the primary role, a second virtual machine hosted by a primary set of physical computer nodes; The number of secondary physical computer nodes of the secondary computer system in the secondary role is less than the number of primary physical computer nodes of the primary computer system in the primary role; and The number of the set of auxiliary physical computer nodes of the auxiliary computer system in the auxiliary role corresponds to the second physical resource allocation.

3. The non-transitory computer-readable storage medium of claim 2, wherein: The instructions, when executed by the machine, further cause the machine to: in preparation of the second computer system, add the at least one physical computer node to the secondary set of physical computer nodes so that the number of the secondary set of physical computer nodes is equal to or greater than the number of the primary set of physical computer nodes of the primary computer system in the primary role.

4. The non-transitory computer-readable storage medium of claim 1, wherein: The second computer system comprises a software defined server; and The instructions, when executed by the machine, further cause the machine to: update a resource map of the software-defined server in preparation for the second computer system.

5. The non-transitory computer-readable storage medium of claim 4, wherein: Updating the resource map includes updating a physical resource map of the software-defined server to include physical resources of the at least one physical computer node.

6. The non-transitory computer-readable storage medium of claim 4, wherein: Updating the resource mapping includes updating a mapping between a virtual resource map of the software-defined server and a physical resource map of the software-defined server, wherein the virtual resource map describes the virtual resources of the software-defined server from the perspective of a guest operating system of the software-defined server.

7. The non-transitory computer-readable storage medium of claim 1, wherein: The preparation of the secondary computer system occurs without interrupting the operating system of the secondary computer system.

8. The non-transitory computer-readable storage medium of claim 1, wherein: The preparation of the auxiliary computer system occurs without interrupting the application programs of the auxiliary computer system.

9. The non-transitory computer-readable storage medium of claim 1, wherein: The primary computer system and the secondary computer system include at least one of a pair of high availability servers or a pair of disaster recovery servers.

10. A system comprising: A pair of servers, the pair of servers comprising a first server associated with a primary role of the pair and a second server associated with a secondary role of the pair, wherein the second server comprises: Virtual machines; a first plurality of physical computer nodes that host the virtual machines and correspond to actual physical resource allocations of the second server; operating system; and a hyperkernel configured to provide the operating system with an abstraction of an available physical resource allocation of the second server, wherein the available physical resource allocation is greater than the actual physical resource allocation; and A failover manager is configured to respond to a failure of the first server and, without interrupting the operating system, add at least one physical computer node to the first plurality of physical computer nodes so that the actual physical resource allocation corresponds to the available physical resource allocation.

11. The system of claim 10, wherein: The second server comprises a software-defined server from a plurality of software-defined servers; and The at least one physical computer node, before being added, corresponds to at least one spare computer node in a spare computer node pool that is available for expansion of the plurality of software-defined servers.

12. The system of claim 10, wherein: The first server comprises a software defined server; and The second server comprises a software defined server.

13. The system of claim 10, wherein: the first server provided application services and maintained a first database state prior to the failure, and an actual physical resource allocation of the first server was sized to provide the application services and maintain the first database state; and An actual physical resource allocation corresponding to the first plurality of computer nodes is sized to maintain a second data state consistent with the first database state but is not sized to provide the application service.

14. The system of claim 10, wherein: The actual physical resource allocation corresponding to the available physical resource allocation includes an actual physical resource allocation that is greater than or equal to the available physical resource allocation.

15. The system of claim 10, wherein: The hyperkernel uses the resource mapping information to map between the virtual resource map and the physical resource map; The virtual resource map describes virtual resources visible to the operating system; and The failover manager further updates the physical resource map to include physical resources of the at least one physical computer node.

16. The system of claim 10, wherein: The failover manager further updates mapping information of the second server to map virtual resources visible to the operating system to physical resources of the at least one additional physical computer node.

17. A method comprising: Providing a secondary server as a backup of the primary server, wherein the secondary server includes a virtual machine and a plurality of physical computer nodes hosting the virtual machine; thinly provisioning physical resources of the auxiliary server; and In response to a failure of the primary server: expanding physical resources of the auxiliary server, wherein the expanding comprises adding at least one physical computer node to the auxiliary server without restarting the auxiliary server; and Designate the secondary server as the primary server.

18. The method of claim 17, wherein: Thinly provisioning the physical resources of the auxiliary server includes causing a hyperkernel of the auxiliary server to provide a perceived physical resource allocation to the auxiliary server that is greater than an actual physical resource allocation to the auxiliary server; and Extending the physical resources includes commensurate the perceived physical resource allocation with the actual physical resource allocation.

19. The method of claim 17, wherein: The auxiliary server comprises a software-defined server among a plurality of software-defined servers located in a data center; and The at least one physical computer node corresponds to a spare computer node in a spare computer node pool of the data center that is available for the plurality of software-defined servers.

20. The method of claim 17, wherein: The primary server comprises a software-defined server and the secondary server comprises a software-defined server.

Images