System wear leveling

Abstract

A technique includes receiving information related to wear of a computer node. Based on the information related to the wear, the technique includes ranking the computer nodes according to respective expected remaining life of the computer nodes. The technique includes reconfiguring the system based on the ranking in response to an event corresponding to at least one of a spare computer node being added to the system or a given computer node being removed from the system.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 345,691, entitled “System Wear Leveling,” filed on May 25, 2022, under 35 U.S.C. §119(e), and the U.S. Provisional Application is hereby incorporated by reference in its entirety. Background Art

[0003] A distributed system includes multiple computer nodes that can run in parallel to provide increased processing throughput compared to a single-node system. The computer nodes can execute corresponding programs to perform corresponding operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 is a block diagram of a non-virtualized distributed system of computer nodes according to an example implementation.

[0005] Figure 2 is a flow chart of a node wear leveling technique according to an example implementation.

[0006] Figure 3 and Figure 4 is a block diagram of a virtualized distributed system according to an example implementation.

[0007] Figure 5 is a flow chart depicting a technique for updating a node wear model according to an example implementation.

[0008] Figure 6 is a flow chart depicting a technique for scaling up or down physical resources of a distributed system, according to an example implementation.

[0009] Figure 7 is an illustration of machine readable instructions stored on a non-transitory machine readable storage medium for causing a system of a computer node to configure based on node wear leveling information according to an example implementation.

[0010] Figure 8 is a block diagram of a distributed system with instructions to identify computer nodes to implement a change corresponding to the number of computer nodes according to an example implementation.

[0011] Fig. 9 is a flow chart depicting a technique for identifying computer nodes based on node wear leveling information to implement a change in the number of computer nodes of a distributed system, according to an example implementation. DETAILED DESCRIPTION

[0012] The following detailed description refers to the accompanying drawings. Where possible, the same reference numerals are used in the drawings and in the following description to refer to the same or similar parts. However, it should be clearly understood that the drawings are only for the purpose of illustration and description. Although several examples are described in this document, modifications, adaptations and other embodiments are possible. Therefore, the following specific embodiments do not limit the disclosed examples. On the contrary, the correct scope of the disclosed examples can be defined by the appended claims.

[0013] The terms used herein are only used to describe the purpose of specific examples, and are not intended to be restrictive. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to also include plural forms. As used herein, the term "multiple" is defined as two or more than two. As used herein, the term "another" is defined as at least the second or more. As used herein, unless otherwise indicated, the term "connected" is defined as connected, whether directly connected without any intervening element or indirectly connected by means of at least one intervening element. Two elements can be mechanically connected, electrically connected or linked by communication channels, paths, networks or system communications. As used herein, the term "and / or" refers to and encompasses any and all possible combinations of associated enumerated items. It should also be understood that although the terms first, second, third, etc. can be used to describe various elements in this article, these elements should not be limited by these terms because these terms are only used to distinguish one element from another element, unless otherwise indicated or the context indicates otherwise. As used herein, the term "includes" means including but not limited to, and the term "including" means including but not limited to. The term "based on" means "based at least in part on".

[0014] Semiconductor devices such as memory devices and central processing units (CPUs) tend to wear out or degrade over time and eventually fail due to factors such as decreasing feature sizes (e.g., transistor channel widths). Accordingly, the degradation of a computer node (e.g., a blade server or a rack server) over time can be at least partially attributable to the degradation of the semiconductor devices of the node. In this context, a "computer node" refers to an actual or physical electronic device having processing resources capable of executing machine-readable instructions. A computer node has a particular lifespan, which is the total time the computer node is in use before the computer node no longer functions as expected (e.g., the computer node fails completely or partially or exhibits performance that has degraded to an unsatisfactory level). The remaining lifespan of a particular computer node is a function of the current degree of degradation or wear of the computer node.

[0015] A distributed computing system (referred to herein as a "distributed system") may have a primary or active pool of computer nodes, and the distributed system may access a secondary or standby pool of computer nodes (referred to herein as a "standby pool"). Standby computer nodes may, for example, provide replacements for failed active computer nodes and / or provide additional physical resources when the physical resources (and number of active computer nodes) of the distributed system are expanded. Active computer nodes may be returned to the standby pool of computer nodes (to reduce or discard physical resources) when the distributed system is reduced. Expanding and reducing a distributed system is an example of what is referred to herein as "reconfiguring" a distributed system. In the context used herein, reconfiguring a distributed system refers to performing a process that includes at least one of removing a computer node from a distributed system of computer nodes or adding a computer node to the distributed system.

[0016] Reconfigure distributed system to add computer node for expanding this system and to select standby computer node from standby pool.One way to do this is to identify this standby computer node based on the associated node identifier (ID) of specific standby computer node.For example, node ID can be selected randomly or pseudo-randomly from the node ID of standby pool.Another way to select standby computer node is to select node ID based on specific numerical sorting scheme.For example, standby pool can include standby computer node corresponding to node ID 3,8 and 12, and in the selection scheme with lowest numerical priority, standby computer node with node ID 3 can be selected next.In a similar manner, reconfigure distributed system to remove active node for reducing this system and can involve to select one of active computer node of system based on node ID for removal.

[0017] A node selection scheme based solely on node IDs fails to evenly distribute wear among computer nodes, resulting in a wide range of remaining lifetimes for active computer nodes of a distributed system and associated standby computer nodes of the system. In this manner, some of the computer nodes may be underutilized and therefore have a relatively long expected remaining lifetime, while other computer nodes may be overutilized (and have relatively more wear) and on the verge of failure.

[0018] According to example embodiments described herein, a distributed system is reconfigured using a computer node selection process based on the predicted or expected remaining life of a candidate computer node. Basing the selection on the expected remaining life causes wear to be more evenly distributed among the computer nodes. This process of evenly distributing wear is referred to herein as "system wear leveling" or "node wear leveling." With system wear leveling, computer nodes with relatively more wear are used less frequently (or not at all), and computer nodes with relatively less wear are used more frequently. In general, system wear leveling reduces (even if it cannot minimize) the deviation between the expected remaining life of computer nodes.

[0019] According to an example embodiment, a node wear leveling engine (e.g., an engine that distributes active computer nodes across a distributed system) controls the selection of a particular computer node in conjunction with a reconfiguration (or configuration) of the distributed system. Reconfiguration can be any of a plurality of processes of adding and / or removing computer nodes. As an example, reconfiguration can be to extend the physical resource allocation of the distributed system, such as thickening a thinly provisioned distributed system or amplifying the physical resources of a fully provisioned distributed system to meet an increased application or job workload. As another example, reconfiguration can reduce the physical resources of the distributed system to be commensurate with the reduced application or job workload. As another example, reconfiguration can involve adding and removing computer nodes in a process that leaves the number of active computer nodes of the distributed system unchanged. For example, reconfiguration can include removing active computer nodes of the distributed system for planned maintenance and replacing the removed computer nodes with standby computer nodes.

[0020] According to example embodiments, a node wear leveling engine selects a given computer node from a pool of candidate computer nodes (e.g., an active computer node or a standby computer node) based on an expected remaining life of the candidate computer node. The remaining life of the candidate computer node is estimated or predicted based on information related to the wear of the computer node. More specifically, according to some embodiments, the node wear leveling engine receives data representing a wear indicator from a node wear monitoring engine (e.g., an engine distributed across active computer nodes of a distributed system). According to example embodiments, the node wear monitoring engine collects data corresponding to the wear indicator from computer nodes associated with the distributed system (e.g., active computer nodes and standby computer nodes). As described herein, a wear indicator represents or indicates a degree of degradation or wear of a physical resource associated with one or more physical resource types or categories. In this manner, as further described herein, for a particular computer node, a wear indicator can indicate or represent a degree of wear associated with physical resources of the computer node, and these physical resources can be categorized according to one or more of the following physical resource types: memory, processor, motherboard, I / O device, network device, storage device, motherboard, expansion card, riser card, network cable, and / or other physical resources.

[0021] According to an example embodiment, a node wear leveling engine processes information related to wear of a computer node to provide data representing a wear indicator of the computer node. According to an example embodiment, a node wear modeling engine (e.g., an engine distributed across active computer nodes of a distributed system) estimates or predicts the remaining life of a computer node (e.g., an active computer node and a standby computer node) based on the wear indicator. According to an example embodiment, the node wear leveling engine prioritizes or ranks the computer nodes according to their respective estimated remaining lifespans to obtain a prioritized list, referred to herein as a "ranking". As an example, the node wear leveling engine may prioritize or rank the computer nodes according to a longest (remaining) life first out (MLFO) order, such that the computer node with the largest expected remaining lifespan appears at the top of the ranking, and the computer node with the smallest expected remaining lifespan appears at the bottom of the ranking. As another example, the node wear leveling engine may rank the computer nodes in reverse order, such that the computer node with the smallest expected remaining lifespan appears at the top of the ranking, and the computer node with the largest expected remaining lifespan appears at the bottom of the ranking. Regardless of the scheme used to order computer nodes, the ranking can be used to efficiently identify computer nodes having relatively smaller expected remaining lifetimes and to efficiently identify computer nodes having relatively larger expected remaining lifetimes.

[0022] For reconfiguration of a distributed system, a node wear leveling engine may select specific computer nodes to be added to or removed from a distributed system based on rankings. Selecting computer nodes based on a remaining lifetime priority order may distribute wear evenly or nearly evenly among computer nodes, compared to a selection process that does not consider computer node wear. As a more specific example, in order to select a computer node from a spare pool to be added to an active pool of a distributed system, the node wear leveling engine may select a computer node that has the maximum expected remaining lifetime of the computer nodes in the spare pool according to the ranking. As another specific example, in order to select an active computer node (in an active computer node pool) to be removed from a distributed system, the node wear leveling engine may select an active computer node that has the minimum expected remaining lifetime of the computer nodes in the active pool according to the ranking.

[0023] According to example implementations, a node wear modeling engine may use machine learning to determine an expected remaining life of a computer node based on the wear indicators of the computer node.

[0024] According to example embodiments, in addition to the priority of the computer node as indicated by the ranking, the identification of a particular computer node may also be based on one or more factors. For example, according to some embodiments, an augmentation reconfiguration may involve adding five additional computer nodes to a distributed system. The node wear leveling engine may first identify a list of computer nodes based on the ranking (e.g., a list of fifteen computer nodes with corresponding fifteen maximum expected remaining lifespans). The node wear leveling engine may then apply one or more other selection criteria (e.g., a random-based selection, a pseudo-random-based selection, or a node ID-based selection) for selecting five computer nodes from the list.

[0025] Although the node wear leveling engine, the node wear monitoring engine, and the node wear modeling engine are described herein as separate components, according to further example embodiments, one, two, or all three of these components may be combined.

[0026] Figure 1 It includes N computer nodes 101 ( Figure 1 1, 101-2, and 101-N) are depicted in FIG. Distributed system 100 is a non-virtualized system. Figure 3 and Figure 4 Examples of virtualized systems are further discussed.

[0027] A computer node is an electronic device with processing resources capable of executing machine-readable instructions. Examples of computer nodes may include server computers (e.g., blade servers, rack-based servers, or stand-alone servers), desktop computers, notebook computers, tablet computers, and other processor-based systems. Figure 1 Specific components for computer node 101-1 are illustrated in and discussed herein. It should be noted that according to example embodiments, other computer nodes 101 may have similar components. According to some embodiments, computer nodes 101 may be homogeneous, or have the same or similar components.

[0028] Computer node 101 includes a physical processor set 112 (single processor or multiple processors). As used herein, the term "set" may refer to a single item or multiple items. The processor may include a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit, a programmable gate array, or another hardware processing circuit. As a more specific example, physical processor 112 may be a CPU core, a CPU core set, a CPU semiconductor package (or "socket"), a GPU core, a GPU core set, or a GPU semiconductor package.

[0029] The computer node 101 also includes a physical memory 120, which can be implemented using a collection of physical memory devices. In general, the memory devices forming the physical memory 121 and the other memories and storage media described are referred to herein as examples of non-transitory machine-readable storage media (e.g., non-transitory storage media that can be read by the computer node 101 and / or can be read by the distributed system 100). According to example embodiments, the machine-readable storage medium can be used for various storage-related and computing-related functions of the distributed system 100, for example, to store machine-readable instructions, such as instructions related to the node wear leveling engine 170, the node wear monitoring engine 172, and / or the node wear modeling engine 174 further described herein. As another example, the storage-related and computing-related functions of the machine-readable storage medium can include storing data associated with the computing functions performed by the computer node 101 and providing access to it, such as storing parameters, initial data sets, intermediate result data sets, final data sets, and data describing jobs to be performed by the computer node. As other examples, the storage-related and computing-related functions of the machine-readable storage medium may include storing machine-readable instructions and data related to an operating system, a baseboard management controller (BMC), a driver, an application, firmware, network processing, security intrusion detection, security intrusion prevention, access rights management, encryption processing, firmware verification, driver verification, and / or other instructions and providing access thereto. As yet another example, the storage-related and computing-related functions of the machine-readable storage medium may include: storing and updating data representing tables, rankings, mapping information, parameters related to system management; and storing and updating other data. As an example, a memory device may include a semiconductor memory device, a flash memory device, a memristor, a phase change memory device, a magnetic storage device, a combination of one or more of the aforementioned storage technologies, and a memory device based on other technologies. In addition, unless otherwise specified herein, a memory device may be a volatile memory device (e.g., a dynamic random access memory (DRAM) device, a static random access (SRAM) device, etc.) or a non-volatile memory device (e.g., a flash memory device, a read-only memory (ROM) device, etc.).

[0030] Computer node 101-1 further includes an operating system (OS) 120 and a set of programs 130 (e.g., applications). Since the example distributed system 100 is a non-virtualized system, OS 120 is a host OS (i.e., an OS that is not running in a virtual machine (VM)). Examples of OS include any or some combination of the following: Linux OS, Microsoft WINDOWS OS, Mac OS, FreeBSD OS, etc. (Multiple) application programs 130 and OS 120 are executable on processor set 112.

[0031] According to an example embodiment, the distributed system 100 includes components for distributing system wear (i.e., distributing system wear between computer nodes associated with the distributed system 100). In this context, the computer nodes associated with the distributed system 100 may include active computer nodes (such as computer nodes 101-1 to 101-N) and spare computer nodes that may be added to the distributed system 100 later and become active computer nodes. "Spare computer nodes" in this context refer to computer nodes that are currently inactive, such as dedicated spare computer nodes or computer nodes that are currently active computer nodes in another distributed system but can be removed from the system. According to an example embodiment, the components for distributing wear between computer nodes associated with the distributed system 100 include a node wear leveling engine 170, a node wear monitoring engine 172, and a node wear modeling engine 174. According to an example embodiment, one, two, or all three of these components may be distributed between computer nodes 101-1 to 101-N. For example, according to some embodiments, each computer node 101 may have software that forms a component of the distributed node wear leveling engine 170. As another example, according to some embodiments, each computer node 101 may have software forming part of the distributed node wear monitoring engine 172. As another example, according to some embodiments, each computer node 101 may have software forming part of the distributed node wear modeling engine 174.

[0032] According to some embodiments, one, two, or all three of the node wear leveling engine 170, the node wear monitoring engine 172, and the node wear modeling engine 174 may be combined. Moreover, according to further embodiments, the individual functions of the node wear leveling engine 170, the node wear monitoring engine 172, and the node wear modeling engine 174 may be distributed among these engines in a manner different from that described herein.

[0033] According to some embodiments, one or more of the node wear leveling engine 170, the node wear monitoring engine 172, and the node wear modeling engine 174 may be located on a subset of the computer nodes 101 that is less than N. For example, according to some embodiments, one or more of the engines may be located only on the computer node 101-1. In addition, according to further embodiments, instead of being located on the computer node 101, one, two, or all three of the node wear leveling engine 170, the node wear monitoring engine 172, and the node wear modeling engine 174 may be hosted by one or more management (management or administrative) nodes (such as management node 199) of the distributed system 100.

[0034] As used herein, an "engine" may refer to one or more circuits. For example, a circuit may be a hardware processing circuit that may include any or some combination of a microprocessor, a core of a multi-core microprocessor, a microcontroller, a programmable integrated circuit (e.g., a programmable logic device (PLD), such as a complex PLD (CPLD)), a programmable gate array (e.g., a field programmable gate array (FPGA)), an application specific integrated circuit (ASIC), or another hardware processing circuit. Alternatively, an "engine" may refer to a combination of one or more hardware processing circuits and machine-readable instructions (software and / or firmware) that may be executed on the one or more hardware processing circuits.

[0035] According to an example embodiment, the node wear monitoring engine 172 collects or gathers information or data related to system wear, that is, information related to the wear of computer nodes associated with the distributed system 100. According to example embodiments described herein, the information related to the wear of computer nodes may directly or indirectly indicate a parameter referred to herein as a "wear indicator." According to example embodiments, the node wear monitoring engine 172 may collect information related to wear from active computer nodes and periodically update the information so that the information is current. In this way, current information related to the wear of a particular active computer node may be used to maintain an accurate prediction of the remaining life of the node.

[0036] In addition, in the event that an active computer node is deactivated and, for example, returned to a standby pool, the expected remaining life of the computer node is available and can be used to decide whether to add the computer node back to the distributed system at a future time, as described herein. According to some embodiments, node wear monitoring engine 172 can collect wear-related information from powered-on standby computer nodes (e.g., standby computer nodes that have never been active in distributed system 100) so that the expected remaining life of these computer nodes can be determined and used to decide whether to add these computer nodes to distributed system 100 in the future.

[0037] For example, node wear monitoring engine 172 may extract information related to the wear of computer node 101 from stored data associated with computer node 101. For example, node wear monitoring engine 172 may read information related to the wear of computer node 101 from hardware registers 116 (e.g., baseboard management controller (BMC) registers, network interface controller (NIC) registers, central processing unit (CPU) registers, motherboard registers, expansion card registers, and / or graphics processing unit (GPU) registers). As another example, node wear monitoring engine 172 may read information related to the wear of computer node 101 from logs 134 (e.g., system event logs and / or health logs) stored in storage associated with computer node 101. As shown in Figure 1 In the example implementation depicted in FIG. 1 , log 134 for computer node 101 - 1 is stored in physical memory 120 of computer node 101 - 1 .

[0038] Depending on the particular implementation, the node wear monitoring engine 172 may take one of many different forms. For example, according to some implementations, the node wear monitoring engine 172 may be disposed on one or more computer nodes 101. According to further implementations, the node wear monitoring engine 172 may be located on one or more management nodes (such as the example management node 199).

[0039] According to further embodiments, the node wear monitoring engine 172 may include the BMC 114 of the computer node 101. As used herein, a "BMC" or "baseboard management controller" is a dedicated service processor that uses sensors to monitor the physical state of a server or other hardware and communicates with a management system through a management network. The BMC may also communicate with an application executed at the operating system level through the following: an input / output controller (IOCTL) interface driver, a representative state transfer (REST) ​​API, or some other system software agent that facilitates communication between the BMC and the application. The BMC may perform hardware-level access to hardware devices located in a server chassis including system memory. The BMC may be able to modify hardware devices directly. The BMC may operate independently of the operating system of the system in which the BMC is set. The BMC may be located on a motherboard or main circuit board of a server or other device to be monitored. The BMC may be installed on another board connected to the motherboard. The fact that the BMC may be installed on the motherboard of a managed server / hardware or otherwise connected or attached to a managed server / hardware does not prevent the BMC from being considered "separated" from the server / hardware. As used herein, a BMC has management capabilities for subsystems of a computing device and is separate from the processing resources that execute the computing device's operating system. The BMC is separate from the processor (such as a central processing unit) that executes a high-level operating system or hypervisor on the system.

[0040] For an example implementation of a given computer node 101, where the node wear monitoring engine 172 includes a BMC 114, the BMC 114 may, for example, read registers and event logs of the computer node 101 and process the data to drive data representing a wear indicator. For example, the BMC 114 may read registers and / or event logs according to a schedule, and / or read registers and / or event logs in response to the occurrence of a particular event of the computer node. According to some implementations, the BMC 114 may store data in registers and / or memory of the BMC 114 that may be read by the node wear leveling engine 170. According to further implementations, the BMC 114 may store data representing a wear indicator in a memory external to the BMC 114.

[0041] As another example, according to some embodiments, the node wear monitoring engine 172 may include a daemon process (e.g., an OS process) of the computer node 101. As another example, according to some embodiments, the node wear monitoring engine 172 may be or include a chassis management controller. For example, the computer node 101 may be a blade server or a rack-based server housed in a rack containing a chassis management controller. As another example, the node wear monitoring engine 172 may be formed by firmware executed on the computer node 101. As another example, the node wear monitoring engine 172 may be executed in a system management mode (SMM) on the computer node 101.

[0042] According to an example embodiment, the node wear leveling engine 170 processes the information related to wear (which is provided by the node wear monitoring engine 172) to obtain a wear indicator for the computer node 101. As an example, the information related to wear can directly indicate or represent a specific wear indicator. As an example, the node wear leveling engine 170 can process the information related to wear to obtain a specific wear indicator. According to further embodiments, the node wear monitoring engine 172 can process the information related to wear collected by the engine 172 to obtain a wear indicator and provide data representing the wear indicator to the node wear leveling engine 170.

[0043] Any of a plurality of different wear indicators may be used to derive a measure of degradation or wear of a given computer node 101 and / or an estimate of the remaining life of a given computer node 101. As an example, a wear indicator may directly indicate the age of a computer node 101, such as the total usage time of the computer node 101 (e.g., the time the computer node has been in use).

[0044] As another example, the wear indicator may represent the wear state of the physical memory of the computer node 101. For example, the wear indicator may be the cumulative number of memory bit errors recorded in the history of the computer node 101. As another example, the wear indicator may be the current time rate of memory bit errors occurring in the computer node 101 (e.g., the time rate measured over a sliding time window extending to or close to the current time). As another example, the wear indicator may be the total number of times the computer node 101 repairs the physical memory using self-healing measures (e.g., soft post-pack repair (sPPR), hard PPR (hPPR), rank sparing, and / or group sparing). As another example, the wear indicator may be the current time rate of self-healing repairs performed by the computer node 101.

[0045] As other examples, the wear indicator may be the total number of times that a particular self-healing measure (e.g., sPPR, hPPR, rank spare, or group spare) has been applied by computer node 101 over the lifetime of computer node 101 or the current rate at which a particular measure has been applied. As other examples, the wear indicator may be the total number of memory bit error corrections (e.g., error correction code (ECC)) or the current time rate of memory bit error corrections performed by computer node 101. As another example, the wear indicator may be the total time that a particular physical memory module of computer node 101 has been in use. As another example, the wear indicator may be the model number of the memory module. As another example, the wear indicator may be a particular pattern of memory errors.

[0046] As another example, a wear indicator may represent the wear state of a physical processor (e.g., a CPU or GPU) of computer node 101. A processor chip may fail after many processing cycles. Typically, as a processor degrades, the temperature increases. As the temperature increases, the clock rate of the processor may be slowed down to reduce the risk of failure. Although the processor may continue to operate in this degraded state (still calculating correct results), it will eventually fail if not corrected. According to an example embodiment, a wear indicator may indicate the total number of times a particular CPU or GPU has operated at a reduced frequency. As another example, a wear indicator may indicate the current time rate at which a particular physical processor is operating at a reduced frequency.

[0047] As an example, a wear indicator may indicate the total number of times a particular physical processor has been operated at an elevated temperature in its history. As another example, a wear indicator may indicate the current rate of time that a particular processor has been operating at a reduced frequency. As another example, a wear indicator may indicate the total time that a particular processor has been in use. As another example, a wear indicator may indicate the total number of times a particular physical processor has been operated at an elevated temperature in a worm gear mode at a frequency exceeding the nominal maximum frequency of the processor. As another example, a wear indicator may indicate the current rate of time that a particular processor has been operating in a worm gear mode. As another example, a wear indicator may indicate the total number of errors associated with a cache of a processor or the current rate of time of errors associated with a cache of a processor. As another example, a wear indicator may be the current average number of cycles per instruction of a processor. As another example, a wear indicator may be the current rate of level 1 (L1) cache or level 2 (L2) cache hits. As another example, a wear indicator may be the current rate of L1 cache or L2 cache misses. As another example, a wear indicator may be the model of a processor. As another example, a wear indicator may be specific to an error. As another example, the one or more wear indicators may be performance parameter(s) provided by a performance monitoring unit (PMU) of a processor.

[0048] As another example, the wear indicator of computer node 101 may indicate the wear status of a motherboard or another board (e.g., a riser card or a mezzanine card) of computer node 101. For example, the wear indicator may be the model of the motherboard, the date when the motherboard was first put into use, or the accumulated usage time of the motherboard.

[0049] As another example, the wear indicator of the computer node 101 can indicate the wear state of the firmware or software of the computer node 101. As an example, the wear indicator can be a version or signature of a unified extensible firmware interface (UEFI) image or a version of a UEFI driver. As another example, the wear indicator can be a version or signature of a basic input / output system (BIOS) image. As another example, the wear indicator can be a version or signature of a container generated file. As another example, the wear indicator can be a version or signature of a firmware management stack of a BMC of the computer node 101. As another example, the wear indicator can be a version or signature of an operating system of the computer node 101.

[0050] As other examples, the wear indicator may be the age or network cable associated with computer node 101, the total number of errors associated with a particular network interface controller (NIC) of computer node 101, the current error rate associated with the NIC, the total accumulated time that the NIC has been in use, or the version of the NIC. As other examples, the wear indicator may be the total number of errors or the current error rate of a storage device of computer node 101. As other examples, the wear indicator may be a history of specific failures of computer node 101, such as a history of temperature failures. As another example, the wear indicator may be the age or version of a power supply of computer node 101. As another example, the wear indicator may be the age or version of a fan of computer node 101. As another example, the wear indicator may be a history of thermal management of computer node 101. As another example, the wear indicator may be a history or number of security check failures, a history or number of security intrusion detections, or a history or number of physical tampering detections. As another example, a wear indicator may be a maintenance history of computer node 101 (e.g., data indicating patterns of hardware component replacements, whether certain hardware components have been replaced, problems with computer node 101, and / or error logs leading to maintenance of computer node 101).

[0051] According to an example embodiment, the node wear leveling engine 170 receives data representing wear indicators for a given computer node 101 from the node wear monitoring engine 172, and uses the node wear modeling engine 174 to determine the expected remaining life of the computer node 101 based on these wear indicators. According to an example embodiment, the node wear leveling engine 170 may also construct a multi-dimensional feature vector from the wear indicators and provide the feature vector to the node wear modeling engine 174. The node wear modeling engine 174 may then provide data representing the expected remaining life back to the node wear leveling engine 170 based on the feature vector. As an example, the feature vector may be a tuple containing a selected set of wear indicators, a tuple containing a weighted set of wear indicators, a tuple containing a selected subset of wear indicators, or a combination of wear indicators in general or features derived from wear indicators.

[0052] According to an example implementation, the node wear leveling engine 170 ranks or prioritizes the expected node remaining lifespans of all computer nodes (e.g., active and standby computer nodes) associated with the distributed system 100 to obtain the node lifespan-based ranking 131. For example, the node wear leveling engine 170 may update the node lifespan-based ranking 131 as the expected node remaining lifespan is updated. Figure 1 As depicted in , the ranking 131 based on node age can be stored as data 126 in the physical memory 120. According to some embodiments, the node wear leveling engine 150 prioritizes or ranks the computer nodes according to the MLFO algorithm.

[0053] In response to the reconfiguration of the distributed system 100, the node wear leveling engine 170 may identify or select a computer node to be added or removed based on the node lifespan-based ranking 131. More specifically, according to some embodiments, for the purpose of adding a spare computer node, the node wear leveling engine 170 selects a node with a maximum expected remaining lifespan based on the ranking 131. As another example, according to some embodiments, for the purpose of removing a computer node 101 from the distributed system 100, the node wear leveling engine 170 selects a computer node 101 with a minimum expected remaining lifespan.

[0054] According to further embodiments, the ranking 131 may be used as one or more factors considered by the node wear leveling engine 170 when evaluating whether a particular computer node should be selected for reconfiguration of the distributed system 100. For example, according to some embodiments, if a particular standby computer node has a relatively long expected remaining life, the node wear leveling engine 170 may give priority to selecting the standby computing node, but the node wear leveling engine 170 may consider, for example, whether the standby computer node is currently part of a lower priority system (and is therefore to be removed from this system) or is not currently affiliated with any other system. As another example, according to some embodiments, the node wear leveling engine 170 may identify a list of a predetermined number of computer nodes from the ranking 131, and further base the selection of one or more computer nodes from the list on one or more other criteria (e.g., the selection process may include randomly or pseudo-randomly selecting (multiple) computer nodes from the list and / or may include further basing the selection on the node ID).

[0055] Depending on the particular implementation, the node wear leveling engine 170 can take one of many different forms. For example, the node wear leveling engine 170 can be part of the OS 120. As other examples, the node wear leveling engine 170 can be or include a BMC, a chassis controller, an operating system daemon, or firmware. According to some embodiments, the node wear leveling engine 170 can be provided on one or more computer nodes 101. According to further embodiments, the node wear leveling engine 170 can be located on one or more management nodes (such as the example management node 199).

[0056] According to some embodiments, the node wear modeling engine 174 can estimate the respective remaining lifespans of computer nodes associated with the distributed system 100 based on information related to the wear of the computer nodes 101 (e.g., feature vectors provided by the node wear leveling engine 170). According to some embodiments, the node wear modeling engine 174 can be disposed on one or more computer nodes 101. According to further embodiments, the node wear modeling engine 174 can be located on one or more management nodes (such as the example management node 199).

[0057] As an example, the node wear modeling engine 174 can use supervised learning, wherein the machine learning model can be initially trained based on labeled training data. As an example, the labeled training data can be obtained from a knowledge database and / or expert guidance. As a more specific example, the training data can represent the history of observed computer nodes, their actual lifespans, benchmarks corresponding to events that occurred with respect to the computer nodes, and data typically collected from the computer nodes during active service of the computer nodes. As further described herein, according to some embodiments, the training data can be collected during the maintenance and / or inspection of a particular computer node, the lifespan of which has been estimated by the node wear modeling engine 174 for use in providing feedback to improve the lifespan estimates provided by the engine 174.

[0058] According to further embodiments, the node wear modeling engine 174 may use unsupervised machine learning. As other examples, the node wear modeling engine 174 may use other types of machine learning (in addition to supervised or unsupervised learning), such as reinforcement learning or semi-supervised machine learning.

[0059] Depending on the particular implementation, the machine learning model used by the node wear modeling engine 174 to estimate the remaining node life can take any of a number of different forms. As examples, the model can be an artificial neural network (ANN), a decision tree, a support vector machine (SVM), a regression process, a Bayesian network, a Gaussian process, or a genetic algorithm.

[0060] According to some embodiments, the node wear leveling engine 170 may prioritize the wear indicators to form a feature vector provided by the engine 170 to the node wear modeling engine 174. For example, according to some embodiments, the node wear leveling engine 170 may assign wear indicator weights and / or select wear indicators for the feature vector for prioritizing processor-related wear indicators (e.g., prioritizing the current rate of under-frequency operation and / or the current rate of over-temperature occurrence) and memory-related wear parameters (e.g., prioritizing the occurrence of a certain memory error mode and / or a certain self-healing action mode) relative to wear parameters associated with other computer node subsystems. According to some embodiments, the priority order of the wear indicators and / or the corresponding bias built into the machine learning model may be used to guide the classification performed by the node wear modeling engine 174. According to further embodiments, the node wear leveling engine 170 may observe over time that some wear indicators have a greater impact on the estimated node life than other wear indicators and adjust the feature vector accordingly to include more influential wear indicators and not include other less influential wear indicators.

[0061] According to further example embodiments, the node wear modeling engine 174 may estimate the remaining node life without using machine learning. For example, the node wear modeling engine 174 may use a tuple of wear level indicators as an index to look up the expected remaining life from a table. As another example, according to some embodiments, the node wear modeling engine 174 may apply different weights to the wear indicator values ​​(to accordingly assign a greater preference to some wear indicators than other wear indicators) and apply a mathematical function to the weighted wear indicator values ​​to obtain the expected remaining life. For example, the weights may prioritize processor-related wear indicators (e.g., prioritizing the current rate of under-frequency operation and / or the current rate of over-temperature occurrence) and memory-related wear parameters (e.g., prioritizing the occurrence of a certain memory error mode and / or a certain self-healing measure mode) relative to wear parameters associated with other computer node subsystems.

[0062] According to some embodiments, the distributed system 100 may be associated with a management node 199. In general, the management node 199 may issue commands related to setting up and managing the distributed system 100. For example, according to some embodiments, the management node 199 may monitor the job or application workload of the distributed system 100 and, in response thereto, initiate (e.g., manually or automatically through a management console, depending on the particular embodiment) a reconfiguration of the distributed system 100 to expand or reduce. As part of the reconfiguration, the management node 199 may transmit a request to the node wear leveling engine 170, requesting the node wear leveling engine 170 to identify one or more computer nodes related to the reconfiguration (e.g., an active computer node to be discarded or a computer node to be added from a spare pool). According to example embodiments, the management node 199 may perform other actions related to the management of the distributed system 100. According to some embodiments, one or more management nodes 199 may perform some or all of the functions described herein for the node wear leveling engine 170, the node wear monitoring engine 172, and the node wear modeling engine 174.

[0063] According to an example implementation, Figure 2 The techniques described in can be used to reconfigure distributed systems. Figure 2, the technique includes the node wear monitoring engine 172 collecting data from computer nodes associated with the distributed system at 204 and providing information 208 related to wear of the computer nodes to the node wear leveling engine 170. In response to the information 208 related to wear and as depicted at 212, the node wear leveling engine 170 can use the node wear modeling engine 174 to estimate the node remaining life 214. More specifically, using the node wear modeling engine 174 can include providing the feature vector to the node wear modeling engine 174 (as depicted at 213) and receiving the estimated node remaining life 214 from the node wear modeling engine 174. As depicted at 220, the node wear leveling engine 170 can perform an update 220 to the node life-based ranking 131 in response to the estimated node remaining life 214.

[0064] In response to the configuration change event 224, the node wear leveling engine 170 may then respond to the configuration event as depicted at 230. This response may include identifying one or more computer nodes to be added and / or replaced based on the node age-based ranking 131.

[0065] Figure 3 3 is a block diagram of a virtualized distributed system 300 according to an example implementation. The distributed system 300 includes N computer nodes 301 ( Figure 3 301 - 2 and 301 -N). Each computer node 301 of the distributed system 300 includes a VM. For example, computer node 301 - 1 may include M VMs 310 ( Figure 3 301 -1). VM 310 is executable on processor set 312 of computer node 301 -1. Computer nodes 301 -2 through 301 -N may include similar components as computer node 301 -1.

[0066] VM 310-1 includes a guest OS 320 and a program set 330 that can be executed on VM 310-1. Address mapping information can also be provided for VM 310-1, which maps between guest physical addresses of the guest physical memory address space of VM 310-1 and physical addresses of the physical address space. The guest physical address space refers to the location of the guest physical memory. From the perspective of the guest OS 320, the guest physical memory is regarded as a physical memory. However, the guest physical memory is actually a virtual memory provided by a hypervisor 350 in the computer node 301-1. The hypervisor 350 manages the sharing of the physical memory 321 in the computer node 301-1 between VMs 310. Other VMs 310-2 to 310-M can include components similar to VM 310-1.

[0067] According to an example embodiment, the hypervisor 350 includes a node wear leveling engine 170. In other examples, the node wear leveling engine 170 or a component thereof may be part of the computer node 301-1 but separate from the hypervisor 350. According to an example embodiment, the computer node 301-1 may also include at least a portion of the node wear monitoring engine 172 and / or the node wear modeling engine 174 (e.g., a portion of a distributed engine). According to some embodiments, one or both or portions thereof of the node wear monitoring engine 172 and the node wear modeling engine 174 may be included within the hypervisor 350. Furthermore, according to further example embodiments, instead of being hosted on the computer node 301, one, two or all of the node wear leveling engine 170, the node wear monitoring engine 172 and the node wear modeling engine 174 may be hosted by one or more management (management or administrative) nodes such as the management node 399.

[0068] According to some embodiments, management node 399 may issue commands related to setting up and managing distributed system 300, including initiating a request to node wear leveling engine 170 to identify one or more computer nodes related to reconfiguration as part of a reconfiguration of distributed system 300. According to example embodiments, management node 399 may perform other actions related to management of distributed system 300.

[0069] Figure 4 4 is a block diagram of a virtualized distributed system 400 according to a further example embodiment. In the distributed system 400, instead of running one or more VMs in each corresponding computer node, VM 450 can be distributed across N multiple computer nodes 401 ( Figure 4 The computer nodes 401-1, 401-2 and 401-N) depicted in FIG. Figure 4 One VM 450 is shown, but it should be noted that there may be at least one other VM that may run across multiple computer nodes 401 - 1 to 401 -N. Figure 4 The arrangement depicted in is referred to herein as a “software defined server” or “SDS” because the computer nodes 401 - 1 through 401 -N collectively form one virtual machine (VM) 450 that hosts a guest operating system (OS) 404 .

[0070] VM 450 includes a guest OS 404 and a program set 406 running in VM 450. Address mapping information 408 is maintained for VM 450, which maps guest virtual memory addresses of the guest physical memory address space (referring to the location of the host virtual memory) to physical addresses of the hyper-kernel physical address space. The hyper-kernel physical address space is a physical address space provided by the hyper-kernel. From the perspective of guest OS 404, guest physical memory is regarded as physical memory. However, guest physical memory is actually provided by N hyper-kernels 468 ( Figure 4 Virtual memory provided by the hyper-kernels 468-1, 468-2, and 468-N) depicted in FIG.

[0071] According to an example embodiment, the hyperkernel 468 performs mapping of guest physical memory addresses to real physical memory addresses. The guest OS 404 performs mapping of guest virtual memory addresses to guest physical memory addresses (using first-level page table mapping). From the perspective of the guest OS 404, the guest physical memory addresses appear to be real physical memory addresses, but they are not. The VM 450 can maintain a virtual resource map that describes the virtual resources available to the guest OS 404 from the perspective of the guest OS 404. According to an example embodiment, the hyperkernel 468 uses second-level page table hardware and second-level address mapping information to map guest physical memory addresses to real physical memory addresses. Each hyperkernel 468 has address mapping information, which is the current resource mapping between the virtual resource map and the physical resources managed by the hyperkernel 468 from the perspective of the hyperkernel 468. According to an example embodiment, each hyperkernel 468 has resource mapping information that describes the physical resources managed by the hyperkernel 468.

[0072] Hyperkernel is software used as a part of distributed management program on each physical computer node. Hyperkernel 468 communicates with each other to jointly perform the tasks of management program. Each hyperkernel 468 can observe the operation of distributed system 400 in real time and optimize the system resources of corresponding computer node 401 to match the requirements of distributed system 400 during operation. Hyperkernel 468 unifies the hardware resources of computer node 401 and presents the unified set to guest OS 404. The hardware resources of distributed system 400 include physical processor 412, physical memory 424 and other physical resources (e.g., network communication resources, input / output (I / O) resources, etc.) of corresponding computer node 401. Due to this abstraction, guest OS 404 will have the view of a single large computer, which contains an aggregated set of processors, memory, I / O resources, network communication resources, etc.

[0073] According to an example implementation, each hypercore 468 includes a corresponding node wear leveling engine 470. For example, hypercore 468-1 includes a node wear leveling engine 470, hypercore 468-2 includes a node wear leveling engine 470, and hypercore 468-N includes a node wear leveling engine 470. Each of the node wear leveling engines 470 performs the same Figure 1 The node wear leveling engine 170 of FIG. 470 may be operated similarly to the node wear leveling engine 170 of FIG. 470. In other examples, the node wear leveling engine 470 may be external to the corresponding hyper core 468.

[0074] According to an example embodiment, each computer node 401 includes a node wear monitoring engine 472 and a node wear modeling engine 474. According to an example embodiment, the node wear monitoring engine 472 and the node wear modeling engine 474 may be similar to Figure 1 The node wear monitoring engine 172 and the node wear modeling engine 174 are provided.

[0075] According to further example embodiments, one, two or all three of the node wear leveling engine 470, the node wear monitoring engine 472 and the node wear modeling engine 474 may be combined. In addition, according to further example embodiments, one, two or all three of the node wear leveling engine 470, the node wear monitoring engine 472 and the node wear modeling engine 474 may be hosted on one or more management (management or administrative) nodes such as the management node 499.

[0076] The guest OS 404 in the VM 450 is presented as having a virtual processor (also referred to as a virtual central processing unit or vCPU) that is a virtualized representation of the physical processor 412 of the computer node 401, as presented by the distributed hypervisor composed of the hyperkernel 468. As an example, if there are five computer nodes and each computer node has 100 physical processors, the distributed hypervisor presents the guest OS 404 as having 500 virtual processors as part of a single SDS. In reality, there are five physical computer nodes, each of which has 100 physical processors.

[0077] According to an example implementation, Figure 5The technique 500 depicted in can be used to update a node wear model. According to some embodiments, the technique 500 can be performed by a node wear leveling engine. According to an example embodiment, the technique 500 includes identifying (box 504) a computer node for maintenance based on a ranking based on an expected node life. According to box 508, the system containing the identified computer node can then be reconfigured to remove the computer node for repair. According to box 512, a maintenance inspection and / or repair of the node can then be performed to obtain an expected remaining node life. This expected node life obtained via the repair / inspection of the computer node can be compared with the expected remaining life of the node provided by the node wear modeling engine.

[0078] As an example, performance and health analysis can be run on a computer node to check various indicators of the expected remaining life of the computer node. As an example, these analyses can check the storage utilization, CPU utilization, and memory utilization of the computer node. As another example, these analyses can evaluate the power consumption of the computer node. As another example, these analyses can involve determining whether the computer node exhibits abnormal behavior compared to the observed behavior of other computer nodes. According to example embodiments, the results of the analysis can be used to determine whether the computer node can be classified as belonging to a cluster associated with a particular estimated remaining life. Depending on the specific embodiment, this classification may or may not involve the use of machine learning.

[0079] According to some embodiments, the results of the performance and health analysis performed on the computer node can be used to independently derive an estimated remaining life of the computer node. The model can then be updated based on the comparison, as depicted in block 516. For example, according to some embodiments, the independent determination of the expected remaining life along with information about the wear of the computer node (which is derived through the node wear leveling model) can be used to form labeled training data that is used as supervisory feedback to the node wear modeling engine.

[0080] refer to Figure 6 According to some embodiments, the technique 600 can be used to reconfigure a distributed system. As an example, according to some embodiments, the technique 600 can be performed by a node wear leveling engine. Figure 6, process 600 includes evaluating the workload (e.g., job and / or application workload) of the distributed system according to box 604. Based on the evaluation, it can then be determined (decision box 608) whether more physical resources should be added to the distributed system. If so, then according to box 612, one or more spare computer nodes are identified based on the ranking based on the expected node life to expand the physical resources of the distributed system. In this way, according to some embodiments, the (multiple) identified spare computer nodes can be the (multiple) spare nodes with the (multiple) maximum expected remaining life. According to box 616, the distributed system can then be configured to add the (multiple) identified spare computer nodes.

[0081] Process 600 may also include determining (decision block 620) whether to remove or discard physical resources of the distributed system based on the assessed workload. If so, then according to block 624, one or more active computer nodes of the distributed system are identified based on the ranking based on the expected node lifespan to reduce the physical resources of the distributed system. In this way, according to some embodiments, the ranking can be used to identify the (multiple) active computer nodes with the (multiple) minimum expected remaining lifespan. According to block 628, the distributed system can then be configured to remove the (multiple) identified active computer nodes from the distributed system.

[0082] Return to reference Figure 4 , now turn to Figure 4 In more specific details of the SDS 400, according to some embodiments, the SDS 400 may use a resource map to transition between a virtual configuration and a physical configuration. For example, according to some embodiments, the resource map of the SDS 400 may include a physical resource map, an initial resource map, and a current resource map.

[0083] The "physical resource map" is a table that describes the physical resources available on each node. For example, the table contains the number and type of processors on each node, devices, available memory and its physical address range, etc. According to an example embodiment, this table is read-only and fixed at boot time.

[0084] The "initial virtual resource map" is fixed before the boot of the guest operating system and describes the available virtual resources from the operating system's perspective. This configuration can be read by the operating system. In some cases, it may be desirable to configure a system that does not match the underlying hardware resources one-to-one (from the operating system's perspective). As an example, it may be desirable for the operating system to have more memory and fewer cores. This can be accomplished by changing the ratio of memory to cores (i.e., by modifying the initial virtual resource map).

[0085] A "current resource map" is created and maintained by each hyperkernel instance. This map describes the current mapping between the virtual resource map and the physical resource map from the perspective of each node. For each entry in the virtual resource map, the definition of the physical resources currently assigned to the virtual resource is maintained. Initially (for example, at boot time), the current resource map is a copy of the initial virtual resource map. The hyperkernel modifies the current resource map over time as it observes the characteristics of the resource load and dynamically changes the mapping of physical resources to virtual resources (and vice versa). For example, the definition of the location of an Ethernet controller in a virtualized machine may refer to different hardware controllers at different times. The current resource map is used by the hyperkernel to dynamically modify virtual hardware resource mappings, such as the virtual memory subsystem, as needed.

[0086] The hyperkernel of the SDS 400 provides a dynamic adaptive reconfiguration framework that enhances system availability in a data center context by allowing the SDS 400 to be dynamically reconfigured without downtime due to failures or allowing required (or desired) elastic capacity changes to improve efficient utilization of computing resources. That is, this additional layer of the hyperkernel allows the SDS 400 to be dynamically adaptively reconfigured while the system is running without having to shut down the system.

[0087] Dynamic reconfiguration as referred to herein refers to the ability to change or modify the set of hardware components that implement or support the SDS 400 while the system continues to run the guest OS 404 and its applications. For example, using the techniques described herein, the resources of a distributed multi-server architecture can be reconfigured while the operating systems running across the multi-server architecture continue to run, without having to reboot and without any interruption to running applications.

[0088] According to example embodiments, the physical hardware components of the SDS 400 may be grouped and the groups may be associated with corresponding resilient logic modules (also referred to herein as "logic modules"). According to example embodiments, the logic modules have relatively graceful failure modes compared to the physical hardware components, rather than immediate catastrophic failures. As described herein, graceful failures involve transitioning to a degraded state such that the SDS 400 continues to function and data is not lost; depending on the degree of redundancy of the available hardware, a later failure may be catastrophic if the failed component is not replaced.

[0089] In this type of approach, the SDS 400 includes elastic logic modules that allow relatively high levels of availability (e.g., "continuous availability" with an arbitrary number of availability "9s") through dynamic reconfiguration despite physical hardware outages (e.g., due to hardware failures, physical resource scaling, etc.). The main causes of data center downtime are typically related to system capacity reduction, system capacity growth, and component failures during runtime (wherein outage times increase as existing data center systems become larger). Using the elasticity / reconfiguration mechanisms described herein, these causes do not necessarily result in any downtime for the SDS 400. This also minimizes outage costs caused by hardware (e.g., component failures, reconfiguration, repairs, etc.) by allowing in-place failures / planned repairs, hot standby, etc., as will be described in further detail below. It also reduces outage costs caused by software (e.g., software errors, operator errors, overloads, etc.) by allowing thin provisioning, which will be described in further detail below.

[0090] As described above, the dynamic reconfiguration mechanism described herein allows various enhancements to system availability in the context of a data center, examples of which are performed by way of non-limiting: allowing dynamic reconfiguration of the SDS 400 without downtime due to failures and allowing required or desired elastic capacity changes to improve efficient utilization of computing resources. Using the techniques described herein, it is achieved to continue the operation of the SDS 400 despite hardware interruptions (e.g., component failures, rescaling, etc.). Further details regarding such enhancements are described in further detail below.

[0091] The ability to facilitate required or desired capacity changes to the SDS 400 (i.e., dynamic reconfigurability of the SDS 400) is referred to herein as "thin provisioning" of the SDS 400. One advantage of the hyperkernel described above is that the hyperkernel creates a software layer between the guest OS 404 and the underlying hardware. As a result, the hardware and the guest OS 404 and its applications are more loosely coupled than in previous data centers. Thin provisioning takes advantage of this loose coupling, allowing available virtual resources to be moved between hardware nodes and dynamically remapped to physical resources (e.g., via the resource migration techniques described above). For example, storage volumes, memory, and central processing units can be instantiated from physical resources as needed, rather than being selected and fixed in capacity for the duration of the operation of the SDS 400. As an example, the SDS 400 may be started with 1TB (terabyte) of physical DRAM (dynamic random access memory) and 10 physical processors, and later upgrade its physical instantiation to have more DRAM and more CPUs while the system is running without having to restart the guest OS 404.

[0092] Because virtual memory pages, virtual CPUs, input / output (I / O) devices, and other virtual resources are movable and dynamically reconfigurable throughout the SDS 400, virtual machines supported / presented by the hyperkernel can use the techniques described herein to add and remove sufficient physical resources to support the virtual resources, and then automatically remap those physical resources to additional or different physical resources. This provides the flexibility to scale up and scale down the system's hardware capacity as needed as workload requirements change, thereby making more efficient use of computing resources.

[0093] Indirectly, this also provides a form of resilience not only to failures but also to improved utilization economies. Using the dynamic reconfiguration mechanism described herein, the expansion and contraction of workloads can be accommodated cost-effectively, easily, and automatically. For example, excess resources can be used by other SDSs until needed, and the power requirements of SDS 400 can be adjusted in proportion to the needs of the workload.

[0094] Being able to dynamically reconfigure the SDS 400 without downtime due to failures allows in-place failures and further allows, for example, repair scheduling. The dynamic reconfiguration mechanism described herein therefore has additional benefits for resilience. Physical devices can be suspended from service for repair, replacement, or upgrade while the SDS 400 and its guest OS 404 continue to run. Replaceable devices can be removed from the SDS without interrupting the SDS 400. In addition, physical devices can remain in place for any period of time until they can be scheduled for removal for replacement or upgrade. This flexibility is an improvement over existing data center environments, where the techniques described herein reduce downtime by allowing the system to continue to run while allowing appropriate repairs to be performed correctly, for example, without the need for an "emergency standby expert."

[0095] Further details regarding handling hardware failures while allowing the SDS 400 and its guest OS 404 to continue to operate are described below.

[0096] As used herein, a "component" refers to any hardware portion of SDS 400, such as a chip, board, housing, cable, etc. In contrast, as used herein, a "module" refers to a combination or collection of one or more physical components that are combined to create or provide the abstract functionality of SDS 400 or a hyperkernel. One property of an SDS 400 module is that the hardware functionality of the module can be understood without understanding the details of the module's construction, and the operating state of the module can be efficiently described.

[0097] According to an example embodiment, resilient commodity hardware components with fault tolerance are used in SDS 400 (and modules are created), such as error correction code (ECC) memory, processor over-temperature throttling, software redundant array of independent disks (RAID), multi-path storage on fabric, combined network, etc.

[0098] Examples of module types in SDS 400 are described in further detail below. In various embodiments, a module includes: a node (e.g., one or more processor chips, a motherboard, its power supply, and a plurality of memory DIMMs (dual in-line memory modules), etc.), an interconnection network (e.g., switches, cables, and NICs (network interface controllers) providing connections between nodes), and an addressable guest network interface (e.g., a port on a hardware NIC).

[0099] As used herein, a resilient logic module (also referred to herein as a "resilient module" or "module") fails gracefully (e.g., degrades rather than fails). A resilient module includes a device that fails into a degraded state. Although suffering from performance degradation, the resilient module continues to function so that there is no data loss. However, a failure can be catastrophic when in a degraded state. Degraded components are components that need to be replaced in a timely manner to maintain system resiliency. Examples of these components include commodity ECC memory, RAID1 / 5 volumes, combined NICs, redundant power supplies, etc.

[0100] According to an example implementation, a resiliency module in SDS 400 is a logical component that can be described as one or more hardware components that have one of five operational states: available, operational, degraded, faulted, and disconnected. An available module is a module that is ready for use in SDS 400 but is not actually in use. An operational module is a module that is connected to SDS 400 and is being used by the hypercore. An operational module is functioning correctly with full performance.

[0101] A degraded module is a module that is connected to the SDS 400 and is being used by the hypercore. Although the degraded module functions correctly, its performance may not meet specifications and should be replaced promptly or it will enter a faulty operating state. A faulty module is a module that is connected to the SDS 400 and causes the hypercore to fail. A faulty module cannot function correctly.

[0102] A disconnected module is a module that is not connected to the SDS 400. If a disconnected module is connected to the SDS 400, it will cause the hypercore to fail. A disconnected module cannot function properly. The SDS 400, whose connected modules are either operational or degraded, is functioning. An SDS 400 with any module that has failed cannot operate. Available modules and disconnected modules are not connected to the SDS 400 and have no impact on system availability.

[0103] Logic modules are designed to fail gracefully. Using resilient physical components as described above (such as error correction memory, CPU modules that reduce their clock speed when overheated, software RAID, redundant switches, combined network ports, etc.), most failures (e.g., 99.9%) will be failures that fail gracefully and degrade, and cause a transition from an operational state to a degraded state. That is, the most common data center failures can be brought into the resilient module (e.g., power supply, network connection (connector, NIC), disk drive, ECC memory, etc.). The failure of redundant components causes the module operation to degrade, but the system remains operational, which requires repair. These failures that cause a transition from a degraded state to a failed state will then cover the second failure. In contrast, only a small number of failures (e.g., 0.01% of failures) will be failures that directly transition the logic module from an operational state to a failed state.

[0104] Available modules can be added to or removed from the SDS 400.

[0105] A disconnect module can be removed from the SDS 400 or repaired in place (e.g., to rewire a network cable). A disconnect module should not be added to the SDS 400.

[0106] As used herein, reconfiguration of the SDS 400 refers to the process of binding and unbinding logical modules to physical components and binding and unbinding virtual machine components to logical modules. The difference between logical modules and physical components is a form of virtualization (although a different type of virtualization than virtualizing processors, memory, and I / O devices to create virtual machines executed by the hyper-kernel). According to an example embodiment, enhancing the hyper-kernel by adding the ability to dynamically reconfigure involves creating a separation that allows replacement of hardware components (on which the hyper-kernel operates). For example, the hyper-kernel is divided into two layers. One layer includes logical modules (described in further detail below) that manage certain physical resources in the SDS 400. The second layer is referred to as the "upper layer" of the hyper-kernel that works with any logical module on any node.

[0107] In an example embodiment, a logic module representing a hardware component or other hardware resource / component group in a hyperkernel is implemented as a data table. As will be described in further detail below, an example of a logic module is a logic module representing a physical disk drive. For example, a corresponding data table describes the location of the disk drive in the SDS 400, transactions not processed by the disk drive, etc. Therefore, the logic module includes metadata about the physical components maintained by the hyperkernel. Since the logic module is not hardware, the logic module can be migrated. That is, the logic module implementation method freely migrates its use of the physical component, and the physical component can be moved transparently. The migration of the logic module can be performed without knowing the guest OS 404. That is, this layer of the logic module is hidden from the guest OS 404.

[0108] Thus, in the dynamic reconfiguration framework described herein, the hyperkernel operates on components of logic modules that are bound to physical components at any particular time. The hyperkernel then also provides a set of virtual resources forming a virtual machine to the guest system by binding and unbinding virtual resources to the logic modules. The dynamic reconfiguration layer described herein includes a layer across the architecture of the hyperkernel, namely the logic modules.

[0109] There is a split between logical modules and virtualized objects. The hyperkernel includes an internal layering of resilient logical modules that are degraded rather than disconnected. This allows the SDS 400 to be dynamically reconfigured and physical resources to be thinly provisioned while the guest OS 404 continues to run. The dynamic reconfiguration techniques described herein also provide solutions to two resiliency aspects, where the system can continue to operate despite the failure of hardware components in the resilient modules, and there is no downtime due to scale changes or preventive maintenance.

[0110] According to example implementations, hardware components (there may be various hardware components serving similar functions) may be abstracted into multiple logical module types: nodes, time bases, network ports, and storage volumes.

[0111] According to an example implementation, a node logic module (also referred to herein as a "logical node") corresponds to a hypercore node. Internally, a node logic module has a CPU and memory. A node logic module may also hold other logic components of other logic module types. Holding represents a higher level aspect of reconfigurability.

[0112] According to an example implementation, there is a time base logic module in the SDS 400. The time base logic module represents the time base of the SDS 400, which is used, for example, for the virtual timestamp counter and various virtual hardware clocks in the synthesis system. In an example implementation of the SDS 400, the time base logic module is the master clock and is logically part of a specific node module.

[0113] The bus port logic module represents a high-speed interconnect from a logical node to other logical nodes attached to an Internet switch. According to an example implementation, there is one bus port logic module in each operational node logic module.

[0114] The network port logic module represents a network interface port. For example, a Virtio network virtual device can be implemented using the network port logic module. The storage volume logic module represents a logical drive controller. For example, a Virtio block virtual device can be implemented using the storage volume logic module.

[0115] The relationship between logical modules and physical components in SDS 400 is not necessarily one-to-one. According to some embodiments, drivers are used that transition between operations on logical modules and operations on physical components. In various embodiments, this may involve mapping logical module identifiers to physical component identifiers, but may also involve more complex mappings when multiple physical components are used to create elastic and reconfigurable logical modules.

[0116] The physical components of the distributed logic module can span multiple nodes. The logic module can relocate functions across different sets of nodes at any time. According to an example embodiment, the relocation is function-specific. For example, to remove a computer node: the network interface switches the MAC address to a NIC port on another computer node; the local drive array across the nodes in the RAID5 structure simply discards the drive on one computer node and adds an empty driver on another computer node; and with respect to the time-based logic module, the main computer node moves itself to another computer node and discards the current computer node. The guest OS 404 is not aware of the relocation. The relocation process does not bring about functional interruption.

[0117] According to an example implementation, the logic module interface in the dynamic reconfiguration framework described herein is defined by a set of commands, which are issued by the hypercore using the logic module. Commands can execute their jobs / tasks immediately or asynchronously. According to an example implementation, the completion of each asynchronous command is signaled by a completion queue. As an example, when issuing a command, each completion is delivered as a kernel event (also referred to as "kernel event (kevent)" herein) on a kernel queue (kernel queue) (also referred to as "kernel queue (kqueue)") specified in the command parameter. Other types of completion queues can also be implemented.

[0118] According to an example implementation, each logical module in SDS 400 has a unique identifier, also referred to herein as its "logical module ID" (LMI). The scope of the LMI is global to SDS 400 during the life of SDS 400 as an entity. A node logical module has its own LMI, which is a node identifier used in the hyperkernel. As an example, the node identifier is used in the upper half of the hyperkernel. In an example implementation, the LMI is represented as a 12-bit unsigned integer (allowing 4096 logical modules in SDS 400) and can be stored anywhere that a node identifier is stored (e.g., in a data structure used for resource tracking of pages). The size of the LMI can be as needed.

[0119] Logic modules may be relocated only rarely, e.g., only when reconfiguration occurs. Commands issued to logic modules on one computer node may need to be transmitted to another computer node. In an example embodiment, updates to logic module locations maintain a strictly consistent mapping of logic module locations across all computer nodes (this can be compared to resource tracking for virtual resources in a hyperkernel, where per-node knowledge of the location of virtual resources may be imperfect).

[0120] According to an example embodiment, an application programming interface (API) used by the hypercore partially implements the dynamic reconfiguration framework. The API may include commands issued to logic modules as procedure calls. According to an example embodiment, a dedicated interconnect is used to convert local procedure calls to remote procedure calls.

[0121] For example, the API for logical volumes includes multi-block commands such as readblox, writeblox, and flush. The API also includes Async for distributed completion queues. The API can be extended for guest memory transfers for block I / O, such as injecting guest pages (even if the current page at the guest physical address is remote) and linking to guest pages (e.g., copy-move on guest pages).

[0122] According to an example embodiment, a logic module is distributed across multiple computer nodes (while the computer nodes themselves are not distributed). For example, components of a logic module may span multiple computer nodes at any one time. By instantiating certain types of logic modules on multiple nodes at once, resiliency or performance may be enhanced. During reconfiguration, when the functionality of a logic module is replaced by a new physical component on a new node, the transient state of the logic module is distributed. Since the hyperkernel continues to run while the reconfiguration occurs, the operation of the logic module is managed across multiple nodes during this period. Distributed logic modules also allow for rapid removal of computer nodes by abandoning components.

[0123] One feature of the distributed logic module involves the use of internal messages between nodes to perform internal coordination of portions of the distributed logic module. Ordinary interconnect messages are addressed from a logic module implementation on one node to a logic module implementation on another node. The logic module implementations do not need to process virtual objects, so they do not need to interact directly with any hypercore threads. In an example embodiment, these logic module implementations are implemented using remote process classes that run in response to messages.

[0124] In an example implementation, a logic module command to a local module is a procedure call that returns quickly after completing its work. A command to a module that is not local to a computer node will be sent to one of the computer nodes involved in the module and will return, for example, when the remote command handler returns. According to an example implementation, a remote asynchronous command forwards a kernel event to a kernel queue specified by the issued command.

[0125] In modern hardware systems, certain components are physically embedded in computer nodes. For example, in particular, memory, processors, and I / O devices are typically embedded in computer nodes. Therefore, repairing and replacing these components typically involves removing the computer node from the system, even if most of the computer nodes continue to operate.

[0126] Therefore, a certain node logic module holds each of the non-node logic modules of the system. In this context, "holding" refers to the binding that represents that a logical device is implemented by a physical component attached to the node logic module. For example, a physical NIC found in a specific physical node can be configured to implement a logical network port module, and a physical node can implement a logical node module. In this example situation, a logical node holds a logical network port module. Then a virtual network device is bound to the logical network port module.

[0127] According to an example implementation, a computer node includes a logic module for managing physical processors in an SDS 400. For example, the logic module is implemented as a thread data structure in a host operating system such as FreeBSD. This allows, for example, standardized VCPUs to be presented to a guest OS 404. The physical processors of computer nodes across the SDS 400 may be heterogeneous, with different capabilities, not all of which are presented to the guest OS 404. The logic module corresponding to the standardized VCPU includes information defining which capabilities of the physical processor are provided / not provided. Thus, even if the hardware CPUs are different (where the virtualized CPUs are implemented on a logic module, which in turn is implemented on a physical resource), a standardized set of the same VCPUs can be presented to the guest OS 404.

[0128] As another example, memory pages in SDS 400 may also be associated with logical memory modules, where the logical module for a virtual memory page includes information about the virtual memory page, such as its GCPP entry, when the logical module is local on a node. When a virtual memory page is migrated, the corresponding logical module is also migrated. Treating each virtual memory page as a logical module may not involve additional overhead because the SDS 400 architecture uses the same data structure to handle page movement between nodes.

[0129] As another example, a physical network adapter may be associated with a virtual network adapter presented to the guest OS 404. For example, two physical network adapters may exist on different computer nodes of the SDS 400. In an example embodiment, in the hyperkernel, a logical network adapter (an example logical I / O adapter) implemented by two different physical network adapters may exist on each of the two computer nodes. Although the guest OS 404 knows the virtual network adapter (it thinks that the virtual network adapter is hardware), it does not know the logical network adapter. The logical network adapter includes information that enables two actual physical network adapters to work together to act as a single adapter. For example, this information includes information such as the location of the two physical network adapters, so that the two computer nodes (the physical network adapters are located on these two computer nodes) do not know each other. When the virtual network adapter moves, the logical element (logical network adapter) of the hyperkernel also moves in order to continue working. Therefore, in this example, the guest OS 404 running in the upper level of the hyperkernel does not know the logical module. The logic module includes information about the two physical network adapters (eg, location information) and makes decisions about which physical network adapter to use to implement requests to the virtual network adapter from the guest OS 404. That is, the internal structure of the logic module includes such information about how to apply instructions to different physical adapters.

[0130] Certain logic modules may be implemented in a distributed manner, spread across multiple nodes. Such logic modules, referred to herein as "distributed logic modules", are held by all nodes involved. As an example, certain storage volume logic modules, such as storage volume logic modules with multipath capabilities, are implemented in hardware or simulated in the dynamic reconfiguration framework / subsystem described herein.

[0131] The handling of modules held across multiple computer nodes generalizes the handling of modules held by one computer node. According to an example implementation, to remove a computer node, all modules held by the computer node are rendered inoperable. For such multi-node modules, this includes changing the distributed implementation of the module so that the removed computer node no longer holds the module. The reverse situation can also be performed, as an operational computer node can be added as a holder of a distributed logic module.

[0132] According to some embodiments, to remove a physical node, the logical node module and all logic modules it holds are removed. In one example embodiment, a logic module is removed based on meeting the following criteria: the logic module is no longer operational and is therefore not used by the hypercore. Another logic module takes over the functionality of the removed module.

[0133] The holding relationship is a difference between the node logic module and other types of logic modules, which also affects the order of the reconfiguration steps. For example, suppose there is a network port logic module New, which is available and meets the criteria for taking over from another network port logic module Old to be removed. Before removing Old in, for example, an operating or degraded state, the virtual logical binding of the virtual network port is changed from using Old to using New instead, and the state of Old is changed to available. This includes assigning the media access control (MAC) address used by Old to New, then waiting for all ongoing operations on Old to complete, while directing new operations to use New. When Old is no longer in operation, Old transitions to an available state or a disconnected state.

[0134] According to an example implementation, before a node logic module is removed from the SDS 400, the node logic module should no longer contain any guest state. Specifically, guest pages and VCPUs should not be active on the node logic module. The process of removing guest pages, VCPUs, and I / O devices is referred to herein as "evacuation." In the hyperkernel architecture described herein, evacuation includes the hyperkernel mobilizing VCPUs, pages, and I / O using a resource migration mechanism such as described above.

[0135] The problem of evacuation is to handle guest pages that are "wired" to the node logic module. In some existing systems, certain guest pages can be permanently mapped to, for example, user space pages for an unlimited period of time. As another example, certain pages can be wired into memory, where such pages will have to be moved from one memory module to another. According to an example embodiment, the hyperkernel moves all virtual pages between node logic modules (i.e., there are no "wired" pages) so that virtual pages can be moved at any time. In an example embodiment, the SDS 400 does not map guest pages to user space, and the I / O device emulation handles the moved pages by pausing when accessing non-local pages, and then requesting that the pages be moved to the node logic module where the emulation is running, or moving the I / O device emulation thread to the node logic module containing the virtual pages.

[0136] Therefore, the dynamic reconfiguration layer described herein is supported in part by guest pages that are not "wired" to a specific node logic module. According to an example embodiment, this allows it to be possible to evacuate pages from a node logic module at all times. It is worth noting that transferring data to or from a guest page does not "wire" the page. According to some embodiments, transfers (whether by VCPU or I / O emulation) are performed as unit operations that keep the guest page in memory only long enough to complete the transfer. According to an example embodiment, two mechanisms avoid many I / O-related data copies: replacing guest virtual pages with non-guest memory pages, and creating "copy-on-write" clones of guest virtual pages. Otherwise, I / O emulations break their transfers into chunks within pages and perform transfers to / from each virtual page, one at a time.

[0137] Whenever a VCPU is paused, the VCPU can be migrated to a different logical node. To evacuate any VCPU that happens to be on a node, a pause is called, and at that pause, migration of the VCPU takes the VCPU away from the node. Similarly, a virtual page can be removed to another node at any time by an operation that moves the owner of the page. For pages that are secondary copies (e.g., read-only pages), these pages can be deleted without transferring the pages anywhere.

[0138] According to an example embodiment, the evacuation of all virtual pages is performed by looping over all pages in a logical node and sending each guest virtual page to another operating logical node in SDS 400. Similarly, in an example embodiment, the evacuation of all VCPUs is performed by looping over all VCPUs in a logical node, triggering a stall that forces each VCPU to move to another operating logical node.

[0139] Two example problems to be addressed during an evacuation while the SDS 400 is operating are: resource tracking and migration blocking. With respect to the resource tracking problem, according to an example implementation, all virtual resources (e.g., VCPUs, virtual pages, and virtual I / O devices) are tracked by a distributed resource tracking algorithm that tracks the location where the resource was last sent on each logical node. In an example implementation, the tracking data includes a series of node pointers to follow in order to find resources that are not on the current logical node. When resources are moved during an evacuation, other logical nodes in the system can still operate under the assumption that the resources are located on the evacuated logical node. Therefore, the evacuated logical node can receive requests for these resources that have been moved. Typically, these messages will be forwarded to the logical node to which the resources have been moved, but this may not be possible once the evacuated logical node becomes inoperable.

[0140] Regarding the migration blocking problem, according to an example implementation, in order to complete or end the evacuation of a logical node, resources should not be moved to the logical node. Therefore, other logical nodes should be notified not to send resources to the node being evacuated.

[0141] The above two example problems are solved using the following example evacuation process performed in multiple steps. First, the evacuated logical node notifies all logical nodes that it is no longer the destination for VCPU, guest virtual page or virtual I / O device migration. The evacuated logical node can still receive requests for resources, and the local node (or the hypercore on the evacuated node) processes or forwards these resources as appropriate during the evacuation phase. Next, the evacuated logical node then begins to evacuate existing pages, VCPUs and virtual I / O devices.

[0142] Each evacuated resource generates a special location update message, which is broadcasted by the evacuated logical node to all other logical nodes, indicating that the resource has been moved from the evacuated logical node to the new logical node (i.e., a message informing other nodes of the updated location of the resource). According to an example embodiment, the evacuation location updates are bundled into messages on the interconnect (e.g., of medium size or any other size as appropriate), thereby reducing the overhead on the interconnect (e.g., as small as about 16 bytes / resource).

[0143] Upon receiving an evacuation location update for a resource R moved from the evacuated logical node E to logical node N, the logical node checks where the node thinks R is, and if it is E, the logical node accepts N as the new location for the resource R. When the evacuation of logical node E is complete, node E broadcasts an evacuation complete message (indicating completion of the resource evacuation) and waits for responses from all other nodes (acknowledging receipt of the evacuation complete message).

[0144] When a logical node receives an Evacuation Complete message from logical node E, the receiving logical node scans its own local table to verify that no resources are now considered to be on logical node E and responds with an Evacuation OK message if this is true. If this is not true, the receiving logical node sends one or more Evacuation Location Requests for the locations of all resources still considered to be on logical node E.

[0145] When the logical node E receives Evacuation OK messages from all logical nodes, the logical node E moves out of the operational state. While waiting for the Evacuation OK message, the logical node E processes the Evacuation Location Request(s) to which it responds with the Evacuation Location Update message.

[0146] I / O mobilization supports the ability to allow virtual I / O devices to be re-bound to different logical modules on different logical nodes. There are two bindings involved: virtual to logical and logical to physical. Virtual I / O devices are mobile in that they can be accessed from any VCPU and can access any virtual page without having to move the logical modules or physical devices used in their implementation. Thus, according to an example implementation, the evacuation process handles virtual I / O devices, whereas logical device binding is handled through a logical module binding action from a component of one physical node to a component of another physical node.

[0147] There are many kinds of storage devices. Locally attached drives, removable drives, network attached storage, and fabric attached storage are just some examples of the various options that can be used. In SDS 400, such storage devices are represented by a single type of (possibly distributed) logical module, a storage volume logical module (also referred to herein as a "logical storage volume"). In particular, according to an example embodiment, the hyperkernel virtualizes block devices (e.g., whether using Virtio or emulating a peripheral component interconnect express (PCIe) hardware interface) using a public internal storage volume API that provides, for example, queued asynchronous I / O requests within a logical node (and can be optimized, for example, for handling virtual 4K page sizes and page-aligned blocks). It is worth noting that some operating systems such as Linux, FreeBSD, and Windows are designed to try to use page granularity on their disk file systems as much as possible and to maintain page-aligned storage caches as much as possible. Although these operating systems can support different disk block sizes in their file system code, they typically use their virtual memory layer to manage their disk caches, so their unified disks only cache cache page-sized, page-aligned blocks.

[0148] According to an example embodiment, a higher-level hyperkernel uses a public storage volume interface to emulate a block device. For example, according to an example embodiment, the hyperkernel implements the emulation of the Virtio block advanced host controller interface (AHCI) / AT attachment package interface (ATAPI) and NVME high-speed (NVMe) virtual device. It is worth noting that although NVMe is defined for hardware implementation, it has benefits as a guest interface for storage virtualization compared to the Virtio block interface because it uses a natural 4K page size transmission and supports a more efficient queuing protocol. In some embodiments, such emulation (e.g., running on a virtual I / O processor, also referred to as "VIOP" herein) obtains a command chain containing descriptors from a queue in the guest virtual memory, and converts these commands into block-level commands to the logical storage volume. The virtual block device emulation code does not need to know how the "driver" that the storage volume is effective implements the device, and the storage volume driver does not need to know the guest virtual memory, interrupt signaling, etc. This separation of concerns provides reconfiguration flexibility.

[0149] The logical storage volume may be distributed among multiple logical nodes (i.e., distributed logical modules), and therefore, any or all logical nodes may be involved in a particular block-level command. For example, a logical storage volume may be implemented using multiple disk drives across multiple nodes, where the drives are mirrored using a redundant array of independent disks (RAID) or are striped to allow parallel transfers to / from different logical nodes. According to an example implementation, the logical storage volume driver code is configured to route the block-level command to the right disk drive(s) on the correct node.

[0150] According to an example implementation, all virtual storage volumes may be configured to have a 4K block size. Zero copy of virtual logical data transfer may be performed through mapping. Logical volumes are independent of physical volumes, wherein RAID / multipath redundancy is managed by the hyper-kernel.

[0151] As a second example, a fabric-attached logical storage volume supporting multipath access may be connected to multiple logical nodes, again presented to the hyperkernel as a distributed logical storage volume across some or all of these logical nodes (or just to a single logical point in a logical node). In this example scenario, block-level commands issued to the distributed logical storage volume may be directed to any of the elements of the distributed storage volume.

[0152] Reconfiguration can temporarily place the logical storage volume in a distributed state as the path to the content and the content itself are transferred between logical nodes.

[0153] According to an example implementation, the interface of the logical storage volume handles page-sized blocks. To write to the logical storage volume, one or more blocks are provided to the logical storage volume, the transfer is completed, and the blocks are released back to the system.

[0154] To read from a logical storage volume, one or more blocks are provided to the logical storage volume, the transfer is completed, and the block is released back to the system if the content is updated. The completion of the transfer is signaled on a completion queue along with a status block indicating the success / failure of the transfer. In an example embodiment, the completion queue uses a kernel event mechanism (kernel queue) in the operating system FreeBSD. According to an example embodiment, a flush command is provided that allows for forcing one or more blocks into stable storage. The same interface can be used even when the transfer is smaller than a block, where, for example, only a portion of a page-sized block is read or written in a physical device.

[0155] Using page-sized or page-aligned blocks makes the most common case of I / O involve only rebinding the current instance of the virtual memory page. This is more efficient than copying the data multiple times. In most cases, I / O can be performed with zero memory-to-memory copies. Reading data from the drive into a block creates a new memory page on the node containing the device. Once the read is complete, the memory page is replaced in the guest virtual memory with the page that is the target of the guest's I / O. During the I / O, the old page still occupies space in memory. However, once the I / O operation begins, the page will eventually be erased. Therefore, once the I / O operation is queued, the hypercore can erase the page. If the target of the I / O is actually on another node, this is not necessarily the case. While the page read is in progress, the target page can be erased and the dormant instance moved to the node performing the I / O. According to an example embodiment, pages in the system that have not yet been zeroed are indicated as "dormant" pages. According to an example embodiment, these dormant pages are zeroed on an ongoing basis or until they are used for the first time (e.g., allocated and written or created). When the page read is completed, the new content is in the page that can be replaced with the dormant page. If the dormant page moves off-node during this time (which is probably unlikely since it is not normally accessed), the new contents of the page can be sent to wherever the page moved to.

[0156] Writing data from a block to a drive is similarly optimizable. In an example embodiment, the page to be written should be on the node of the logical storage volume driver (if distributed, the page can be moved to any node that can perform the write most efficiently). If it is not there yet, a copy can be used. A page may be designated as a PRIMARY page, and moved to the device as a SECONDARY copy of the page, leaving the PRIMARY page where it is (where PRIMARY and SECONDARY are examples of states that a page or page copy may be in). This forms a "copy on write" situation for guest pages during the time when a write to a block is in progress. A virtual page instance used by a logical storage volume can be moved or deleted, but the instance will retain its contents and will not be released until the write operation is completed. In an example embodiment, this feature is implemented by adding a flag to each page in memory that marks the page as in use by a logical storage volume write.

[0157] Logical storage volumes contain a large amount of guest data that must be retained and will continue to be accessed during dynamic reconfiguration. Therefore, when a logical node is removed, the data in any logical storage volumes held by the node should be retained. Different retention processes can be used depending on the way the logical storage volumes are implemented in the physical media. However, in all cases, the same command to the logical storage volume can be used to complete the retention, wherein this command to the logical storage volume is referred to as logical storage volume relocation in this article.

[0158] According to an example implementation, a logical storage volume is relocated from a set of nodes containing devices to a new set of nodes containing the same or other devices. Commands issued to the logical storage volume (which knows how the command is currently implemented and on which nodes the command is currently implemented) specify a target set of nodes to be used (which may overlap with the current set of nodes used) and any new devices that may be used on those nodes. The logical storage volume driver is configured to perform steps to implement the relocation. During the relocation, the logical storage volume is expanded to include the source set of nodes and the target set of nodes, and then shrunk or contracted to include only the target set of nodes. This expansion / contraction method allows the API of the logical storage volume to hide information about the internal implementation of the logical storage volume driver from the rest of the hyperkernel.

[0159] For fabric-attached storage with multipathing capabilities, relocation consists of updating the set of nodes from which the remote storage device can be accessed and initializing any new connections in the relocation target set, while closing any connections on nodes that are dropped from the relocation set. Since the content is remote, there is no content transfer.

[0160] As another more complex example, a single volume involving one or more locally attached drives on a single computer node can be relocated. An example method involves transferring content by moving the physical drive itself from a slot on one computer node to a slot on another computer node. In this method, there can be a short period of time (which can be on the order of a few seconds) during which the guest will not be able to access the drive. Further, a human operator may be required to manually move the physical drive, which, if not done correctly, may cause the system to crash. Other methods for improving system availability are described below.

[0161] As another approach, assume that there is a spare, empty locally attached drive, or a remote fabric-attached backup storage device, on another computer node (e.g., on a replacement node that takes over from the failed node). The relocation can be performed by copying the contents from one volume to the other. Assuming that the removed computer node continues to operate fully, such a migration can be performed, but may take time. The logical volumes are temporarily distributed across the two logical nodes, and commands to be redirected to the devices are managed internally, including the current copies of the blocks as they are copied.

[0162] As another example, a more complex form of local storage can avoid having to keep a degraded node running for a long time when its logical storage volume is copied away. In an example embodiment, this example form involves a RAID style distribution using a subdivision or part of a RAID volume across multiple computer nodes. For example, a logical storage volume can be implemented as a RAID5 array across four computer nodes, each of which contains a driver. Each pair of logical blocks will be represented as three physical blocks on three consecutive drivers, wherein two of these physical blocks are the contents of the logical blocks, and one of these physical blocks is a parity block, which contains the bitwise exclusive or of the contents of the logical blocks. If a failure is detected in one of the four drivers, the logical storage volume is transformed into a degraded state. However, the physical contents of the drive can be rebuilt by a bit combination of the physical blocks from the other three drives. Therefore, a single drive failure can be handled by adding a new empty drive to the RAID distributed volume on any node--this is an example of a relocation operation. However, if one of the nodes is degraded in some other way, the drive on the node can be rebuilt in the same way without the need to degrade the drive on the node at all.

[0163] Therefore, one beneficial result of using a local distributed logical storage volume with a RAID5 style structure is that a node can be replaced quickly, so that the reconstruction of the logical storage volume content on the node can be carried out leisurely after the reconfiguration is completed.

[0164] One possible cost of using a RAID5-style distributed logical storage volume across multiple computer nodes is that writes to the logical storage volume may be slower because any write involves updating both the logical block and the parity block. Instead of writing once, this requires an additional write, and if the parity block is not in memory, a read of another logical block covered by the parity block may be required. By distributing physical devices across computer nodes, the data of at least one block may also have to be transferred between nodes. However, the RAID structure also has performance advantages, where, for example, the RAID structure allows data to be loaded into memory at high speed due to doubling the bandwidth from physical storage to memory.

[0165] Block I / O is accomplished through a public API that communicates with the logical storage volume abstraction implemented by the logical storage volume driver within the hyperkernel. According to an example embodiment, the logical storage volume driver is implemented as a distributed logic module on multiple nodes. The distributed logic module can be relocated so that one or more nodes no longer hold the logical storage volume. Such relocation is a way to replace the node holding the logical storage volume.

[0166] The logical storage volumes are designed in a way that, for example, hides many details of the common storage system from the guest virtual machines supported by the hyperkernel distributed virtual machine manager, which is instead presented and sees simple block drivers. According to example embodiments, the details of the storage devices, connections, etc. can be reflected to the guest OS 404, which can then handle these details itself, even through the mapping provided across nodes by devirtualization. However, it may be more beneficial to apply the modularity achieved by devirtualization to hide the complexity from the guest OS 404 while allowing better performance and higher resiliency.

[0167] The logical storage volume abstraction described above creates a strict separation between how the virtual block device gets its instructions from the guest OS 404 and how those instructions are mapped to commands issued on the physical device.

[0168] According to an example embodiment, another category of I / O devices (in addition to the block devices described above) is a network interface. Compared to logical storage volumes, network interfaces do not require management of "content". Replacing one physical NIC port with another can be performed. For example, a MAC address, an aspect of a NIC port that uniquely identifies a NIC port, can be reassigned quickly (e.g., within a few milliseconds).

[0169] SDS 400 splits the emulation of guest network interface ports (e.g., implemented using Virtio networking or any other standard as appropriate) into two parts - the guest-facing top half and the network port logic module (also referred to as a "logical network port") that handles all issues dealing with the physical hardware.

[0170] Compared to block devices, where guest operating systems typically perform I / O using page-granular, page-aligned buffers, existing operating system networks are typically poorly organized for zero-copy transfers organized around page-sized units. As an example, the structure of an I / O operation to a network port is as follows. Commands are issued synchronously or asynchronously to the network port logic module to read a data packet or write a data packet to a physical device currently used by the network port to implement the logical network port. According to an example implementation, when a data packet is sent, the data packet is copied from the guest virtual memory to a list of real local memory blocks by the hyper-kernel, and this copy is provided to the network port logic module. In some cases, the copy will arrive at an mbuf (an example of a memory buffer) on the node where the page resides, where the mbuf is provided to the logic module. This simplifies sending from / through a local device or remote device because mbufs are used in the hyper-kernel of both the local device and the remote device.

[0171] The network port logic module can transfer a copy across the dedicated network interconnect from which the copy is made to the node of the physical device. This copy can be smaller than a page and avoids the need to copy a large amount of other data in the page to the device, thereby reducing interconnect traffic. After completion, the status is placed in the completion queue.

[0172] According to an example embodiment, the received data packets are placed in local memory buffers, which are returned in the completion queue using, for example, the command / completion queue interface described above. According to an example embodiment, the memory buffers are provided to the device emulation code, which copies these memory buffers to the guest virtual memory. Again, according to an example embodiment, mbuf chains are used as the data form returned before completing the reception of the data packets. The mbuf chains can then be copied into the virtual page one block at a time, and if the page is remote, the mbuf chains can be transmitted via another interconnect, which is configured to process the mbuf as part of its data packet.

[0173] According to example implementation, the initialization of the computer node places the corresponding logical node module in an available state. The shutdown of the computer node can be performed after the corresponding logical node module is in an available or disconnected operating state. The details of the removal of the node have been described herein, which places the logical node in an available or disconnected operating state. In the available and disconnected states, the hypercore and the logic module are still in play, and their data structures of tracking logic modules and memory states are initialized, etc. These states can have similarities. An example of the difference between available and disconnected is that the computer node belonging to the disconnected state undergoes repair before the computer node is added back to the SDS 400, and the computer node belonging to the available state is ready to use.

[0174] According to an example implementation, a computer node subject to the available state is powered on and ready to be added to the SDS 400. According to an example implementation, initialization of the computer node occurs before the computer node subject to the available state.

[0175] According to an example embodiment, shutdown of a computer node may include various steps, where, for example, certain states may need to be saved before the node is powered off (i.e., state preservation may be performed during shutdown). According to an example embodiment, in order to implement the dynamic reconfiguration framework described herein, initialization of the hyperkernel is structured as two steps and a third step of starting the SDS 400. According to an example embodiment, each logical node is initialized individually, thereby bringing the node into a usable operational state. According to an example embodiment, this initialization does not involve interaction between logical nodes, as node initialization can occur in parallel. In an example embodiment, a recipe or process for initializing each logical node is transmitted to the logical node in text form.

[0176] Initialization of the hyperkernel may then include applying SDS-specific parameters to one or more logical nodes, thereby placing these logical nodes in an operational state. According to some embodiments, this includes two types of descriptions: SDS-wide parameters (e.g., guest address space size, number of VCPUs, etc.), and definitions of logical modules to be initially stored on each logical node.

[0177] Initialization of the hyperkernel may then include selecting a single logical node as a "boot node" and providing boot information to start the SDS 400. According to an example embodiment, all VCPUs and non-dormant memory are initialized on the boot node. However, after such initialization, there is no distinction or difference between the boot node and other logical nodes. For example, the boot node may not even be logical node 0. One characteristic of the boot node is that the VCPUs are initially found on that node.

[0178] According to example embodiments, any computer node may be removed (eg, to reduce required resources or to perform replacement due to a failure). This includes computer nodes associated with a boot node, which, as described herein, no longer differs from other logical nodes after initialization.

[0179] According to an example implementation, a management node (such as Figure 4 The management node 499 of the SDS 400 manages the SDS 400. The management node may issue service commands related to node actions during startup, reconfiguration, removal, etc. According to some embodiments, the management node may not be used to maintain the operation of the SDS 400, thereby preventing the management node from becoming a single point of failure that will cause the SDS 400 to fail. The management node may provide monitoring / management capabilities to manage the physical components in the SDS 400. For example, the management node may perform external monitoring of the health of the physical components and allow manual or automatic failure response. According to an example embodiment, the nodes of the SDS 400 maintain the configuration and health information used by the management node in the event of a management service failure. In an example embodiment, the SDS 400 provides ports (which may also be replicated) on each node, which can be used by any (authorized) node that provides a management server to manage each node. These ports provide paths for recording data about operations, collecting system health information or other information. According to an example embodiment, these ports also support protocols that allow querying the current system configuration status. Rather than maintaining the system configuration in an external structure that may be inconsistent or unreliable, a dynamic reconfiguration layer on each node provides introspection capabilities so that a new management can discover the information needed to manage the SDS 400 if it is the first management node.

[0180] The functionality of the introspection interface provides the ability to inventory all logical modules currently operating in the SDS 400, all nodes (whether operating or not), and the physical devices in each node and their use by the logical modules. This is also the coordination point with the management console. In an example embodiment, the management console for the SDS 400 is a program that runs outside the hyperkernel to control dynamic configuration changes by using the introspection interface. According to an example embodiment, the management console is implemented in a separate computer connected to the SDS 400 or on one of the hardware nodes of the system as a process running simultaneously with the hypervisor. If the management console is implemented on one of the nodes of the SDS, it is stopped and restarted on another computer in the event that the node is to be removed from the server.

[0181] According to an example embodiment, the time base logic module (also referred to herein as the "time base module") is responsible for ensuring invariants related to guest virtual time on all nodes of SDS400. The design invariants of the time sources and time event sources in the hyperkernel are twofold: from the perspective of the guest operating system, virtual time does not "go back"; and virtual time maintains a constant rate relative to the external clock time, as seen by all virtual time sources and time event sources. The invariant rules about "going back" can be complex because there is a causal order between the time read from the virtual clock device and the guest virtual memory content. For example, if two VCPUs read the same clock, the ordering of memory operations they perform before and after the clock read should conform to the causal serialization of memory accesses performed by the VCPUs. In many guest synchronization techniques that use, for example, the timestamp counter (TSC) of the Intel instruction set and memory access ordering constraints, the causal order is considered correct.

[0182] According to an example embodiment, these invariants are implemented in the SDS 400 by using a single distributed very high resolution clock that is effectively synchronized between each pair of logical nodes (e.g., with a difference of less than a microsecond between each pair of nodes) and a rule that effectively delays each guest-level transmission information between nodes so that such a transmission never arrives at a local time earlier than the time at which the guest-level transmission information was sent. If the interconnect has a transmission delay that is greater than the difference between the logical node pairs, then no delay may be needed at all, unless time synchronization allows for time deviations. This can be achieved by controlling the effective rate of the virtual clock so that the virtual clock of each node has the same rate of about, for example, a few parts per million.

[0183] With respect to time-based logic modules and dynamic reconfiguration, the maintenance of synchronization and rate is also taken into account during the addition or removal of logical nodes. For example, when a new logical node becomes operational, the new logical node is synchronized to the existing logical nodes.

[0184] According to an example embodiment, the interconnections between the computer nodes of SDS 400 are made resilient by adding redundancy. According to an example embodiment, the dynamic reconfiguration framework achieves this by using multiple ports and multiple switches in each computer node, so that the failure of a switch or cable only degrades performance, but there are alternative (redundant) transmission paths.

[0185] In an example implementation, the hyper-kernel employs ordered messaging, and the interconnect protocol manages the selection of paths to specific node destinations within the bus port logic module, which corresponds to, for example, a network kernel implementation of the hyper-kernel. According to an example implementation, the selection of paths and the detection of failed paths are performed in the network kernel driver. Thus, multiple network ports on a computer node are represented by a single bus port logic module at the reconfiguration framework API.

[0186] Future growth of SDS 400 (such as expansion of a thinly provisioned SDS 400) may involve adding more computer nodes than the initial portion of SDS 400 at startup. This means that the set of logical nodes actually in use may change over time. In an example embodiment, in order to handle multiple switches, multiple ports, etc., the interconnection protocol is configured so that it separates logical node identifiers from physical layer constraints. When broadcasting a hyper-kernel message, inoperable logical nodes do not need to receive or process such a hyper-kernel broadcast, because if these logical nodes do so, it may cause failure. According to an example embodiment, a hyper-kernel broadcast is implemented so that there is a "multicast group" maintained for all operational logical nodes, and the hyper-kernel broadcast operation uses this multicast group instead of a total broadcast (which may also result in a broadcast to an inoperable logical node).

[0187] According to an example embodiment, unicast messages can be sent to logical nodes, whether operational or not. This capability can be used for messages used to manage reconfiguration of SDS 400 (e.g., messages transmitted to add another computer node and corresponding logical node to SDS 400). According to an example embodiment, reconfiguration also uses broadcasts to all nodes on a segment - this can be used for management functions to determine the status of all nodes, not just operational nodes, while the system is running.

[0188] According to an example implementation, the dynamic reconfiguration framework of the SDS 400 divides the hyperkernel into two layers, one layer handles the implementation of a single scalable virtual hardware system (e.g., a single scalable virtual hardware system), and a lower layer provides an abstraction for logic modules that implement all functions for allowing dynamic reconfiguration while the SDS 400 continues to operate. The logic modules can encapsulate fault tolerance mechanisms by supporting a particular logic module to operate in a "degraded" state when a hardware component associated with the logic module fails.

[0189] According to an example embodiment, an external management system (e.g., implemented in a console of a management node) is configured to detect / monitor signals indicating hardware component failures. These signals may be provided by various per-node hypercores. Each hypercore may listen for fault identification signals, e.g., by polling physical components and / or obtaining reported values ​​from physical components.

[0190] The hyper-kernel can detect faults in components by accessing status information collected by the server motherboard. For example, the server motherboard can collect and / or report the performance of various hardware components (e.g., memory chips, processors, fans, block devices, network devices, etc.) attached to the motherboard. This can include, for example, telemetry data collected in memory registers, such as temperature, voltage, fan speed, detected and corrected errors, etc., which are not seen by the guest OS 404 but can be accessed by the hyper-kernel. According to an example embodiment, each node hyper-kernel is configured to poll the motherboard of the node to obtain such telemetry information. In other embodiments, the motherboard reports such telemetry information to the hyper-kernel. The hyper-kernel can then pass the telemetry information to an external management system (wherein the hyper-kernel can also perform processing (e.g., summarizing, such as determining event counts, event rates, etc.) on the telemetry information before the telemetry information is passed to the external management system).

[0191] The hardware may periodically (e.g., based on a timer interrupt) report the state of the hardware, which may be read by the hyper-core. For example, various registers may exist throughout the system, such as on a motherboard and a processor, indicating what faults have occurred. A fault may be one that does not change the behavior of the processor. As an example, consider a detected memory fault. A detected memory fault may be recorded and stored in a register of the motherboard. In existing systems, such telemetry information collected on the motherboard is typically not reported anywhere outside of the basic registers that capture the telemetry information. In an example embodiment, the hyper-core utilizes such information to identify failed / degraded hardware devices / components.

[0192] In an example embodiment, the hyper-kernel observes the count of correctable memory errors. For example, the memory bank can use this technique to determine the correctable error rate on any time scale as appropriate (e.g., correctable errors per minute, per hour, etc.). In a typical computing system, the correctable error rate detected can be approximately zero per hour or zero per week. However, as the system ages, the correctable error rate may increase to, for example, one or two correctable errors per month. Although the memory may not be completely unusable (i.e., the memory is in a degraded state) and does not need to be completely removed, this information can be used by the hyper-kernel to initiate remedial actions to address the degraded component before it fails.

[0193] In general, according to various embodiments, for any of a number of different reconfiguration operations, a running SDS 400 may be dynamically reconfigured based on a ranking based on computer node lifespans. For example, the dynamic reconfiguration may be part of a preventive maintenance procedure, wherein one or more active computer nodes of the SDS 400 (e.g., active computer nodes with the smallest expected remaining lifespan(s), as indicated by the rankings) are selected or identified based on the rankings and these active computer nodes are removed from the SDS 400 for performing maintenance (e.g., repairing and performing diagnostic tests) on the removed computer nodes. Additionally, as another part of this dynamic reconfiguration, one or more spare computer nodes (e.g., spare computer nodes with the largest expected remaining lifespan(s), as indicated by the rankings) are selected or identified from a pool of spare computer nodes and these spare computer nodes are added to the SDS 400. Adding and removing computer nodes to and from the SDS 400 occurs without interrupting the guest operating system or applications executing the SDS 400.

[0194] Typically, a preventive maintenance program can be established to improve system reliability. As an example of a preventive maintenance program, the replacement of computer nodes can be scheduled on a time-driven basis (e.g., annually or semi-annually). This ability to reconfigure the SDS 400 while the SDS 400 continues to operate and provide this high availability without any downtime is an improvement over existing data centers, compared to, for example, shutting down operating systems and applications running on distributed systems for hours or even days and accounting for the additional time to perform maintenance and restart the system.

[0195] As another example of dynamic reconfiguration, a thinly provisioned SDS 400 can be dynamically reconfigured to add one or more computer nodes to a server based on a ranking (e.g., adding (multiple) spare computer nodes having (multiple) maximum expected remaining lifespans according to the rankings to a server) without interrupting the guest operating system or the application executing the SDS 400. "Thin provisioning" refers to the ability to dynamically and transparently adjust the size of a computing system (such as an SDS 400) depending on the workload required by the system. According to example embodiments described herein, a thinly provisioned system is a system that has an apparent physical resource allocation that is less than the actual physical resource allocation of the system. In other words, the physical resources of a thinly provisioned computer system are oversubscribed.

[0196] Computers include resources such as processors, memory, and I / O functions such as disk drives and network connections. Applications run on these computers and utilize these resources. In some cases, applications may need or attempt to use more processors or memory than are present in the computer. An example of this is that an application addresses more random access memory (RAM) than is present in the computer. At this point, memory pages that are not actively used are moved from RAM to a storage device (e.g., a hard disk drive HDD or a solid-state drive SSD) and saved to make room for other memory pages that have been saved in the storage medium to be retrieved and entered into RAM. This "paging" deceives the application into believing that it has more RAM than is physically present. There are similar mechanisms that allow processors and I / O functions to be virtualized and therefore may be oversubscribed on a computer (i.e., available hardware). As an example, a virtual processor may be saved and therefore idle, while another processor is (re)started using an available physical processor. Operating a computer system in this oversubscribed state involves executing instructions to save a processor state, thereby saving the state in memory, and then loading a new processor state, which will then be restored.

[0197] As an example, a thinly provisioned software-defined server can be a secondary or backup server for a pair of servers (e.g., a high availability server pair or a disaster recovery server pair), where the other server of the pair is a primary server. The primary server can serve as a primary role (e.g., a database server) and is fully provisioned for this purpose. A secondary server can serve as a secondary role that does not require the actual physical resources required by the primary role. For example, the primary server may include application logic that processes transactional data with a client and submits the data to permanent storage. The secondary server may include logic that synchronizes the server with data submitted to permanent storage. The secondary server may have the same software (e.g., applications and operating system) as the primary server, so that if the primary server fails, the secondary server can take over the primary role and take over the additional functions performed by the now failed primary server (e.g., application logic that processes transactional data).

[0198] The software installed on the auxiliary server may expect the same physical resource allocation as that of the primary server. However, allocating this level of physical resources to the standby auxiliary server may be quite expensive because the physical resources are not fully utilized. Another approach may be to thinly provision the auxiliary server and in response to the primary server failing, power off the auxiliary server, add additional resources and then power on the auxiliary server to transfer to the primary role. However, this approach may consume considerable valuable time, which may be quite expensive for the client.

[0199] Another method available when the secondary server is a thinly provisioned SDS 400 is to thinly provision the secondary SDS 400. Here, thinly provisioning the SDS 400 means that the full set of computer nodes of the SDS 400 provides an actual physical resource allocation that is less than the actual physical allocation required when the SDS 400 plays the primary role. Due to thin provisioning, the secondary SDS 400 utilizes fewer physical resources (compared to the secondary server with full physical resource allocation). A spare, fully powered-on computer node pool can be used to allow computer nodes to be added to the secondary SDS 400. Therefore, in response to a failure of the primary server, the thinly provisioned secondary SDS 400 can be quickly thickened (i.e., physical resource allocation can be increased) by adding one or more spare computer nodes to the server, all without interrupting the server's operating system or executing applications. According to an example embodiment, the (multiple) spare nodes to be added to the secondary SDS 400 are identified based on a ranking (e.g., the (multiple) spare nodes with the (multiple) maximum expected remaining life are selected to be added).

[0200] Thickening auxiliary servers is an example of expanding the number of computer nodes of SDS 400. According to example embodiments, SDS 400 can be dynamically reconfigured to expand servers for other reasons. For example, due to the rate at which digital data continues to grow, data centers must handle increasingly large amounts of data. However, it is often difficult for customers to determine the appropriate size of their data centers when specifying a data center. It is also difficult for application developers to understand the size of the computing environment required for their applications (which size may change over time). In existing data systems, application developers may over-specify the amount of resources (e.g., processor cores and / or memory) required to run an application, where data centers can make their own determinations about the amount of resources required to try to keep resource utilization high and control costs.

[0201] Rather than depending on a specific size of computer resources, the dynamically scalable SDS 400 can be expanded or reduced over time, depending on the workload of the server. The benefit of this dynamic scalability is that when developing applications, there are no longer constraints based on the available size of the computer. Compared with existing data centers, when running workloads, it is no longer necessary to make significant sizing efforts to determine the appropriate size of the computer purchased or deployed to run the application. Alternatively, using the techniques described herein, the SDS 400 can determine its own size based on the workload / application. If more physical resources are needed, computer nodes can be transparently added from a spare pool of, for example, unused computer nodes. The process of adding (multiple) spare nodes may include identifying (multiple) spare computer nodes to be added based on an expected remaining life ranking, as described herein (e.g., selecting (multiple) spare computer nodes with (multiple) maximum expected remaining lifespans to add). When the resources of the computer are no longer used or needed, one or more active computer nodes of the SDS 400 can be transparently withdrawn and returned to the pool. The process of removing (multiple) computer nodes from the SDS 400 can include identifying (multiple) active computer nodes based on the expected remaining life ranking (e.g., selecting (multiple) active computer nodes with (multiple) minimum remaining expected remaining life for removal). In fact, the user who writes or runs the application / job believes that the amount of resources available does not need to match the actual amount of physical resources of the SDS 400. When resources are needed, the SDS 400 can scale / grow so that it continues to run in an optimal manner. Such adjustments can be performed dynamically at a fast response rate and / or over an extended period of time as the size of the data set / application grows. Further, the dynamic ability to re-determine the size can have additional benefits, such as allowing resources from relatively low priority jobs or applications to be dynamically and quickly transferred to relatively high priority jobs / applications. For this purpose, applications or jobs can be assigned priority labels such as "mission critical", "best effort" or "when available".

[0202] Dynamic expansion and contraction of resources for SDS 400 may be requested manually (eg, by an operator of the system) or automatically (eg, based on automatic monitoring of the load on servers in the system).

[0203] refer to Figure 7According to an example embodiment, the non-transitory machine-readable storage medium 700 includes instructions 710. The instructions 710, when executed by a system, cause the system to receive information related to wear of a computer node. According to an example embodiment, the system may be a distributed system, such as a virtualized system or a non-virtualized system. According to an example embodiment, the virtualized system may be a software-defined server. According to an example embodiment, the information related to wear may be a wear indicator associated with a physical processor of the computer node. As an example, the wear indicator may be a current time rate at which the processor has been operating at a reduced frequency. As another example, the wear indicator may be the total number of times the processor has been operating at a reduced frequency. As another example, the wear indicator may be associated with a physical memory. For example, the wear indicator may be an error correction mode applied by the computer node. As an example, other wear indicators may be wear indicators related to I / O devices, network devices, network cables, and motherboards of the computer node. The instructions 710, when executed by the system, cause the system to rank the computer nodes according to their respective expected remaining lifespans based on the information related to wear. Instructions 710, when executed by the system, cause the system to reconfigure the system based on the ranking in response to an event corresponding to at least one of a spare computer node being added to the system or a given computer node being removed from the system.

[0204] Instructions 710, when executed by the system, cause the system to rank the computer nodes according to their expected remaining lifespans based on information related to wear. According to some embodiments, the ranking may be longest life first out (MLFO). Instructions 710, when executed by the system, cause the system to reconfigure the system based on the ranking in response to an event corresponding to at least one of a spare computer node being added to the system or a given computer node being removed from the system.

[0205] The reconfiguration may include identifying a particular computer node based on the ranking. In addition, the identification may include applying an additional selection criterion (e.g., a random selection criterion, a pseudo-random selection criterion, or a node ID selection criterion). For example, the identification may include identifying a list of a predetermined number of computer nodes based on the ranking and then randomly or pseudo-randomly selecting one or more computer nodes from the list.

[0206] According to some embodiments, the event corresponds to scaling up the distributed system, and a spare computer node identified as having the greatest expected remaining lifetime according to the ranking is selected to be added to the system. According to an example embodiment, the event corresponds to scaling down the computer system, and a given computer node is identified or selected as having the least expected remaining lifetime according to the ranking.

[0207] refer to Figure 8According to an example embodiment, a distributed system 800 includes a plurality of computer nodes 810. According to an example embodiment, the distributed system 800 may be a software defined server, a virtualized system including virtual machines on respective computer nodes, or a non-virtualized system. The computer node 810 includes a computer node wear leveling instruction 820.

[0208] According to an example embodiment, the information related to wear can be a wear indicator associated with a physical processor of a computer node. As an example, the wear indicator can be the current time rate that the processor has been operating at a reduced frequency. As another example, the wear indicator can be the total number of times the processor has been operating at a reduced frequency. As another example, the wear indicator can be associated with physical memory. For example, the wear indicator can be an error correction mode applied by the computer node. As an example, other wear indicators can be wear indicators related to I / O devices, network devices, network cables, and motherboards of the computer node. Instructions 820 can be executed in computer node 810 to receive an indication of an event corresponding to a change in the number of computer nodes.

[0209] Instructions 820 are executable to identify, in response to the event, a particular computer node associated with implementing the change from a pool of candidate computer nodes based on respective wear levels associated with the candidate computer nodes. The identifying may include identifying the particular computer node based on a ranking. In addition, the identifying may include applying an additional selection criterion (e.g., a random selection criterion, a pseudo-random selection criterion, or a node ID selection criterion). For example, the identifying may include identifying a list of a predetermined number of computer nodes based on the ranking and then randomly or pseudo-randomly selecting one or more computer nodes from the list.

[0210] According to an example implementation, the event corresponds to a reduction in the computer system, and a given computer node is identified or selected as having a minimum expected remaining lifetime based on the ranking.

[0211] refer to Fig. 9 According to an example embodiment, a technique for a distributed system including a plurality of computer nodes includes receiving (block 904) information related to wear of the computer nodes by a node wear leveling engine. According to an example embodiment, the distributed system may be a software defined server, a virtualized system including virtual machines on respective computer nodes, or a non-virtualized system. The technique includes ranking, by the node wear leveling engine, the computer nodes according to respective expected remaining lifespans based on the information related to wear (block 908).

[0212] According to an example embodiment, the information related to wear can be a wear indicator associated with a physical processor of a computer node. As an example, the wear indicator can be the current time rate that the processor has been operating at a reduced frequency. As another example, the wear indicator can be the total number of times the processor has been operating at a reduced frequency. As another example, the wear indicator can be associated with physical memory. For example, the wear indicator can be an error correction mode applied by the computer node. As an example, other wear indicators can be wear indicators associated with I / O devices, network devices, network cables, and motherboards of the computer node.

[0213] According to block 912, technique 900 includes identifying, by a wear leveling engine, in response to an event corresponding to a change in the number of computer nodes, a particular computer node associated with implementing the change from a pool of candidate computer nodes based on a ranking. According to an example embodiment, the event corresponds to a reduction in the computer system, and a given computer node is identified or selected as having a minimum expected remaining life based on the ranking. Identifying the computer node may include applying additional selection criteria (e.g., random or pseudo-random selection). For example, the identifying may include identifying a list of a predetermined number of computer nodes (including the particular computer node) based on the ranking and then randomly or pseudo-randomly selecting one or more computer nodes from the list.

[0214] According to an example embodiment, the event corresponds to a maintenance service in which a given computer node is removed from the system for maintenance. The reconfiguration includes removing the given computer node from the system. The instructions further cause the system to select the given computer node to be removed from the system in response to determining that the respective expected remaining lifetime of the given computer node is less than at least one other respective expected remaining lifetime. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0215] According to an example implementation, the instructions further cause the system to estimate the expected remaining life using the model and then update the model in response to the expected remaining life determined from a maintenance check of the given computer node after the given computer node is removed. Among potential advantages, the predictive accuracy of the model used to estimate the remaining node life can be improved.

[0216] According to an example implementation, the event corresponds to a maintenance service, wherein a given computer node is removed from the system for maintenance, and the reconfiguration includes adding a spare computer node to the system. The instructions further cause the system to select the spare computer node in response to determining that a respective remaining lifetime of the spare computer node is greater than at least one other respective expected remaining lifetime of another spare computer node. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0217] According to an example implementation, the system includes a distributed system, and the instructions, when executed, further cause the system to evaluate an application workload provided by a computer node; determine that resources provided by the computer node to support the application workload are undersubscribed. The instructions, when executed, further cause the system to initiate an event to augment resources in response to the determination; and associated with augmentation, select the spare computer node in response to determining that a corresponding remaining life of the spare computer node is greater than at least one other corresponding remaining life of another spare computer node. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0218] According to an example implementation, the system includes a distributed system, and the instructions further cause the system to evaluate an application workload provided by a computer node; determine that resources provided by the computer node to support the application workload are oversubscribed. The instructions, when executed, further cause the system to initiate an event to scale back resources in response to the determination; and associated with the scale back, select the given computer node to be removed from the system in response to determining that a corresponding expected remaining lifetime of the given computer node is less than at least one other corresponding expected remaining lifetime. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0219] According to an example implementation, the instructions, when executed, further cause the system to select the given computer node in response to determining that the given computer node's corresponding expected remaining lifetime is the minimum expected remaining lifetime. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0220] According to an example embodiment, the information related to wear includes information related to at least one of errors attributable to physical memory of the computer nodes or age of the physical memory. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0221] According to an example implementation, the information related to wear includes information related to at least one of a temperature of a physical processor of a computer node, a clock rate of the physical processor, or an age of the physical processor. Among potential advantages, wear is more evenly distributed among computer nodes.

[0222] According to an example embodiment, the information related to wear includes information related to at least one of errors attributable to a physical network interface of the computer node or an age of the physical network interface. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0223] According to an example embodiment, the information related to wear includes information related to at least one of errors attributable to a physical storage device associated with the computer node or an age of the physical storage device. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0224] According to an example implementation, the information related to wear is from a program that reads data indicative of wear from at least one of a hardware register of a computer node or a system event log of the computer node. Among potential advantages, wear is more evenly distributed among computer nodes.

[0225] According to an example implementation, information related to wear is provided by a baseboard management controller. Among potential advantages, wear is more evenly distributed among computer nodes.

[0226] According to an example implementation, the system includes a distributed system associated with a single virtual machine, and reconfiguration of the system is performed through a hypercore of a computer node. Among potential advantages, wear is more evenly distributed among computer nodes.

[0227] According to an example embodiment, the system comprises a distributed system, a hypervisor is distributed among a plurality of computer nodes, and reconfiguring the system is performed by the distributed hypervisor. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0228] According to an example embodiment, identifying a particular computer node based on a ranking may include selecting a list of computer nodes based on the ranking. The list includes the particular computer node. The identifying further includes randomly or pseudo-randomly selecting the particular computer node from the list. Among potential advantages, wear is more evenly distributed among the computer nodes.

[0229] While the present disclosure has been described with respect to a limited number of embodiments, numerous modifications and variations of the present disclosure will become apparent to those skilled in the art having the benefit of this disclosure. The appended claims are intended to cover all such modifications and variations.

Claims

1. A non-transitory machine-readable storage medium comprising instructions that, when executed, cause a system having a plurality of computer nodes to perform the following operations: receiving information related to wear of the computer node; ranking the computer nodes according to their respective expected remaining lifetimes based on the information related to the wear; and The system is reconfigured based on the ranking in response to an event corresponding to at least one of: a spare computer node being added to the system, or a given computer node of the plurality of computer nodes being removed from the system.

2. The non-transitory machine-readable storage medium of claim 1, in: The event corresponds to a maintenance service, wherein the given computer node is removed from the system for maintenance; The reconfiguring comprises removing the given computer node from the system; and The instructions, when executed, further cause the system to select the given computer node for removal from the system in response to determining that the respective expected remaining lifetime of the given computer node is less than at least one other respective expected remaining lifetime of the expected remaining lifetimes.

3. The non-transitory machine-readable storage medium of claim 2, in: The instructions, when executed, further cause the system to estimate the expected remaining life using a model and then update the model in response to the expected remaining life determined from a maintenance check of the given computer node after the given computer node is removed.

4. The non-transitory machine-readable storage medium of claim 1, in: The event corresponds to a maintenance service, wherein the given computer node is removed from the system for maintenance; The reconfiguring includes adding the spare computer node to the system; and The instructions, when executed, further cause the system to select the backup computer node from a plurality of candidate backup computer nodes in response to determining that the respective remaining lifetime of the backup computer node is greater than at least one other respective expected remaining lifetime of another backup computer node in the plurality of candidate backup computer nodes.

5. The non-transitory machine-readable storage medium of claim 1, in, The system comprises a distributed system, and the instructions, when executed, further cause the system to: accessing application workloads provided by the plurality of computer nodes; determining that resources provided by the plurality of computer nodes to support the application workload are undersubscribed; In response to the determination, initiating the event to augment the resource; as well as In association with the augmenting, the backup computer node is selected from a plurality of candidate backup computer nodes in response to determining that the respective remaining lifetime of the backup computer node is greater than at least one other respective expected remaining lifetime of another backup computer node in the plurality of candidate backup computer nodes.

6. The non-transitory machine-readable storage medium of claim 1, in, The system comprises a distributed system, and the instructions, when executed, further cause the system to: accessing application workloads provided by the plurality of computer nodes; determining that resources provided by the plurality of computer nodes to support the application workload are oversubscribed; In response to the determination, initiating the event to scale down the resource; as well as In association with the reducing, the given computer node is selected for removal from the system in response to determining that the respective expected remaining lifetime of the given computer node is less than at least one other respective expected remaining lifetime of the expected remaining lifetimes.

7. The non-transitory machine-readable storage medium of claim 6, in, When executed, the instructions further cause the system to perform the following operations: The given computer node is selected in response to determining that the respective expected remaining lifetime of the given computer node is a minimum expected remaining lifetime among the expected remaining lifetimes.

8. The non-transitory machine-readable storage medium of claim 1, in, The information related to the wear includes information related to at least one of: errors attributable to physical memory of the plurality of computer nodes or an age of the physical memory.

9. The non-transitory machine-readable storage medium of claim 1, in, The information related to the wear includes information related to at least one of: a temperature of physical processors of the plurality of computer nodes, a clock rate of the physical processors, or an age of the physical processors.

10. The non-transitory machine-readable storage medium of claim 1, in, The information related to the wear includes information related to at least one of errors attributable to physical network interfaces of the plurality of computer nodes or an age of the physical network interfaces.

11. The non-transitory machine-readable storage medium of claim 1, in, The information related to the wear includes information related to at least one of errors attributable to physical storage devices associated with the plurality of computer nodes or an age of the physical storage devices.

12. The non-transitory machine-readable storage medium of claim 1, in, The information related to the wear is from a program that reads data indicative of the wear from at least one of hardware registers of the plurality of computer nodes or system event logs of the plurality of computer nodes.

13. The non-transitory machine-readable storage medium of claim 1, in, The information related to the wear is provided by a baseboard management controller.

14. The non-transitory machine-readable storage medium of claim 1, in, The system includes a distributed system associated with a single virtual machine, and reconfiguration of the system is performed by a hyperkernel of the plurality of computer nodes.

15. The non-transitory machine-readable storage medium of claim 1, in, The system comprises a distributed system, a hypervisor is distributed among the computer nodes, and reconfiguring the system is performed by the distributed hypervisor.

16. A distributed system, include: Multiple computer nodes; The plurality of computer nodes include instructions that can be executed to perform the following operations: receiving an indication of an event corresponding to a change in a number of the plurality of computer nodes; and In response to the event, a particular computer node associated with implementing the change is identified from a pool of candidate computer nodes based on respective wear levels associated with the candidate computer nodes.

17. The distributed system of claim 16, in: The event comprises adding the particular computer node to the plurality of computer nodes; The pool of candidate computer nodes includes a plurality of spare computer nodes; and The computer node wear leveling instructions are further executable to perform the following operations: receiving wear leveling information associated with the plurality of spare computer nodes; and A spare computer node among the plurality of spare computer nodes is identified as the specific computer node based on the wear leveling information.

18. The distributed system of claim 16, in: The event includes removing the particular computer node from the plurality of computer nodes; The plurality of computer nodes corresponds to a pool of candidate computer nodes; and The computer node wear leveling instructions are further executable to perform the following operations: receiving wear leveling information associated with the plurality of computer nodes; as well as A particular computer node associated with implementing the change is identified by the wear leveling engine from the pool of candidate computer nodes based on the ranking.

19. A method of a distributed system comprising a plurality of computer nodes, the method include: Receiving, by a node wear leveling engine, information related to wear of the computer node; Based on the information related to the wear, ranking, by the node wear leveling engine, the computer nodes according to their respective expected remaining lifetimes; as well as In response to an event corresponding to a change in the number of computer nodes, a particular computer node associated with implementing the change is identified by the node wear leveling engine from a pool of candidate computer nodes based on the ranking.

20. The method of claim 19, in, The identifying further comprises: selecting the list of computer nodes based on the ranking, wherein the list includes the particular computer node; and The particular computer node is selected randomly or pseudo-randomly from the list.

Images