Method and apparatus for delayed server recovery in a computing system

Abstract

Embodiments of the present disclosure relate to a deferred server recovery method and apparatus in a computing system. Various techniques for deferred server recovery are disclosed herein. In one embodiment, a method includes receiving a notification of a fault from a host in a computing system. The host is performing one or more computing tasks for one or more users. The method may include then determining whether recovery of the fault in the received notification is deferred on the host. In response to determining that the fault in the received notification is deferred, the method includes setting a time delay to perform a pending recovery operation on the host at a later time, and prohibiting additional allocation of computing tasks to the host.

Description

[0001] This application is a divisional application of an invention patent application with an international application date of December 29, 2016, which entered the Chinese national stage on June 12, 2018, Chinese national application number 201680072913.1, and invention name “Delayed server recovery method and device in a computing system”. Technical Field

[0002] The present application relates to computing systems, and more particularly to deferred server recovery in computing systems. Background Art

[0003] Data centers that provide cloud computing services typically include routers, switches, bridges, and other physical network devices that interconnect a large number of servers, network storage devices, and other types of physical computing devices via wired or wireless network links. Individual servers can host one or more virtual machines or other types of virtualized components accessible to cloud computing clients. Virtual machines can exchange messages (such as email) via virtual networks according to one or more network protocols supported by physical network devices. Summary of the invention

[0004] This Summary is provided to introduce some concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] In cloud computing, virtual machine availability generally refers to the ability to create new virtual machines based on requests or the uninterrupted accessibility of existing virtual machines on a particular server. However, server downtime events (such as restarts, power cycles, system upgrades, etc.) may cause system downtime and reduce virtual machine availability. For example, users may often experience five to even thirty minutes of downtime during a server restart. In addition, state information (e.g., calculation results, cached temporary data, etc.) in the virtual machines hosted on the restarted server may also be lost during the restart, which results in the loss of data or work results.

[0006] Several embodiments of the disclosed technology are directed to increasing virtual machine availability by deferring certain types of server failures, errors or problems, and improving the predictability of virtual machine downtime. In some embodiments, a controller (e.g., a data center manager) may use, for example, sensors, agents or other suitable mechanisms to monitor and detect hardware and / or software failures, errors or problems before immediate recovery. The controller may then determine whether a separate hardware / software failure, error or problem requires immediate recovery or whether recovery can be deferred to a later date / time. An example of a deferrable hardware failure may be a control layer problem (e.g., not responding to remote control instructions) with respect to a power distribution unit (PDU) or a top of rack (TOR) router serving a server. Such a control layer problem does not typically prevent the PDU or TOR router from continuing to operate, but may affect subsequent attempts to power on / off or perform other operations. An example of a deferrable software failure may include a vulnerability in an operating system or device driver that manifests itself as a "file in use" problem that prevents the creation and / or deletion of a virtual machine. A server restart may typically mitigate or correct such a problem. However, a server restart will also affect other virtual machines that are not affected by the vulnerability but are hosted on the same server. The detected deferrable faults may be stored in persistent computer-readable memory on the server, or reside in another storage location and associated with a particular server.

[0007] In response to determining that the recovery of the detected fault can be postponed, the controller can designate the specific server corresponding to the detected fault as unavailable for hosting additional virtual machines. When the server is designated as unavailable, the controller can also perform one or more operations in the operation to prepare for the final recovery of the designated server. For example, in one embodiment, the controller can set a time delay in which the designated server can perform a restart, power cycle, hardware replacement or other types of recovery operations. The controller can also continue to monitor several virtual machines or other tasks being performed by the designated server. In response to detecting that the server no longer hosts virtual machines or other tasks, the controller can instruct the server to perform pre-arranged (multiple) recovery operations regardless of the set time delay. In another embodiment, the controller can also instruct the designated server to retain the state information of all virtual machines currently hosted on the server. The state information can be stored on the server itself, on a network storage device, on a controller, or in other suitable storage locations.

[0008] When compared to conventional techniques, several embodiments of the disclosed technology can increase server uptime and improve user experience. For example, as discussed above, instead of performing a server recovery immediately upon detecting a failure, error, or problem, the server can continue to operate until, for example, a set time delay expires, or the server no longer hosts a virtual machine or other task. Thus, a "file in use" problem that prevents the deletion of one virtual machine will not cause a restart of the server and will not affect other virtual machines accessed or used by other users. Customer analysis has shown that most virtual machines have a short lifespan. For example, Microsoft More than 70% of virtual machines in a data center have a lifespan of 24 hours or less. Therefore, deferring server restarts by even 24 hours can significantly increase uptime and improve the user experience for a large number of cloud computing clients.

[0009] When compared to conventional techniques, several embodiments of the disclosed technology can also increase the predictability of server downtime. For example, by deferring the recovery of a fault (e.g., a file problem in use), the deferred recovery can be combined with other (multiple) faults or user-initiated operations (e.g., the initiation of a new virtual machine) at a later time, so that the user only experiences a single downtime event instead of multiple downtime events. In another example, a controller can provide a notification (e.g., a prompt, an email, etc.) to a user of a virtual machine currently hosted on a server designated as unavailable to notify the user of a pending restart on the server. In response, the user can manage the upcoming downtime by, for example, copying local data to a remote location, moving tasks to other virtual machines, arranging system / application upgrades to comply with the pending restart, or performing other suitable operations. As a result, the predictability and efficiency of managing virtual machine downtime are improved compared to conventional techniques. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a schematic diagram illustrating a computing network with deferred server recovery according to an embodiment of the disclosed technology.

[0011] Figure 2 is a diagram illustrating an embodiment according to the disclosed technology, Figure 1 Schematic diagram of some hardware / software components of a computing network.

[0012] Figure 3 is a diagram illustrating an embodiment of the disclosed technology, applicable to Figure 1 A block diagram of the hardware / software components of a computing network's controller.

[0013] Figure 4is a flow chart illustrating a process of postponing server recovery according to an embodiment of the disclosed technology.

[0014] Figure 5 is a flow chart illustrating a process of analyzing device failures according to an embodiment of the disclosed technology.

[0015] Figure 6 is a flow chart illustrating a process of performing a deferred recovery according to an embodiment of the disclosed technology.

[0016] Figure 7 is a flow chart illustrating a process of combining deferred recovery according to an embodiment of the disclosed technology.

[0017] Figure 8 Is applicable to Figure 1 A computing device that is part of a computing network. DETAILED DESCRIPTION

[0018] The following describes certain embodiments of systems, devices, components, modules, routines, data structures, and processes for delayed server recovery in a data center or other suitable computing network. In the following description, specific details of the components are included to provide a thorough understanding of certain embodiments of the disclosed technology. Those skilled in the relevant art will also understand that the technology may have additional embodiments. Figure 1-Figure 8 The technology can also be implemented without several details of the described embodiments.

[0019] As used herein, the term "computing network" generally refers to an interconnected computer network having multiple network nodes that connect multiple servers or hosts to each other or to an external network (i.e., the Internet). The term "network node" generally refers to a physical network device. Example network nodes include routers, switches, hubs, bridges, load balancers, security gateways, or firewalls. "Host" generally refers to a physical computing device configured to implement, for example, one or more visual computing devices or components or other suitable functions. For example, a host may include a server with a hypervisor that is configured to support one or more virtual machines or other suitable virtual components.

[0020] The computing network can be conceptually divided into an overlay network implemented on an underlying network. An "overlay network" generally refers to an abstract network implemented and operated on top of an underlying network. The underlying network may include multiple physical network nodes interconnected to each other and to physical endpoints. An overlay network may include one or more virtual networks. A "virtual network" generally refers to an abstraction of a portion of an underlying network in an overlay network. A virtual network may include one or more virtual endpoints referred to as "tenant sites", which are used individually by users or "tenants" to access virtual networks and associated computing, storage, or other suitable resources. A tenant site may host one or more tenant endpoints ("TEPs"), for example, virtual machines. A virtual network may interconnect multiple TEPs on different hosts. Virtual network nodes in an overlay network may be connected to each other via virtual links, which correspond individually to one or more network routes along one or more physical network nodes in the underlying network.

[0021] In cloud computing, virtual machine availability is a priority for a satisfactory user experience. Server restarts and / or other repairs are often required to recover from unexpected failures, errors, or problems with servers. However, such recovery may have a serious impact on virtual machine availability and result in significant downtime. Several embodiments of the disclosed technology can address at least some of the aforementioned shortcomings by deferring server recovery for certain types or classes of hardware and software failures, errors, or problems (collectively referred to as "deferrable problems").

[0022] In some embodiments, servers associated with deferrable issues may be designated as "awaiting deferred recovery" or "unavailable," and further allocation of virtual machines to these servers may be prohibited. As a result, no additional virtual machines may be deployed on these servers. At the same time, any existing virtual machines already hosted on the unavailable servers may continue to operate until expired, for example, by the corresponding user. In other embodiments, a programmatic notification may be given to affected users of the existing virtual machines. The notification may inform the affected users of, for example, the deferred issue and the scheduled time point for performing server recovery operations. The servers associated with the deferred issue may then be recovered by, for example, restarting at the scheduled time point, or restarting based on input from the affected users or administrators via, for example, an application program interface. Reference Figure 1-Figure 8 , additional embodiments of the disclosed technology are described in more detail below.

[0023] Figure 1 is a schematic diagram illustrating a computing network 100 with deferred server recovery according to an embodiment of the disclosed technology. Figure 1As shown, computing network 100 may include an underlying network 108 that interconnects multiple hosts 106, multiple tenants 101, and recovery controller 126. Figure 1 Specific components of the computing network 100 are shown, and in other embodiments, the computing network 100 may also include additional and / or different components. For example, in some embodiments, the computing network 100 may also include a network storage device, a maintenance manager, and / or other suitable components (not shown).

[0024] like Figure 1 As shown, the underlying network 108 may include one or more network nodes 112 that interconnect multiple hosts 106, tenants 101, and recovery controllers 126. In some embodiments, the hosts 106 may be organized into racks, action areas, groups, sets, or other suitable divisions. For example, in the illustrated embodiment, the hosts 106 are grouped into three host sets that are individually identified as first, second, and third host sets 107a-107c. In the illustrated embodiment, each of the host sets 107a-107c is operatively coupled to a corresponding network node 112a-112c, respectively, and the network nodes 112a-112c are generally referred to as "top of rack" or "TOR" network nodes. The TOR network nodes 112a-112c can then be operatively coupled to additional network nodes 112 to form a computer network that is a hierarchical, flat, mesh, or any other suitable type of topology that allows communication between the hosts 106, the recovery controller 126, and the tenants 101. In other embodiments, multiple sets of hosts 107a - 107c may share a single network node 112 .

[0025] Host 106 can be individually configured to provide computing, storage, and / or other suitable cloud computing services to tenant 101. For example, as described below with reference to Figure 2 In more detail, one of the hosts 106 may initiate and maintain one or more virtual machines 144 (in Figure 2). The tenant 101 may then utilize the initiated virtual machine 144 to perform computing, communication, and / or other suitable tasks. In some embodiments, one of the hosts 106 may provide virtual machines 144 for multiple tenants 101. For example, the host 106' may host three virtual machines 144 that correspond individually to each of the tenants 101-101b. During operation, the first tenant 101a may encounter a problem (e.g., a file problem in use) that may be recovered if the host 106' is restarted. However, the second tenant 101b and the third tenant 101c may not experience the same file problem in use. Thus, if the host 106' is restarted immediately, all of the first, second, and third tenants 101a-101c will experience a downtime event and thus negatively impact the user experience.

[0026] According to several embodiments of the disclosed technology, the recovery controller 126 can be configured to manage the recovery of the host 106 when such a deferrable problem is detected. In some embodiments, the recovery controller 126 may include a stand-alone server, a desktop computer, a laptop computer, or other suitable type of computing device operatively coupled to the underlying network 108. In other embodiments, the recovery controller 126 may be implemented as one or more network services executed on and provided by one or more of the hosts 106 or another server (not shown). Figure 3 Example components of resume controller 126 are described in greater detail.

[0027] Figure 2 is a diagram illustrating an embodiment of the disclosed technology implemented in Figure 1 Schematic diagram of an overlay network 108' on an underlying network 108. Figure 2 For the sake of clarity, only Figure 1 Certain components of the underlying network 108. Figure 2 As shown, the first host 106a and the second host 106b can each include a processor 132, a memory 134, and an input / output component 136 operatively coupled to each other. The processor 132 may include a microprocessor, a field programmable gate array, and / or other suitable logic devices. The memory 134 may include volatile and / or non-volatile media (e.g., ROM; RAM, magnetic disk storage media; optical storage media; flash memory devices, and / or other suitable storage media) and / or is configured to store data received from the processor 132 and instructions for the processor 132 (e.g., for executing the following reference Figure 4-Figure 7The input / output components 136 may include a display, a touch screen, a keyboard, a mouse, a printer, and / or other suitable types of input / output devices configured to accept input from an operator and / or an automated software controller (not shown) and provide output to an operator and / or an automated software controller (not shown).

[0028] The first host 106a and the second host 106b may separately contain instructions in memory 134 executable by the processor 132, so that the separate processors 132 provide a hypervisor 140 (individually identified as a first hypervisor 140a and a second hypervisor 140b) and a state agent 141 (individually identified as a first state agent 141a and a second state agent 141b). Even though the hypervisor 140 and the state agent 141 are shown as separate components, in other embodiments, the state agent 141 may be part of the hypervisor 140 or an operating system (not shown) executed on the corresponding host 106. In other embodiments, the state agent 141 may be a stand-alone application.

[0029] The hypervisor 140 may be individually configured to generate, monitor, expire, and / or otherwise manage one or more virtual machines 144 organized as tenant sites 142. For example, Figure 2 As shown, the first host 106a may provide a first hypervisor 140a that manages a first tenant site 142a and a second tenant site 142b, respectively. The second host 106b may provide a second hypervisor 140b that manages a first tenant site 142a' and a second tenant site 142b', respectively. Figure 2 106a and the second host 106b may host a plurality of virtual machines 144 for a particular tenant (not shown). For example, the first host 106a and the second host 106b may host a plurality of virtual machines 144 for a first tenant 101a ( Figure 1 ) of the tenant sites 142a and 142a'. The first host 106a and the second host 106b may host a second tenant 101b ( Figure 1 ) of both tenant sites 142b and 142b'. Each virtual machine 144 may be executing a corresponding operating system, middleware and / or application.

[0030] Still Figure 2, the computing network 100 may include an overlay network 108' having one or more virtual networks 146 interconnecting tenant sites 142a and 142b across multiple hosts 106. For example, a first virtual network 142a interconnects first tenant sites 142a and 142a' at a first host 106a and a second host 106b. A second virtual network 146b interconnects second tenant sites 142b and 142b' at a first host 106a and a second host 106b. Even though a single virtual network 146 is shown as corresponding to one tenant site 142, in other embodiments, multiple virtual networks 146 (not shown) may also be configured to correspond to a single tenant site 146.

[0031] Even if the virtual machines 144 are located on different hosts 106, the virtual machines 144 on the virtual network 146 can also access each other via the underlying network 108 ( Figure 1 ) communicate with each other. The communication of each virtual machine in the virtual network 146 can be isolated from the other virtual networks 146. In some embodiments, communication from one virtual network 146 across another virtual network can be allowed through a security gateway or other controlled manner. A virtual network address can correspond to one of the virtual machines 144 in a particular virtual network 146. Therefore, different virtual networks 146 can use the same one or more virtual network addresses. Example virtual network addresses can include IP addresses, MAC addresses, and / or other suitable addresses.

[0032] State agent 141 may be configured to provide notifications of hardware and / or software errors, failures, or problems to recovery controller 126. State agent 141 may also provide the operating status of host 106 to recovery controller 126. For example, in some embodiments, state agent 141 may provide the number of active virtual machines 144 currently hosted on a particular host 106 to recovery controller 126. In other embodiments, state agent 141 may also provide the CPU usage, storage capacity, and / or other suitable operating parameters of host 106 to recovery controller 126. Even if state agent 141 is Figure 2 106, in other embodiments, the network node 112 ( Figure 1 ) and / or other suitable components of computing network 100 may also separately include state agents that are generally similar to those on host 106.

[0033] Figure 3 is a diagram illustrating an embodiment of the disclosed technology, applicable to Figure 1 and Figure 2 A block diagram of certain hardware / software components of a recovery controller of the computing network 100 is shown. Figure 3In and in other figures herein, separate software components, objects, classes, modules and routines may be computer programs, processes or procedures written as source code in C, C++, C#, Java and / or other suitable programming languages. Components may include, but are not limited to, one or more modules, objects, classes, routines, features, processes, threads, executable files, libraries or other components. Components may be in source or binary form. Components may include aspects of source code before compilation (e.g., classes, features, programs, routines), compiled binary units (e.g., libraries, executable files) or components (e.g., objects, processes, threads) instantiated and used at runtime.

[0034] The components within the system can take different forms within the system. As an example, a system including a first component, a second component, and a third component may, but is not limited to, include a system in which the first component is a property of the source code, the second component is a binary compiled library, and the third component is a thread created at runtime. A computer program, process, or process may be compiled into an object, an intermediate, or a machine code, and is presented to be executed by one or more processors of a personal computer, a network server, a laptop computer, a smart phone, and / or other suitable computing devices. Likewise, a component may include a hardware circuit. Those of ordinary skill in the art will recognize that hardware can be considered as rigid software, and software can be considered as liquefied hardware. As just one example, the software instructions in the component can be burned into a programmable logic array circuit, or can be designed as a hardware circuit with a suitable integrated circuit. Likewise, hardware can be simulated by software. Various implementations of source, intermediate, and / or target code and associated data can be stored in a computer memory, including a read-only memory, a random access memory, a disk storage medium, an optical storage medium, a flash memory device, and / or other suitable computer-readable storage media in addition to a propagation signal.

[0035] like Figure 3 As shown, the host 106 (e.g., Figure 1 The first host 106a or the second host 106b) may include a state agent 141, which is operatively coupled to a host database 160 containing a record 162 of state information and a problem record 163. The state information 162 may include calculated values, operating parameters, or information related to the virtual machine 144 ( Figure 2 ) and other suitable information associated with the document. For example, state information 162 may include an accumulated value of a counter associated with virtual machine 144 and configured to count the total number of words in the document. In some embodiments, state information 162 may be temporarily stored in a cache (not shown) on host 106. In other embodiments, state information 162 may be stored in a permanent memory (not shown) on host 106.

[0036] Problem records 163 may individually contain data about failures, errors, or problems (collectively referred to as "problems") of host 106 detected by status agent 141. For example, in one embodiment, problem records 163 may contain the following data indicating that when hypervisor 107 ( Figure 2 ) attempts to delete the virtual machine 144, the virtual machine 144 on the host 106 encounters a problem with the file in use. In other embodiments, the problem record 163 may also include other suitable data records of hardware / software problems. In some embodiments, the problem record 163 may be temporarily stored in a cache on the host 106. In other embodiments, the problem record 163 may be stored in a permanent memory (not shown) on the host 106.

[0037] like Figure 3 As shown, state agent 141 may include a state module 154, a fault module 156, and a recovery module 158 operatively coupled to each other. Figure 3 106, but in other embodiments, at least one of the aforementioned modules may be part of other hardware / software components of the host 106. For example, in some embodiments, the recovery module 158 may be an operating system (not shown) or a hypervisor 107 ( Figure 2 ). In other embodiments, the recovery module 158 may be an independent application. In other embodiments, the state agent 141 may also include an input, output or other suitable type of module.

[0038] The status module 154 can be configured to monitor the status 172 of the host 106 and transmit the status 172 to the host 106 via, for example, Figure 2 The overlay network 108' and Figure 1 The status module 154 may be transmitted to the recovery controller 126 via the underlying network 108. In some embodiments, the status module 154 may include one or more hardware components, such as, for example, a thermal coupling configured to measure the operating temperature of the host 106. In other embodiments, the status module 154 may include one or more software sensors, such as, for example, configured to monitor the number of virtual machines 144 currently hosted by the host 106. In further embodiments, the status module 154 may include a combination of the foregoing hardware and software components.

[0039] The fault module 156 can be configured to monitor the detected problem 173 and transmit it to the recovery controller 126. In some embodiments, the fault module 156 can include a passive interface configured to receive notification of the problem as the problem occurs. In other embodiments, the fault module 156 can also include an active component configured to actively probe the internal and / or peripheral components of the host 106. For example, the fault module 156 can be configured to transmit a probe signal to, for example, a corresponding TOR network node 112. If the target response is not received from the TOR network node 112, the fault module 156 can indicate that a problem exists with respect to the TOR network node 112. In other embodiments, the fault module 156 can include a combination of passive interfaces and active components.

[0040] The recovery module 158 may be configured to receive instructions 174 from the recovery controller 126. In one embodiment, the instructions 174 may include data representing an accumulated timer value, the expiration of which will result in the initiation of a restart or other suitable type of recovery operation on the host 106. In response to such instructions 174, the recovery module 158 may be configured to instantiate a timer using the accumulated timer value and initiate a countdown (or countup) of the timer. In another embodiment, the instructions 174 may include data representing a command to immediately initiate the execution of a restart or other suitable type of recovery operation. In response, the recovery module 158 may cause the host 106 to restart by, for example, transmitting a restart command to the operating system of the host 106. In another embodiment, the recovery module 158 may also be configured to perform a timer reset, a timer adjustment, or other suitable operations in response to the corresponding instructions 174.

[0041] like Figure 3 As shown, the recovery controller 126 may include a processor 131 operatively coupled to a database 150 containing a state information record 162, a problem record 163, and an allocation record 165. The state information 162 and the problem record 163 may be generally similar to those described above with reference to the host 106, except with an indication of the associated host 106. The allocation record 165 may include data indicating one or more of: (i) the number of virtual machines 144 allocated to an individual host 106; (ii) the number of hosts 106 designated as unavailable to accept additional allocations of virtual machines 144; or (iii) remaining capacity for allocating additional virtual machines 144 in the computing network 100 (or a subdivision thereof).

[0042] Also like Figure 3As shown, the processor 131 can execute instructions to provide an interface component 133 and a processing component 135. The interface component 133 can be configured to receive a status 172 and a question 173 from the host 106, and transmit an instruction 174 to the host 106. The interface component 133 can also be configured to make the received question 173 as a question record 173 in the database 150. In some embodiments, the interface component 133 can include a network interface driver. In other embodiments, the interface component 133 can also include an application programming interface and / or other suitable components.

[0043] The processing component 135 can be configured to manage the delayed recovery of the host 106 based on the received status notification 172 and / or problem notification 173. Figure 3 As shown, the processing component 135 may include an analysis module 164, an allocation module 166, and a control module 168 operatively coupled to each other. The analysis module 164 may be configured to determine whether the received problem notification 173 is related to a deferrable problem or requires immediate recovery operation. In one embodiment, the analysis module 164 may determine that the problem is deferrable based on a rule set provided by, for example, an administrator. In other embodiments, the analysis module 164 may determine whether the problem is deferrable based on administrator input or other suitable criteria. Figure 5 Example embodiments of analyzing problems are described in more detail.

[0044] Allocation module 166 may be configured to designate host 106 as unavailable for further allocation of virtual machines in response to a determination by analysis module 164 that problem notification 173 is related to a deferrable problem. Allocation module 166 may also be configured to update allocation record 165 regarding the unavailability designation, and / or to update currently available capacity in computing network 100 (or a subdivision thereof) based on the unavailability designation.

[0045] In some embodiments, allocation module 166 may be configured to determine whether the current available capacity in computing network 100 (or a subdivision thereof) is less than an administrator-selected threshold. In response to determining that the current available capacity is less than the threshold, allocation module 166 may refrain from designating host 106 as unavailable, even when the problem is a deferrable problem. Instead, in one embodiment, allocation module 166 may cause control module 168 to generate a command for immediately performing a recovery operation on host 106. In another embodiment, allocation module 166 may designate host 106 as available, but only when a particular capacity (e.g., 85%, 90%, 95%, or other suitable percentage) in computing network 100 (or a subdivision thereof) has been exhausted.

[0046] The control module 168 is configured to generate one or more instructions 174 and cause the interface component 133 to transmit the one or more instructions 174 to the host 106. When the host 106 is designated as unavailable, the control module 168 may perform one or more of the following:

[0047] Calculating a date / time at which performance of a reboot or other suitable type of recovery operation will be initiated at the host 106;

[0048] Generates a command to save current state information 162 on the host 106;

[0049] Generate a command to save the received question 173 as question record 163 on the host 106; or

[0050] Generates a command to retrieve current state information 162 from the host 106 and permanently stores the retrieved state information 162 in the database 150 on the recovery controller.

[0051] In some embodiments, the calculated date / time may take into account other scheduled maintenance operations, such as a scheduled date / time for an upcoming hardware and / or software upgrade on host 106 .

[0052] In other embodiments, the calculation date / time (or time delay) may take into account previously scheduled maintenance repairs, unexpected downtime events on host 106, or other suitable information. In other embodiments, the calculation date / time may also be based on tenant 101 ( Figure 1 ) relative priority. For example, when the problem is related to a tenant with a higher priority than the second tenant 101b ( Figure 1 ) A higher priority first tenant 101a ( Figure 1 ), the calculated date / time may have a longer delay than if the problem was associated with the second tenant 101b. On the other hand, if the first tenant 101a has a lower priority than the second tenant 101b, the calculated date / time may include the same or a shorter delay than if the problem was associated with the second tenant 101b. The priority of the tenant 101 may be based on the subscription level or other suitable criteria. In another embodiment, the calculated date / time may be based on the capacity, utilization, or virtual machine turnover rate in the computing system in addition to or in lieu of scheduled maintenance, unexpected downtime events, user priorities, or other suitable parameters.

[0053] On the other hand, if analysis module 164 determines that the problem requires an immediate recovery operation (e.g., a reboot), control module 168 may generate and transmit a command to host 106 to immediately initiate a reboot or other suitable type of recovery operation. Figure 3 Not shown, but interface component 131 may also be configured to receive input from, for example, an administrator to manually cause control module 168 to generate and transmit a command to immediately initiate a reboot or other suitable type of recovery operation.

[0054] In operation, the fault module 156 of the state agent 141 may continuously, periodically, or in other suitable ways monitor any problems with the operation of the host 106. In response to a detected problem, the fault module 156 may generate a problem notification 173 and transmit the problem notification 173 to the recovery controller 126. Upon receipt, the interface component 133 of the recovery controller 126 communicates the problem notification 173 to the processing component 135 and optionally stores the received problem notification 173 in the database 150.

[0055] The analysis module 164 of the processing component 135 can then determine whether the received problem notification 173 is related to a deferrable problem based on, for example, a rule set provided by an administrator. In response to determining that the received problem notification 173 is related to a deferrable problem, in some embodiments, the allocation module 166 can designate the host 106 as unavailable for further allocation of virtual machines 144. The allocation module 166 can also cause the control module 168 to generate instructions 174 containing data about a delayed timer (e.g., an accumulated time value), the expiration of which will cause the host 106 to perform a reboot or other suitable type of recovery operation.

[0056] In some embodiments, the state module 154 of the state agent 141 may also monitor the operational state of the host 106 and transmit a state notification 172 to the recovery controller 126. In some embodiments, the state notification 172 may include an indication of the number of virtual machines 144 currently hosted on the host 106. The analysis module 164 may then determine whether the number of virtual machines 144 currently hosted on the host 106 is less than a threshold value (e.g., two, one, or zero). In response to determining that the number of virtual machines 144 currently hosted on the host 106 is less than the threshold value, the analysis module 164 may indicate to the allocation module 166 that the host 106 is ready to immediately perform a recovery operation. In response, the allocation module 166 may cause the control module 168 to generate another instruction 174 to command the host 106 to immediately initiate the execution of a restart or other suitable type of recovery operation.

[0057] Figure 4is a flow chart illustrating a process 200 for postponing server recovery according to an embodiment of the disclosed technology. Figure 1 and Figure 2 The computing system 100 and Figure 3 In other embodiments, the process 200 may also be implemented in other suitable systems. Figure 4 As shown, process 200 includes, at stage 201, from host 106 ( Figure 1 ) receives notification of an operational problem. Process 200 may also include analyzing the operational problem at stage 202 to determine whether the operational problem is a deferrable problem. In some embodiments, analyzing the operational problem may be based on a rule set that identifies which problems or categories of problems may be deferrable. In other embodiments, the foregoing analysis may be based on administrator input or other suitable criteria. Figure 5 Example embodiments of analyzing operational issues are described in greater detail.

[0058] In response to determining that the operational problem can be deferred at stage 204, process 200 may include designating the host as "unavailable" at stage 206, and performing a deferred recovery at stage 210. Figure 6 An example embodiment of performing a deferred recovery of a host is described in greater detail. Otherwise, process 200 may include initiating an immediate recovery of the host at stage 208 .

[0059] Figure 5 is a flow chart illustrating a process of analyzing an operational problem according to an embodiment of the disclosed technology. Figure 5 As shown, process 202 may include a decision stage 212 to determine whether the operational problem is a virtual machine-level problem. In response to determining that the operational problem is a virtual machine-level problem, process 202 may include: causing the host to perform virtual machine-level recovery operations, such as, for example, terminating an existing virtual machine, initiating a new virtual machine, and / or other suitable operations. In response to determining that the operational problem is not a virtual machine-level problem, process 202 may include another decision stage 216 to determine whether immediate recovery of the host is guaranteed. In one embodiment, immediate recovery of the host is guaranteed when the operational problem substantially impairs the basic functions of the host. For example, the host has suffered a physical storage failure. In other embodiments, immediate recovery of the host can be guaranteed based on a rule set provided by an administrator or other suitable criteria.

[0060] In response to determining that immediate recovery of the host is warranted, process 202 may include indicating a non-deferrable problem at stage 218. In response to determining that immediate recovery of the host is not warranted, process 202 may optionally include another decision stage 216 to determine whether a limit for designating unavailable hosts has been reached. In one embodiment, the limit for designating unavailable hosts may be based on the computing network 100 ( Figure 1 ) available capacity. In other embodiments, the limit for designating unavailable hosts may be based on a percentage of utilization or other suitable parameter of the computing network 100. In response to determining that the limit for designating unavailable hosts has been reached, the process 202 may return to indicate a non-deferrable problem at stage 218. Otherwise, the process 202 may include indicating a deferrable problem at stage 220.

[0061] Figure 6 2 is a flow chart illustrating a process 210 for performing a deferred recovery according to an embodiment of the disclosed technology. Figure 6 As shown, process 210 may include performing one or more of the following: adjusting the allocation of virtual machines to hosts at stage 222; initiating the persistence of state information at stage 224; or notifying the user of the virtual machines on the host at stage 226. In one embodiment, adjusting the allocation of virtual machines may include preventing further allocation of virtual machines to hosts and setting a delay timer for performing recovery operations. In other embodiments, the host may be associated with a low allocation class so that additional virtual machines are not allocated to the host until, for example, the available capacity of the computing network is below a preset threshold.

[0062] In some embodiments, initiating the persistence of the state information may include causing the host to permanently store the state information on the host. In other embodiments, initiating the persistence of the state information may include retrieving the state information from the host and permanently storing the state information on the recovery controller 126 ( Figure 1 ) or other suitable storage locations. In one embodiment, notifying the user may include sending an email to a user currently using the virtual machine hosted on the host. In other embodiments, notifying the user may also include using desktop notifications, simple text messages, or other suitable messaging technologies.

[0063] The process 210 may also include monitoring the server status at stage 228. The server status may include the current number of virtual machines, CPU usage, memory usage, and / or other suitable parameters. The process 200 may then include a decision stage 230 to determine whether the number of virtual machines hosted by the host is less than a preset threshold (e.g., two, one, or zero). In response to determining that the number of virtual machines hosted by the host is less than the preset threshold, the process 210 includes causing the host to initiate an immediate recovery at stage 232, regardless of the expiration of the delay timer. Otherwise, the process 210 includes another decision stage 231 to determine whether the set delay timer has expired. In response to determining that the set delay timer has expired, the process 210 proceeds to stage 232 to initiate an immediate recovery; otherwise, the process 210 returns to monitoring the server status at stage 228.

[0064] Figure 7 2 is a flow chart illustrating a process 240 of combining deferred recovery according to an embodiment of the disclosed technology. Figure 7 As shown, process 240 may include receiving notification of a new problem from a host at stage 241. Process 240 may then include a decision stage 242 to determine whether the host is associated with one or more pre-existing problems or scheduled maintenance operations (collectively referred to as "existing problems"). In response to determining that the host is associated with one or more existing problems, process 240 may include another decision stage 244 to determine whether the new problem can be combined with any existing problems.

[0065] In some embodiments, a new problem may be combined with an existing problem when a single restore operation may alleviate or at least partially resolve both the new problem and the existing problem. For example, both a new problem (e.g., a file in use problem) and an existing problem (e.g., an operating system upgrade) may require a reboot. Other examples of combinable existing problems may include, among others, planned or unplanned hardware maintenance, hardware failure, power failure, operating system crash, user updates (e.g., resizing a virtual machine) that result in the deletion and re-creation of a virtual machine.

[0066] In other embodiments, for example, based on corresponding recovery operations, new problems and existing problems can be assigned priorities or precedences. For example, a problem that requires invasive hardware repairs and long downtime (e.g., a hardware problem that requires manipulation of the host's memory, processor, or other hardware components) can be assigned a higher priority than another problem that only requires a restart (e.g., a software organization). Therefore, in one case, if the new problem has a higher priority than the existing problem, the new problem can replace the existing problem. In another case, if the new problem has a lower priority than the existing problem, the new problem can be incorporated into the existing problem that requires a more expensive recovery operation. In other embodiments, based on administrator input or other suitable criteria, the new problem can be combined with the existing problem.

[0067] In response to determining that the new question can be combined with the existing question, process 240 includes combining the new question with the existing question at stage 246 by, for example, setting a delay timer for both the new question and the existing question. In response to determining that the new question cannot be combined with the existing question, process 240 includes processing the new question at stage 248, example embodiments of which are described above with reference to Figure 4-Figure 6 Describe in more detail.

[0068] Figure 8 Is applicable to Figure 1 The computing device 300 may be adapted to include certain components of the computing network 100. For example, the computing device 300 may be adapted to include certain components of the computing network 100. Figure 1 The host 106 or the tag server 126. In a very basic configuration 302, the computing device 300 may include one or more processors 304 and a system memory 306. A memory bus 308 may be used to communicate between the processor 304 and the system memory 306.

[0069] Depending on the desired configuration, the processor 304 can be of any type, including but not limited to: a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 304 can include one or more levels of cache (such as a level 1 cache 310 and a level 2 cache 312), a processor core 314, and registers 316. An example processor core 314 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP core), or any combination thereof. An example memory controller 318 can also be used with the processor 304, or in some implementations, the memory controller 318 can be an internal part of the processor 304.

[0070] Depending on the desired configuration, system memory 306 may be of any type, including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.), or any combination thereof. System memory 306 may include an operating system 320, one or more applications 322, and program data 324. Figure 8 As shown, the operating system 320 may include a hypervisor 140 for managing one or more virtual machines 144. The basic configuration 302 described in Figure 8 denoted by those components within the inner dashed line.

[0071] The computing device 300 may have additional features or functions and additional interfaces to support communication between the basic configuration 302 and any other devices and interfaces. For example, a bus / interface controller 330 may be used to support communication between the basic configuration 302 and one or more data storage devices 332 via a storage interface bus 334. The data storage device 332 may be a removable storage device 336, a non-removable storage device 338, or a combination thereof. Examples of removable storage devices and non-removable storage devices include magnetic disk devices such as floppy disk drives and hard disk drives (HDDs), optical disk drives such as compact disk (CD) drives or digital versatile disk (DVD) drives, solid state drives (SSDs), and tape drives, etc. Example computer storage media may include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information such as computer-readable instructions, data structures, program modules, or other data. The term "computer-readable storage medium" or "computer-readable storage device" does not include propagated signals and communication media.

[0072] System memory 306, removable storage 336, and non-removable storage 338 are examples of computer-readable storage media. Computer-readable storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 300. Any such computer-readable storage media may be part of computing device 300. The term "computer-readable storage media" does not include propagated signals and communication media.

[0073] The computing device 300 may also include an interface bus 340 for supporting communication from various interface devices (e.g., output devices 342, peripheral device interfaces 344, and communication devices 346) to the basic configuration 302 via the bus / interface controller 330. Example output devices 342 include a graphics processing unit 348 and an audio processing unit 350, which may be configured to communicate with various external devices such as a display or speakers via one or more A / V ports 352. Example peripheral device interfaces 344 include a serial interface controller 354 or a parallel interface controller 356, which may be configured to communicate with external devices such as input devices (e.g., keyboards, mice, pens, voice input devices, touch input devices, etc.) or other peripheral devices (e.g., printers, scanners, etc.) via one or more I / O ports 358. Example communication devices 346 include a network controller 360, which may be arranged to support communication with one or more other computing devices 362 via one or more communication ports 364 over a network communication link.

[0074] The network communication link can be an example of a communication medium. Communication media can generally be implemented by other data in a computer-readable instruction, a data structure, a program module, or a modulated data signal (e.g., a carrier wave or other transmission mechanism), and can include any information transfer medium. A "modulated data signal" can be a signal that can be set or change one or more characteristics in a way that can encode information in a signal. As an example and not limitation, communication media can include wired media such as a wired network or a direct wired connection and wireless media such as acoustics, radio frequency (RF), microwaves, infrared (IR) and other wireless media. The term computer-readable medium as used herein can include both storage media and communication media.

[0075] The computing device 300 may be implemented as part of a small portable (or mobile) electronic device such as a cellular phone, a personal digital assistant (PDA), a personal media player device, a wireless network watch device, a personal headset device, a dedicated device, or a hybrid device including any of the above functionality. The computing device 300 may also be implemented as a personal computer including both laptop and non-laptop configurations.

[0076] The specific embodiments of the present technology have been described above for the purpose of illustration. However, various modifications may be made without departing from the foregoing disclosure. In addition, many elements of one embodiment may be combined with other embodiments in addition to or in place of the elements of other embodiments. Therefore, the present technology is not limited except as set forth in the appended claims.

Claims

1. A method performed by a computing device in a computing system having a plurality of hosts interconnected by a computer network, the method comprising: receiving a notification of a failure from a host in the computing system, the host currently executing one or more computing tasks for providing computing services to a user; In response to receiving the notification, determining whether recovery from the failure in the received notification is deferrable on the host, wherein recovery from the failure is deferrable when the host is still able to continue to provide the computing service to the user by executing the one or more computing tasks currently being executed by the host despite the failure in the received notification; as well as In response to determining that the fault received in the notification is deferrable, setting a time delay for subsequently performing a recovery operation on the host; as well as Allocating other computing tasks to the host is prohibited.

2. The method of claim 1 further comprising, in response to determining that the failure in the received notification is deferrable, transmitting instructions to the host to permanently store state information related to the one or more computing tasks currently being executed by the host.

3. The method of claim 1, further comprising, in response to determining that the failure in the received notification is not deferrable, transmitting instructions to the host for immediately performing a recovery operation on the host.

4. The method of claim 1 further comprises, after setting the time delay, notifying a user of a pending recovery operation and the set time delay, receiving input from the user, and initiating an immediate recovery of the host based on the received input when the set time delay has not expired.

5. The method according to claim 1, further comprising: determining whether the time delay set for subsequently performing a pending recovery operation on the host has expired; as well as In response to determining that the set time delay has expired, an instruction is transmitted to the host for performing the pending recovery operation on the host.

6. The method according to claim 1, further comprising: Monitoring the number of the one or more computing tasks currently being executed by the host; as well as In response to determining that the host is not currently executing any computing task, even if the set time delay has not expired, an instruction for executing a pending recovery operation on the host is transmitted.

7. The method according to claim 1, further comprising: In response to determining that the fault received in the notification is deferrable, determining whether a total number of hosts with pending recovery operations exceeds a threshold; as well as In response to the total number of hosts having pending recovery operations exceeding the threshold, instructions are transmitted to the hosts for immediately performing recovery operations on the hosts.

8. The method according to claim 7, further comprising: The threshold is determined by: The threshold is set using a static value based on user input, or The threshold is dynamically calculated based on one or more of a capacity or a usage rate of the computing system.

9. The method according to claim 1, wherein: said notification of said fault being a first notification of a first fault; The method further comprises: receiving a second notification of a second failure, a notification of an unplanned downtime event, or a planned maintenance event from the host in the computing system; determining whether recovery from the first failure is combinable with recovery from the second failure, the unplanned downtime event, or the planned maintenance event; and In response to determining that recovery from the first failure is combinable with recovery from the second failure, the unplanned downtime event, or the planned maintenance event, recovery from the first failure is performed together with recovery from the second failure, the unplanned downtime event, or the planned maintenance event.

10. The method of claim 1, wherein setting the time delay comprises: The time delay is set to a predetermined value, or is calculated based on one or more of a capacity, a usage rate, or a virtual machine turnover rate in the computing system.

11. The method of claim 1 , wherein setting the time delay comprises: The time delay is calculated based on a priority level of a user associated with the detected fault.

12. A computing device interconnected to a plurality of hosts via a computer network in a computing system, the computing device comprising: processor; as well as a memory operatively coupled to the processor, the memory containing instructions executable by the processor to cause the computing device to: receiving a notification of a failure from a host among a plurality of hosts in the computing system, the host currently executing a computing task for providing a computing service to a user; determining whether an immediate recovery of the fault in the received notification is required on the host, wherein the immediate recovery of the fault is not required when the host is still able to continue to provide the computing service to the user by executing the computing task currently being executed by the host despite the presence of the fault in the received notification; as well as In response to determining that immediate recovery of the failure in the received notification is not required, Setting a time delay so as to subsequently perform a recovery operation for the failure on the host; as well as Allocating other computing tasks to the host is prohibited.

13. A computing device according to claim 12, wherein the memory includes additional instructions, which are executable by the processor to cause the computing device to: in response to determining that the immediate recovery of the fault in the received notification is not required, transmit instructions to the host to permanently store state information related to the computing task currently being executed by the host.

14. A computing device according to claim 12, wherein the memory includes additional instructions, which are executable by the processor to cause the computing device to: in response to determining that the immediate recovery of the fault in the received notification is required, transmit instructions to the host for immediately performing recovery operations on the host.

15. The computing device of claim 12, wherein the memory includes additional instructions executable by the processor to cause the computing device to: After setting the time delay, notifying a user of a pending restore operation and said time delay being set; receiving input from the user allowing immediate resumption of the host; as well as Even when the set time delay has not expired, an immediate recovery of the host is initiated based on the input received.

16. The computing device of claim 12, wherein the memory includes additional instructions executable by the processor to cause the computing device to: determining whether the time delay set to subsequently perform a pending recovery operation on the host has expired; and In response to determining that the set time delay has expired, an instruction is transmitted to the host for performing the pending recovery operation on the host.

17. The computing device of claim 12, wherein the memory includes additional instructions executable by the processor to cause the computing device to: monitoring the number of the computing tasks currently being executed by the host; and In response to determining that the host is not currently executing any computing task, even if the set time delay has not expired, an instruction for executing a pending recovery operation on the host is transmitted.

18. A method performed by a computing device in a computing system having a plurality of hosts interconnected by a computer network, the method comprising: receiving a notification of a failure from a host in the computing system, the host currently executing one or more computing tasks for providing computing services to a remote user; as well as In response to receiving the notification of the fault from the host, determining whether the host can continue to provide the computing service to the user by executing the one or more computing tasks currently being executed by the host despite the failure in the received notification; as well as In response to determining that the host is still able to provide the computing service despite the failure in the received notification, delaying a recovery operation on the host until a later time, the recovery operation being configured to at least mitigate the fault in the notification received from the host; as well as The host from which the notification of the failure is received is prevented from being assigned other computing tasks.

19. The method of claim 18, wherein: said notification of said fault being a first notification of a first fault; The method further comprises: receiving a second notification of a second failure, a notification of an unplanned downtime event, or a planned maintenance event from the host in the computing system; determining whether recovery from the first failure is combinable with recovery from the second failure, the unplanned downtime event, or the planned maintenance event; and In response to determining that recovery from the first failure is combinable with recovery from the second failure, the unplanned downtime event, or the planned maintenance event, recovery from the first failure is performed together with recovery from the second failure, the unplanned downtime event, or the planned maintenance event.

Images