Hot restart of the hypervisor

Abstract

A hot restart of a hypervisor by replacing a running first hypervisor with a second hypervisor with minimal perceptible downtime to guest partitions. The first hypervisor executes on a computing system. The first hypervisor is configured to create one or more guest partitions. During the hot restart, a service partition is created and initialized with a second hypervisor. At least a portion of the runtime state of the first hypervisor is migrated and synchronized to the second hypervisor using a reverse hypercall. After synchronization, the second hypervisor is devirtualized from the service partition to replace the first hypervisor. The devirtualization includes transferring control of hardware resources from the first hypervisor to the second hypervisor using the previously migrated and synchronized runtime state.

Description

[Background technology]

[0001] background

[0001] Virtualization in computing often refers to abstracting physical components into logical objects. A virtual machine (VM) can virtualize hardware resources, including processors, memory, storage, and network connectivity, and present the virtualized resources to a host operating system. The process of virtualization of a VM includes at least two parts: (1) mapping virtual resources or state, such as registers, memory, or files, to real resources in the underlying physical hardware, and (2) using machine instructions and / or system calls to perform actions specified by the virtual machine instructions and / or system calls, such as emulation of the virtual machine's Application Binary Interface (ABI) or Instruction Set Architecture (ISA) interface.

[0002]

[0002] A hypervisor is a software layer that provides the environment (i.e., virtualized hardware partition) within which virtual machines operate. The hypervisor sits between the VMs and the physical resources on a physical hardware computing system. Without a hypervisor, operating systems communicate directly with the underlying hardware; disk operations go directly to the disk subsystem and memory calls are fetched directly from physical memory. When multiple operating systems in multiple VMs are running simultaneously on a single machine, the hypervisor manages the interaction between each VM and the shared hardware to prevent simultaneous control of the shared hardware by multiple VMs.

[0003]

[0003] When a hypervisor is reset or restarted (e.g., due to a software upgrade), all VMs running on the hypervisor are interrupted (e.g., restarted, paused, etc.), which can significantly impact the workload and services running on the VMs.

[0004]

[0004] The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in such environments. Rather, this background is only provided to illustrate certain example technology areas where some embodiments described herein may be practiced. Summary of the Invention [Means for solving the problem]

[0005] Quick Overview

[0005] This Summary is provided to introduce a selection of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.

[0006]

[0006] The embodiments described herein relate to a hot restart of a hypervisor that replaces a first hypervisor with a second hypervisor with little interference to currently running guest partitions. The embodiments described herein are implemented in a computing system. First, the computing system executes a first hypervisor on the computing system. The first hypervisor is configured to create one or more guest partitions. During a soft restart of the hypervisor, a service partition is created. A second hypervisor is initialized in the service partition. The service partition is created with an identity mapping between its guest physical address (GPA) space and its host physical address (HPA) space. Optionally, the service partition may be given additional privileges beyond a traditional partition / virtual machine to facilitate its initialization. Any execution environment that can meet the boot and security requirements of the platform may create and / or initialize the service partition. A component in a trusted computing core (TCB) of the computing system, such as the first hypervisor, may be tasked with creating or initializing the second hypervisor. During initialization, at least a portion of the state of the first hypervisor is initialized to the second hypervisor. The state of the first hypervisor includes, but is not limited to, (1) a static system state, (2) a dynamic system state, and (3) a logical software state. The static system state includes, but is not limited to, the system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes the hardware architecture state visible to the guest, including, but is not limited to, general purpose registers, mode control registers. The dynamic system state also includes the virtualization instruction set architecture (ISA) specific hardware state, including, but is not limited to, the state of the guest VM, such as the state of the virtual central processing unit (CPU), the second level page table (also known as nested page table), and the list of allocated hardware devices (e.g., network cards).The logical software state includes, but is not limited to, a page frame number (PFN) database. Because the dynamic system state and the logical software state may change continuously during runtime, these states are also referred to as runtime states. Finally, a second hypervisor is devirtualized from the service partition to replace the first hypervisor, and the state of each VM is restored by the second hypervisor.

[0007] In some embodiments, at least one of the one or more guest partitions includes a privileged parent partition, which runs a host operating system including an orchestrator configured to coordinate the initialization and synchronization of the second hypervisor. In some embodiments, the first hypervisor enables and supports the orchestrator to register and complete certain requests (e.g., hypercall intercepts, register intercepts) received from the second hypervisor. These requests (sent from the second hypervisor to the first hypervisor) are also referred to as "intercepts" because they are intercepted (e.g., registered and completed) by the orchestrator. In some embodiments, in response to receiving an intercept, the orchestrator issues a "reverse hypercall" to the second hypervisor that conforms to the application binary interface (ABI) of the hypercall to transition the associated state. A hypercall is a call from a guest partition to a hypervisor, and a "reverse hypercall" is a call from a privileged software or hardware component (e.g., the orchestrator) to the second hypervisor in the service partition. In some embodiments, the orchestrator may share some of the memory pages with the first hypervisor and the second hypervisor for communication purposes (e.g., hypercalls, intercepts, and / or reverse hypercalls).

[0008]

[0008] Although example embodiments are implemented within a dedicated partition (e.g., a service partition) for providing services to a hypervisor, the overall technique of pre-initializing data structures within a virtual machine and / or migrating / synchronizing runtime state within a VM / partition to reduce service downtime and provide better uptime and continuity may be used in any part of the OS of any VM.

[0009]

[0009] Additional features and advantages will be set forth in the following description, and will be in part apparent from the description, or may be learned by the practice of the teachings herein. The features and advantages of the invention may be realized by means of the instruments and combinations particularly pointed out in the appended claims. Features of the invention will become more fully apparent from the following description and the appended claims, or may be learned by the practice of the invention as set forth hereinafter.

[0010] BRIEF DESCRIPTION OF THE DRAWINGS

[0010] To explain how the above-listed advantages and features, as well as other advantages and features, can be obtained, a more particular description of what has been briefly described above will be made by reference to specific embodiments illustrated in the accompanying drawings, in which, understanding that these drawings depict only typical embodiments and therefore should not be considered limiting in scope, the embodiments will be described and explained with further specificity and detail using the accompanying drawings. [Brief description of the drawings]

[0011] [Figure 1A]

[0011] An example of a computing system including a hypervisor that hosts one or more virtual machines is shown. [Figure 1B]

[0011] An example of a computing system including a hypervisor that hosts one or more virtual machines is shown. [Diagram 2]

[0012] 1 shows a schematic diagram of isomorphism between a guest partition and a host computing system. [Figure 3A]

[0013] 1 illustrates an example embodiment of VM state management. [Figure 3B] 1 illustrates an example embodiment of VM state management. [Figure 4]

[0014] 4 illustrates an example embodiment 400 of memory virtualization using deep memory page tables. [Figure 5A]

[0015] 1 illustrates an example of a computing system in which hot restart of a hypervisor is enabled. [Figure 5B] 1 illustrates an example of a computing system in which hot restart of a hypervisor is enabled. [Figure 6A]

[0016] 1 illustrates a flow diagram of an example of a method for hot restart of a hypervisor. [Figure 6B]

[0017] 1 illustrates a flow diagram of an example of a method for initializing a service partition. [Figure 6C]

[0018] 1 illustrates a flow diagram of an example method for synchronizing a portion of a runtime state of a first hypervisor with a second hypervisor. [Figure 6D]

[0019] 1 illustrates a flow diagram of an example method for collecting runtime state of a first hypervisor. [Figure 6E]

[0020] 1 illustrates a flow diagram of an example method for sharing runtime state of a first hypervisor with a second hypervisor. [Figure 6F]

[0021] 1 illustrates a flow diagram of an example method for replicating a runtime state of a first hypervisor with a second hypervisor. [Figure 6G]

[0022] 1 illustrates a flow diagram of an example of a method for devirtualizing a second hypervisor. [Figure 7]

[0023] 1 illustrates an example of a computing system in which the principles described herein can be employed. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Detailed Description

[0024] The embodiments described herein relate to a hot restart of a hypervisor that replaces a first hypervisor with a second hypervisor with little interference to currently running guest partitions. The embodiments described herein are implemented in a computing system. First, the computing system executes a first hypervisor on the computing system. The first hypervisor is configured to create one or more virtual machines / partitions, each of which hosts a guest operating system. During the soft restart of the hypervisor, a service partition is created having identity-mapped second level page tables. Then, the second hypervisor is initialized in the service partition and synchronized with at least a portion of the state of the first hypervisor. Any execution environment that can meet the boot and security requirements of the platform can create and / or initialize the second hypervisor. A component in a trusted computing base (TCB), such as the first hypervisor, can be tasked with creating and / or initializing the service partition.

[0013]

[0025] In some embodiments, the state of the first hypervisor includes, but is not limited to, (1) static system state, (2) dynamic system state, and (3) logical software state. The static system state includes, but is not limited to, the system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes the hardware architecture state visible to the guest, including, but is not limited to, general purpose registers, mode control registers. The dynamic system state also includes the virtualization instruction set architecture (ISA) specific hardware state, including, but is not limited to, the state of the guest VMs, such as the state of the virtual central processing unit (CPU), the second level page table (also known as nested page table), and the list of assigned hardware devices (e.g., network cards). The logical software state includes, but is not limited to, the page frame number (PFN) database. Because the dynamic system state and logical software state may change continuously during runtime, these states are also referred to as runtime states. Finally, the second hypervisor is devirtualized from the service partition to replace the first hypervisor, and the state of each VM is restored by the second hypervisor.

[0014]

[0026] In some embodiments, at least one of the one or more guest partitions includes a privileged parent partition, the parent partition running a host operating system including an orchestrator configured to coordinate the initialization and synchronization of the second hypervisor. In some embodiments, the first hypervisor is part of a trusted computing board (TCB) of the system, and the orchestrator executes within a trusted execution environment (TEE) to maintain the reliability of inputs generated for the second hypervisor. In some embodiments, the first hypervisor enables and supports the orchestrator to register and complete certain requests received from the second hypervisor. These requests (sent from the second hypervisor to the first hypervisor) are intercepted (e.g., registered and completed) by the orchestrator, such requests are also referred to as "intercepts." In some embodiments, in response to receiving an intercept, the orchestrator issues a "reverse hypercall" to the second hypervisor that conforms to the hypercall's application binary interface (ABI) to transition the associated state. A hypercall is a call from a guest partition to the hypervisor that typically requires an appropriately privileged operation. Switching from the guest partition context to the hypervisor context is accomplished using platform-specific instructions. Control is returned from the hypervisor context to the guest context upon completion of the privileged operation using platform-specific instructions / mechanisms. A "reverse hypercall" requests an operation from a privileged component (e.g., an orchestrator) to the hypervisor in a service partition that uses a hypercall ABI. Switching from the orchestrator to a second hypervisor context is accomplished by allowing execution of a virtual processor in the service partition with registers formatted as required by the hypercall ABI.Upon completion of the reverse hypercall, the second hypervisor generates a very specialized intercept that causes an intentional context switch from the second hypervisor back to the orchestrator in the service partition. In some embodiments, the orchestrator may share one or more memory pages with the first and second hypervisors for communication purposes (e.g., hypercalls, intercepts, and / or reverse hypercalls).

[0015]

[0027] In some embodiments, initializing the second hypervisor includes the first hypervisor generating a loader block for the second hypervisor. The loader block includes logical constructs that describe a portion of the static system state for the initialization. The static system state includes various system invariants such as system topology, memory map, and layout. The memory map includes an identity map of substantially all system memory. In some embodiments, the identity map includes at least the memory visible to the first hypervisor being replaced. For example, in some cases, the first (old) hypervisor and the second (new) hypervisor both know about all of the memory in the system. In some cases, at least one of the first hypervisor and / or the second hypervisor does not know about all of the memory, such as when some RAM containing an error must be taken offline or when the second (new) hypervisor must know about new memory that was hot added during a hot restart. The orchestrator obtains one or more system invariants of the computing system, including at least one or more characteristics supported by the hardware resources of the computing system, (1) directly from the hardware or system resources if the particular resource is not virtualized by the hypervisor, and / or (2) from the first hypervisor if the orchestrator's access to the underlying physical hardware resources is virtualized by the first hypervisor. In some embodiments, the orchestrator migrates the relevant system invariants by a reverse hypercall. A second hypervisor is then initialized based on the shared system invariants. In some embodiments, initializing the second hypervisor may also include providing the second hypervisor with read-only native access to certain physical resources.

[0016]

[0028] After the second hypervisor is initialized with the static system state in the service partition, the computing system migrates the runtime state of the first hypervisor to the second hypervisor and keeps it synchronized. The runtime state includes at least the dynamic system state and the logical software state. Synchronizing the runtime state of the first hypervisor may include the orchestrator collecting the runtime state of the first hypervisor and sharing / migrating the runtime state with the second hypervisor on a reverse hypercall. As discussed above, the dynamic system state includes the hardware architecture state visible to the guest, including but not limited to general purpose registers, mode control registers. The dynamic system state also includes the state of the guest VM, such as but not limited to the state of the virtual central processing unit (CPU), the second level page table (also known as nested page table), and the virtualization instruction set architecture ISA-specific hardware state, including the list of allocated hardware devices (e.g., network cards). The logical software state includes but is not limited to the page frame number (PFN) database. The second hypervisor then replicates the shared second level memory page tables and / or PFN database and keeps them dormant until the second hypervisor is eventually de-virtualized.

[0017]

[0029] In addition, during this synchronization, guest partitions executing on the first hypervisor continue to operate and issue hypercalls to the first hypervisor. Thus, after the initial synchronization of computationally intensive state, additional incoming hypercalls to the first hypervisor that affect previously transmitted state may also need to be synchronized to the second hypervisor. In some embodiments, such state changes are also transmitted using reverse hypercalls.

[0018]

[0030] Once the first and second hypervisors are fully or nearly synchronized, the second hypervisor is de-virtualized. In some embodiments, de-virtualization involves the first hypervisor "trampolined" to the second hypervisor, relinquishing all physical hardware control to the second hypervisor. A "trampoline" may refer to a memory location that holds addresses pointing to interrupt service routines, I / O routines, etc. In this context, "trampolined" refers to transferring control of all physical hardware from the first hypervisor to the second hypervisor. In some embodiments, trampolining may be accomplished by reusing the computing system's existing control transfer mechanism for kernel soft reboot. In some embodiments, no other system or user software, barring system firmware, is allowed to execute during the trampolining process. However, direct memory access (DMA) that was programmed prior to de-virtualization continues to execute and is allowed to complete. In some embodiments, the second hypervisor may undergo additional initialization to verify hardware state, re-initialize hardware, and / or initialize new hardware not previously programmed by the first hypervisor. De-virtualization may also include freezing each of the guest partitions currently running on the first hypervisor and transmitting the final state of the first hypervisor to the second hypervisor. Each of the guest partitions is then switched onto the second hypervisor. The second hypervisor then thaws each of the guest partitions. The memory footprint of the first hypervisor may be reclaimed quickly or slowly by the computing system under overall memory pressure.

[0019]

[0031] Because the embodiments described herein are implemented within a VM environment capable of simultaneously supporting multiple guest partitions, each running its own operating system and associated application programs, some introductory discussion of virtualization and hypervisors is provided with respect to FIGS. 1-4.

[0020]

[0032] Hypervisor-based virtualization often allows a privileged host operating system (running in a "parent" partition) and multiple guest operating systems (running in "child" partitions) to simultaneously share access to a single computing system's hardware, with each operating system being given the illusion of having access to a full set of system resources. To create this illusion, in some embodiments, a hypervisor in a computing system creates multiple partitions that operate as virtual hardware machines (or VMs), each running its own operating system and associated application programs. Each operating system controls and manages a set of virtualized hardware resources.

[0021]

[0033] 1A illustrates an example of a computing system 100A that includes a hypervisor 140A that hosts one or more VMs 110A, 120A. The computing system 100A includes various hardware devices 150, such as one or more processors 151, one or more storage devices 152, and / or one or more peripheral devices 153. The peripheral devices 153 may be configured to provide input and output (I / O) for the computing system 100A. Ellipsis 154 indicates that different or additional hardware devices may be included within the computing system 100A. Hereinafter, the physical hardware 150 that runs the hypervisor 140A is also referred to as the "host computing system" 100A (also referred to as the "host"), and the VMs 110A, 120A hosted on the hypervisor 140A are also referred to as the "guest partitions" (also referred to as the "guests").

[0022]

[0034] As shown in FIG. 1A, the hypervisor 140A hosts one or more guest partitions (e.g., VM A 110, VM B 120). Ellipsis 130 indicates that any number of guest partitions may be included in the computing system 100A. The hypervisor 140A allocates a portion of the computing system 100A's physical hardware resources 150 to each of the guest partitions 110A, 120A, providing the guest partitions 110A, 120A with the illusion of ownership of the resources 114A, 124A. These "illusory" resources are referred to as "virtual" resources. Each of the VMs 110A, 120A uses its own virtual resources 114A, 124A to run its own operating system (OS) 113A, 123A. The operating systems 113A, 123A can allocate the virtual resources 114A, 124A to their various user applications 112A, 122A. For example, guest partition 110A executes OS 113A and user application 112A. If OS 113A is a Windows® operating system, then user application 112A will be a Windows® application. As another example, VM 120A executes OS 123A and user application 1122A. If OS 123A is a Linux operating system, then user application 122A will be a Linux application.

[0023]

[0035] Each virtual hardware resource 114A, 124A may or may not have a corresponding physical hardware resource 150. If a corresponding physical hardware resource 150 is available, the hypervisor 140A determines how to provide access to the guest partitions 110A, 120A that request its use. For example, the resource 150 may be partitioned or time-shared. If a virtual hardware resource 114A, 124A does not have a matching physical hardware resource, the hypervisor 140A can emulate the desired hardware resource action, typically through a combination of software and other hardware resources physically available on the host computing system 100A.

[0024]

[0036] As shown in FIG. 1A, in some embodiments, the hypervisor 140A may be the only software that runs within the highest privilege level defined by the system architecture. Such a system is called a native VM system. Conceptually, in a native VM system, the hypervisor 140A is first installed on the bare hardware, and then the guest partitions VM A 110A and VM B 120A are installed on the hypervisor 140A. The guest operating systems 113A, 123A and other applications 112A, 122A with lower privileges run within a level of privileges lower than that of the hypervisor. This generally means that the privilege levels of the guest OSs 113A, 123A may be emulated by the hypervisor 140A.

[0025]

[0037] Alternatively, in some embodiments, the hypervisor is installed on a host platform that is already running an existing OS. Such a system is called a Hosted VM system. In a Hosted VM system, the hypervisor utilizes the functionality already available on the host OS to control and manage the resources desired by each of the guest partitions. In a Hosted VM system, the hypervisor may be implemented at the user level or at a privilege level similar to the host operating system. Alternatively, part of the hypervisor is implemented at the user level and another part of the hypervisor is implemented at the privilege level.

[0026]

[0038] In some embodiments, one of the guest partitions running on the same computing system may be considered more privileged than the other guest partitions. FIG. 1B illustrates an example of such an embodiment, where the more privileged guest partition may be referred to as parent partition 110B and the remaining partitions as child partitions 120B. In some embodiments, parent partition 110B includes operating system 113B and / or virtualization service module 115B. Each of operating system 113B and / or virtualization service module 115B may have direct access to hardware devices 150 through device drivers 114B. Thus, operating system 113B may also be referred to as a host operating system. Additionally, virtualization service module 115B may be capable of creating child partitions 120B using hypercalls. Depending on the configuration, virtualization service module 115B may expose a subset of hardware resources to each child partition 140B through virtualization client module 125B. Child partition 120B generally does not have direct access to a physical processor and / or cannot handle real interrupts. Instead, child partition 120B can use virtualization client module 125B to obtain a virtual view of virtual hardware resources 124B.

[0027]

[0039] In some embodiments, parent partition 110B may also include a VM management service application 112B that may enable a user (e.g., a system administrator) to view and modify the configuration of virtualization service module 115B. For example, hypervisor 140B may be a Microsoft® Hyper-V hypervisor and parent partition may run Windows® Server. The user interface of the parent partition may provide a window that displays the full user interface of child partition 120B. Interaction with applications running on child partition 120B may occur within the window. If host operating system 113B is Windows®, a graphical window may be established on the desktop interface to interact with child partition 120B on the same platform. Elements 113B, 122B, 123B, and / or 124B of FIG. 1B are similar to elements 113A, 122A, 123A, and / or 124A of FIG. 1A and therefore will not be discussed further.

[0028]

[0040] Whether in a native or hosted VM system, the relationship between the hypervisor and the guest partition is generally similar to the relationship between an operating system and an application program in a traditional computing system. In a traditional computing system, the operating system generally functions in a privilege level higher than that of an application, e.g., in kernel mode as opposed to user mode. Similarly, in a VM environment, the hypervisor also runs in a privilege mode higher than that of the guest partition. When a guest partition needs to perform a privileged operation, such as updating a page table, the guest partition uses a hypercall to request such an operation, just like a system call in traditional operations.

[0029]

[0041] Thus, the described embodiments of the invention are applicable to both native and hosted VM systems, and the term "hypervisor" herein refers to a hypervisor implemented by any type of VM system.

[0030]

[0042] To further understand how a hypervisor operates, one must also understand how the hypervisor maintains the state of each guest partition. In a computing system, the constructed state of the computing system is contained within and maintained by the hardware resources of the computing system. Typically, there is a constructed hierarchy of state resources that ranges from registers at one end of the hierarchy to secondary storage (e.g., a hard drive) at the other end of the hierarchy.

[0031]

[0043] In a VM environment, each guest partition has its own constructed state information, and there may or may not be sufficient physical resources in the host computing system to map each element of the guest's state to its natural level in the host's memory hierarchy. For example, a guest's register state may actually be kept in the host platform's main memory as part of a register context block.

[0032]

[0044] In normal operation, the hypervisor 140A periodically switches control between the guest partitions 110A, 120A. Operations on the guest state modify the state maintained on the host computing system 100A as they are performed on the guest operating systems 113A, 123A. In some embodiments, the hypervisor 140A builds an isomorphism that maps the state of the virtualized guest operating systems 113A, 123A to the state of the physical host computing system 100A.

[0033]

[0045] 2 shows a schematic diagram of isomorphism 200 between a guest partition 210 and a host computing system 220. The isomorphism 200 maps guest states 211, 212 to host states 221, 222 using functions 230, namely V(state A)=state A', and V(state B)=state B'. For a sequence of operations in the guest partition 210 that modify the state of the guest partition 210 from state A 211 to state B 212, there is a corresponding sequence of operations in the host computing system 220 that modify the state of the host computing system from state A' 221 to state B' 222. The isomorphism 200 between the guest partition 210 and the host computing system 220 is managed by a hypervisor (e.g., 140A, 140B).

[0034]

[0046] In an embodiment, there are two basic ways to manage guest state such that this VM isomorphism is achieved. One way is to use a degree of indirection by keeping each guest's state in a fixed location in the host computing system's memory hierarchy, with a hypervisor-managed pointer pointing to the currently active guest state. When the hypervisor switches between guest partitions, the hypervisor changes the pointer to match the current guest. FIG. 3A shows an example of such an embodiment 300A of indirect state management. With reference to FIG. 3A, the host computing system's memory 320, managed by the hypervisor, stores the register values ​​of each VM A and B in register context blocks 321, 322. The register block pointer 311A ​​of the processor 310A points to the register context block 322 of the currently active guest partition (e.g., VM B). When a different guest partition is activated, the hypervisor modifies the pointer 311A ​​stored in the processor to point to the register context block 321, 322 of the activated guest partition, loads the program counter to point to the activated VM program, and begins execution.

[0035]

[0047] Another way to manage guest state is to replicate the guest's state information to its natural level in the memory hierarchy whenever it is activated by the hypervisor, and replicate it back when a different guest is activated. Figure 3B shows an example of such an embodiment 300B of guest state management by replication. As shown in Figure 3B, the host memory 320 managed by the hypervisor also stores the register values ​​of VMs A and B of each guest partition in register context blocks 321, 322. However, unlike in Figure 3A, here the hypervisor replicates all guest register contents 322 into the register file 311B of the processor 310B upon activation of VM B (after saving the previous guest's registers back to memory 320).

[0036]

[0048] The choice between indirection and duplication may depend, for example, on frequency of use and whether the guest state managed by the hypervisor is kept in a different type of hardware resource than on the native system. For frequently used state information such as general purpose registers, it may be preferable to swap the virtual machine state into the corresponding physical resource every time the virtual machine is activated. However, in either case, the register values ​​321, 322 of each VM are often kept in the host platform's main memory 320 as part of a register context block, as shown in Figures 3A and 3B.

[0037]

[0049] In addition to VM state management, memory management is also worth discussing. In a VM environment, each guest partition has its own set of virtual memory tables, also called "first level" memory page tables. Address translations in each of the first level memory page tables transform addresses in its virtual address space to locations in the guest physical memory. Here, the guest physical memory does not correspond to the host physical memory on the host computing system. Instead, the guest physical address (GPA) undergoes a further mapping to determine an address in the host hardware's physical memory, also called the host physical address (HPA). This mapping from GPA to HPA is done by another set of virtual memory tables on the host computing system, also called the "second level" or nested memory page tables. Note that the combined total size of the guest physical memory of all guests can be larger than the actual physical memory on the system. In an embodiment, the hypervisor maintains its own swap space, separate from the swap space of each guest, and the hypervisor manages physical memory by swapping guest physical pages in and out of its own swap space. Additionally, the state of all virtually or physically allocated pages and their corresponding attributes are stored in a list called the Page Frame Number (PFN) list. A track of the virtually or physically allocated pages is stored in a database called the PFN database.

[0038]

[0050] FIG. 4 illustrates an example of a memory virtualization embodiment 400 that uses multi-level memory page tables 440 and 450. Each entry in the first level memory table 440 maps a location (e.g., a PFN) in the virtual memory PFN database 410 to a location (e.g., a PFN) in the guest physical memory PFN database 420. As shown in FIG. 4, a portion of the PFN database 410 tracks the virtually allocated pages of a program running on VM A, and a portion of the PFN database 420 tracks the physical memory of VM A's guest VM. In addition, to translate the GPA to the HPA, the hypervisor also maintains a second level memory page table 450 that maps guest physical pages to host physical pages. There is also a PFN database 430 that tracks the physical memory of the host computing system.

[0039]

[0051] 4, physical page frame numbered 1500 is assigned to guest physical page frame numbered 2500, which is assigned to virtual memory page frame numbered 3000. Similarly, physical page frame 2000 is assigned to guest physical page frame numbered 6000, which is assigned to virtual memory page frame numbered 2000. The remaining physical memory pages can be allocated to other VMs or to the hypervisor itself. These remaining physical memory pages (e.g., memory 320), including those allocated to the hypervisor itself to record register values ​​for each VM, are also tracked by PFN database 430.

[0040]

[0052] 4 is merely a schematic diagram to illustrate a simplified concept of memory management using multi-level page tables. Additional mechanisms may be implemented to achieve the same or similar memory management objectives. For example, in some embodiments, page translation is supported by a combination of page tables and a translation lookaside buffer (TLB).

[0041]

[0053] Having described the virtual environment and how the hypervisor manages and virtualizes various hardware resources with reference to FIGS. 1A-4, a specific embodiment of a hot restart of the hypervisor will now be described with reference to FIGS. 5A and 5B.

[0042]

[0054] 5A illustrates an example of a computing system 500A in which hot restart of a hypervisor is enabled. The computing system 500A may include a native VM system, as shown in FIG 1A, in which the hypervisor 520A is the only software running within the highest privilege level defined by the system architecture. Alternatively, the computing system 500A may include a hosted VM system, in which the hypervisor 520A is installed on a computing system already running a host OS.

[0043]

[0055] Whether computing system 500A includes a native or hosted VM system, during a hot restart of the hypervisor, a new partition called a service partition 560 is created. In an embodiment, the service partition 560 is treated differently from other partition types. Upon creation of the service partition 560, hardware resources including at least some processor and memory resources are assigned to the service partition 560. In some embodiments, the allocation of hardware resources may be based on user input. In an alternative embodiment, the hypervisor 520A or the component that created the service partition 560 automatically assigns a predetermined portion of the processor and / or memory resources to the service partition 560.

[0044]

[0056] The processor resource allocation may specify the total amount of processing power required by service partition 560, leave the allocation of available processors to workload management software, or service partition 560 or hypervisor 520A may specify that certain processors in the system are dedicated for service partition 560's use. Service partition 560 or hypervisor 520A may specify that service partition 560 requires a certain number of processors, but that service partition 560 is willing to share those processors with other partitions. For example, if a total of eight processing units are required by service partition 560, service partition 560 or hypervisor 520A may specify that service partition 560 requires eight dedicated processors, or that service partition 560 requires sixteen processors, but only half the available computing power of each processor. The memory allocation (including RAM and / or hard disk) may specify the amount of memory in particular granular chunks, such as 1 MB.

[0045]

[0057] The service partition 560 is then initialized. Any component in a trusted computing platform (TCB) may be tasked with creating and / or initializing the service partition 560. In some embodiments, the first hypervisor 520A is part of the TCB, and the first hypervisor 520A creates and / or initializes the service partition 560. The initialization process includes bootstrapping, which includes a series of actions, each action activating a function that allows the next action to be performed until finally the entire system is activated. In some embodiments, the hypervisor 520A builds a loader block for the second hypervisor 561. Executing the initialization code allows other aspects of the service partition 560 to be initialized. As shown in FIG. 5A, unlike other regular guest partitions 540A, 550A, where conventional OSs 541A, 551A are loaded, the hypervisor 561 is loaded into the service partition 560. For clarity, hereinafter the hypervisors 520A and 561 are referred to as the first hypervisor 520A and the second hypervisor 561. In some embodiments, initializing the second hypervisor 561 may also include providing the second hypervisor with unique read-only access to certain physical resources 510.

[0046]

[0058] The purpose of hot restart of hypervisors is to eventually replace the first hypervisor 520A with the second hypervisor 561 with minimal to imperceptible interruption to the guest virtual machines. Before the second hypervisor 561 replaces the first hypervisor 520A, the first hypervisor 520A initializes the second hypervisor 561 with system invariants available to it (e.g., features supported by hardware resources 510) and then synchronizes the runtime state 512 with the second hypervisor 561. In some embodiments, the runtime state 512 is stored in memory (e.g., RAM) managed by the hypervisor 520A. As described with respect to Figures 3A, 3B, and 4, the runtime state 512 may include hardware architecture state, such as register values ​​321, 322 for each guest partition (e.g., guest A 540A, 550A), virtualization hardware state, such as second level memory page table 450 for each guest partition (e.g., guest A 540A, 550A), and / or software-defined state, such as PFN database 430 associated with the first hypervisor 520A.

[0047]

[0059] In some embodiments, communication between the first hypervisor 520A and the second hypervisor 561 during initialization and synchronization is coordinated by the orchestrator 530A. The first hypervisor 520A enables and supports the orchestrator 530A to register and complete certain requests received from the second hypervisor 561. The orchestrator 530A is a software component of the host computing system 500A that is configured to coordinate communication between the first hypervisor 520A and the second hypervisor 561. In some embodiments, the first hypervisor 520A is part of the system's Trusted Computing Board (TCB) and the orchestrator 530A executes within a Trusted Execution Environment (TEE) to maintain the reliability of inputs generated for the second hypervisor.

[0048]

[0060] In some embodiments, the orchestrator 530A may use a reverse hypercall to transmit state to the second hypervisor 561 and / or the second hypervisor 561 may use an intercept to request some services from the orchestrator 530A (e.g., characteristics of physical resources to which the second hypervisor 561 does not have access). In some embodiments, the orchestrator 530A shares some of the memory pages 511 with the second hypervisor 561 so that data can be efficiently communicated between the orchestrator 530A, the first hypervisor 520A, and the second hypervisor 561. In some embodiments, the orchestrator 530A issues a reverse hypercall to the second hypervisor that conforms to the hypercall's Application Binary Interface (ABI) to transfer the associated state.

[0049]

[0061] As discussed above, when service partition 560 is created and loaded, second hypervisor 561 must first obtain various system invariants, such as characteristics of hardware resources 510. In some embodiments, there is a strict binding between the characteristics permitted by first hypervisor 520A and the characteristics permitted by second hypervisor 561.

[0050]

[0062] Alternatively, in other embodiments, there is a loose coupling between the first hypervisor and the second hypervisor. The orchestrator 530A can query the first hypervisor via a hypercall to obtain characteristics and properties of hardware resources related to the initial configuration of the second hypervisor 561 and / or can query the hardware resources directly. For example, the second hypervisor 561 can attempt to read a CPUID or MSR value. However, within the virtual machine itself, the second hypervisor 561 may only have access to virtualized values ​​and not physical values ​​of the underlying physical computing system. In that case, the orchestrator 530A can invoke a hypercall to the first hypervisor 520A to obtain the corresponding physical values ​​of the corresponding physical resources. For example, the second hypervisor 561 may want to query processor support for XSAVE features and instructions. If the first hypervisor 520A does not support virtualization of XSAVE, the orchestrator 530A can query the characteristics of the underlying processor and determine that the processor supports XSAVE. The obtained query results may also be stored in shared memory pages 511 so that second hypervisor 561 has access to the query results. In some cases, second hypervisor 561 may not be able to obtain all characteristics of hardware resources 510. In some embodiments, such characteristics are left unknown during initialization and obtained later after devirtualization.

[0051]

[0063] In addition to obtaining system invariants, the runtime state of the first hypervisor 520A is also synchronized with the second hypervisor 561. The runtime state includes at least a dynamic system state and a logical software state. The dynamic system state includes the hardware architecture state visible to the guest, including but not limited to general purpose registers and control registers. The dynamic system state also includes the state of the guest VM, such as but not limited to the state of the virtual central processing unit (CPU), the second level page table (also known as nested page table), and the virtualization instruction set architecture ISA-specific hardware state, including the list of assigned hardware devices (e.g., network cards). Synchronizing the runtime state 512 of the first hypervisor includes at least synchronizing one or more second level memory page tables and / or one or more PFN databases to the second hypervisor 561. The second level memory table is a memory page table that maps the GPA of each guest partition to the HPA (e.g., the second level memory page table 450 of FIG. 4). The PFN databases that are synchronized may include a PFN database that tracks the physical memory of the host computing system (eg, PFN database 430).

[0052]

[0064] However, after the initial synchronization of the runtime state between the two hypervisors 520A and 561, and before the devirtualization of the second hypervisor 561, the guest partitions 540A, 550A are still running on the first hypervisor 520A, and those guest partitions 540A, 550A may continue to invoke hypercalls. When an incoming hypercall is serviced by the first hypervisor 520A, the state of the guest partition 540A or 550A that invoked the hypercall changes, and the previously synchronized state of the second hypervisor 561 is no longer accurate. To solve this problem, incoming hypercalls (before the devirtualization of the second hypervisor) also need to be recorded and synchronized with the second hypervisor 561.

[0053]

[0065] In some embodiments, the orchestrator 530A is also tasked with recording and synchronizing each incoming hypercall. For example, when the first hypervisor 520A receives and services a hypercall invoked by a guest partition 540A, 540B, the orchestrator 530A logs the hypercall and the actions that occurred during the servicing of the hypercall by the first hypervisor 520A. At the same time, the orchestrator 530A feeds the hypercall to the second hypervisor 561 via a reverse hypercall. In some embodiments, the instruction point of the second hypervisor 561 is in the hypercall dispatch loop, so that when the reverse hypercall is provided to the second hypervisor 561, the second hypervisor 561 processes it. Upon receiving the reverse hypercall, the second hypervisor 561 processes the reverse hypercall to switch the context of the guest partition 540A, 540B that invoked the hypercall and to reconstruct the necessary software state and / or quiesced hardware state. Upon completion of the reverse hypercall, the second hypervisor 561 signals the completion to the orchestrator 530A.

[0054]

[0066] In some cases, a series of hypercalls may be serviced within a short period of time, and orchestrator 530A may only send state associated with the last or relevant action of the series of hypercalls. In that case, second hypervisor 561 may only replay a partial log of actions, i.e., a condensed replay. This process may be repeated as many times as necessary until second hypervisor 561 is fully or at least substantially synchronized with first hypervisor 520A. Second hypervisor 561 is then de-virtualized from service partition 560 to replace first hypervisor 520A. De-virtualization involves first hypervisor 520A trampolining to the second hypervisor and relinquishing all physical hardware control to second hypervisor 561. In some embodiments, trampolining may be achieved by reusing existing control transfer mechanisms of computing system 500A for kernel soft reboot.

[0055]

[0067] In some embodiments, the second hypervisor 561 may undergo additional initialization to verify hardware state, re-initialize hardware, and / or initialize new hardware not previously programmed by the first hypervisor. In some embodiments, devirtualization may include the first hypervisor 520A freezing all guest partitions 540A, 550A, transmitting details of the final state to the second hypervisor 561, and migrating the guest partitions onto the second hypervisor. The second hypervisor then defrosts each of the guest partitions. When the second hypervisor 561 initiates devirtualization, the first hypervisor 520A is effectively terminated. No other system software, user software, and / or system firmware is permitted to run during devirtualization. However, DMAs programmed prior to devirtualization may continue to run and complete.

[0056]

[0068] For example, when a guest partition (e.g., child A 540A, 550A) is provided with access to a physical device, the guest partition can initiate a DMA using its GPA as the source or target of the DMA operation. The second level page table (which may be programmed into the input / output memory management unit (IOMMU) by the first hypervisor) translates the GPA to the HPA and provides the HPA to the DMA engine, in addition to performing the necessary permission checks. As mentioned above, the first hypervisor has transmitted / synchronized the architecture guest state as well as the architecture virtual state to the second hypervisor. The architecture virtual state includes, among other things, second level page tables for CPU, IOMMU, and / or device domain information. Thus, when the second hypervisor de-virtualizes and re-initializes the hardware, it carefully programs the hardware with the new page table that it built during the previous synchronization phase. Since the effective address translation and permissions of the new page table are the same for two different instances, all new translation requests from the DMA engine continue to run without loss of fidelity and can successfully use the same page table.

[0057]

[0069] Additionally, in some embodiments, the memory footprint of the first hypervisor 520A may be reclaimed quickly or slowly by a computing system under overall memory pressure.

[0058]

[0070] FIG. 5B illustrates another example of a computing system 500B in which hot hypervisor restart is enabled. Computing system 500B corresponds to computing system 100B of FIG. 1B, where one of the guest partitions (e.g., parent partition 110B, 530B) is more privileged than the remaining guest partitions (e.g., child partitions 120B, 540B, 550B). In some embodiments, parent partition 530B includes a host operating system and / or a virtualization service module 531B, either one of which may be configured to create and manage child partitions 540B and 550B and handle various system management functions and device drivers. Similar to the embodiment illustrated in FIG. 5A, a service partition 560 is created, a second hypervisor 561 is initialized within service partition 560, and finally, the first hypervisor 520B is replaced with the second hypervisor 561 to complete hot hypervisor restart.

[0059]

[0071] In some embodiments, because parent partition 530B has a higher level of privilege than child partitions 540B, 550B in computing system 500B, orchestrator 532B may be implemented within parent partition 530B as part of its host operating system or virtualization service module 531B. Orchestrator 532B functions similarly to orchestrator 530B to coordinate communications between first hypervisor 520B and second hypervisor 561. Elements 541B, 551B of FIG. 5B are similar to elements 541A, 551A of FIG. 5A and will not be discussed further.

[0060]

[0072] Additionally, in some embodiments, a hot restart of the hypervisor 520A, 520B may also be accompanied by a kernel soft reboot or reset of the guest partitions 540A, 550A, 530B, 540B, 550B. In a kernel soft reboot, the guest partitions 540A, 550A, 530B, 540B, or 550B may be recreated and reinitialized as new partitions, and the runtime state of the corresponding guest partitions is synchronized with the new partition. Once all runtime states of the guest partitions are synchronized with the new partition, the new partition may replace the corresponding guest partition to complete a kernel soft reboot of the corresponding guest partition. In some embodiments, only the parent partition 530B is hot restarted in conjunction with the hot restart of the hypervisor 520B. Alternatively or additionally, each of the guest partitions 540A, 550A, and / or child partitions 540B, 550B is hot restarted in conjunction with the hot restart of the hypervisor 520A, 520B.

[0061]

[0073] Some embodiments restart VM-related components in parent partition 530B without restarting the host operating system in parent partition 530B. For example, some embodiments restart virtualization services module 531B in conjunction with a hot restart of the hypervisor without restarting host operating system 113B. In this manner, the hypervisor can be upgraded and restarted along with its operating system level management components without restarting host operating system 113B.

[0062]

[0074] The hot hypervisor restart described herein, unlike conventional hypervisor restarts, significantly reduces the disruption time imposed on running guest VMs. Unlike a typical hypervisor restart, which can take several minutes depending on the number of VMs hosted and the amount of hardware resources being managed, the second hypervisor 561 described herein is initialized and synchronized while the first hypervisor 520A or 520B is still running, thus causing only a short (e.g., less than a second or a few seconds) freeze period in each running guest partition that may not even be noticeable to the user.

[0063]

[0075] The following discussion next refers to certain methods and method actions that may be performed. Although the method actions may be discussed in a certain order or may be illustrated in a flow diagram as occurring in a particular order, no particular ordering is required unless otherwise specified or required because an action is dependent on another action being completed before an action can be performed.

[0064]

[0076] FIG. 6A illustrates a flow diagram of an example of a method 600 for hot restart of a hypervisor. The method 600 is implemented on a computing system that may correspond to the computing system 100A, 100B, or 500A, 500B. The method 600 includes executing a first hypervisor (610). In response, the first hypervisor creates one or more guest partitions (620), each of which may host a guest operating system. The purpose of the hot restart of the hypervisor is to replace the first hypervisor with a new hypervisor. When the hot restart of the hypervisor is performed, the computing system creates a service partition (630) and initializes a second hypervisor in the service partition (640). At least a portion of the runtime state of the first hypervisor is then synchronized with the second hypervisor (650). Once the synchronization is complete or nearly complete, the computing system devirtualizes (660) the second hypervisor to replace the first hypervisor with the second hypervisor.

[0065]

[0077] FIG. 6B illustrates a flow diagram of an example method 640 for initializing a service partition, corresponding to step 640 of FIG. 6A. The method 640 includes generating (641) a loader block for a second hypervisor by a first hypervisor. The loader block is a logical construct that describes some of the static system characteristics and resources for initialization. The method 640 also includes obtaining (642) one or more system invariants and sharing (643) the one or more system invariants with the second hypervisor. The second hypervisor is then initialized based on (644) the one or more system invariants. In some embodiments, the system invariants include one or more characteristics of the hardware resources of the computing system (645). In some embodiments, there is a tight coupling between the first hypervisor and the second hypervisor (646). In some embodiments, there is a loose coupling between the first hypervisor and the second hypervisor. In that case, one or more characteristics of the hardware resource may be obtained from the first hypervisor via a hypercall (647), or may be obtained directly from the hardware resource via a system call (648).

[0066]

[0078] FIG 6C illustrates a flow diagram of an example of a method 650 for synchronizing a portion of a runtime state of a first hypervisor with a second hypervisor, corresponding to step 650 of FIG 6A. Method 650 may be performed by orchestrator 530A, 532B of FIG 5A or FIG 5B. Method 650 includes collecting (651) a runtime state of the first hypervisor. Method 650 also includes sharing (652) the runtime state with the second hypervisor, which may be performed by the orchestrator 530A, 532B via a reverse hypercall, which may or may not further rely on shared memory pages to facilitate signaling and / or message / data passing. The shared runtime state of the first hypervisor is then replicated (653) by the second hypervisor. Before the second hypervisor is devirtualized, guest partitions running on the first hypervisor continue to operate and can still make hypercalls, and the runtime state collected quickly becomes inaccurate. Thus, this process may be repeated several times until the second hypervisor and the first hypervisor are fully or nearly synchronized.

[0067]

[0079] FIG 6D illustrates a flow diagram of an example of a method 651 for collecting runtime state of a first hypervisor, corresponding to step 651 of FIG 6C. The method 651 includes collecting current data related to one or more second level memory page tables and one or more PFNs (651-A). The method 651 also includes collecting data related to an incoming hypercall (651-B). Specifically, when a hypercall is invoked by a guest partition, the first hypervisor receives (651-C) and services (651-D) the received hypercall. The computing system (e.g., orchestrator 530A or 532B) then logs (651-E) the received hypercall and the actions that occurred during the servicing of the hypercall.

[0068]

[0080] Figure 6E illustrates a flow diagram of one example of a method 652 for sharing a runtime state of a first hypervisor with a second hypervisor, corresponding to step 652 of Figure 6C. The method 652 includes feeding the second hypervisor one or more second level page tables and one or more PFNs (652-A). The method 652 also includes feeding the second hypervisor an incoming hypercall (652-B) and feeding the second hypervisor a log of actions that occurred during servicing of the hypercall (by the first hypervisor) (652-C).

[0069]

[0081]

[0031] Figure 6F illustrates a flow diagram of an example method 653 for replicating the runtime state of a first hypervisor with a second hypervisor, corresponding to step 653 of Figure 6C. The method 653 includes receiving a reverse hypercall from an orchestrator in the second hypervisor (653-A). The second hypervisor then switches the context of the guest partition that invoked the hypercall (653-B) and processes the reverse hypercall to reconstruct the necessary software state and / or quiesced hardware state (653-C). Upon completion of processing of the reverse hypercall, the second hypervisor informs the orchestrator (653-D).

[0070]

[0082] FIG 6G illustrates a flow diagram of an example of a method 660 for devirtualizing a second hypervisor, corresponding to step 660 of FIG 6A. The method 660 includes freezing each guest partition currently running on the first hypervisor (661). The final state of the first hypervisor is then transmitted to the second hypervisor (662). Each of the guest partitions is then switched onto the second hypervisor (663), which defrosts each of the guest partitions (664). Finally, the first hypervisor is terminated (665).

[0071]

[0083] It should be noted that even though the example embodiments above are implemented within a dedicated partition (e.g., a service partition) for servicing the hypervisor, the overall technique of pre-initializing data structures to save time and / or migrating / synchronizing runtime state may be used for any part of any VM.

[0072]

[0084] Finally, because the principles described herein may be implemented in the context of a computing system (e.g., computing systems 100A, 100B of FIGS. 1A, 1B, and / or 500A, 500B of FIGS. 5A or 5B), some introductory discussion of computing systems is provided with respect to FIG. 7.

[0073]

[0085] Currently, computing systems increasingly take a wide variety of forms. Computing systems can be devices not traditionally considered computing systems, such as handheld devices, appliances, laptop computers, desktop computers, mainframes, distributed computing systems, data centers, and even wearables (e.g., glasses). In this description and claims, the term "computing system" is broadly defined to include any device or system (or combination thereof) that includes at least one physical and tangible processor and physical and tangible memory that can have computer-executable instructions thereon that can be executed by the processor. The memory can take any form and can depend on the nature and type of the computing system. A computing system can be distributed across a network environment and can include multiple constituent computing systems.

[0074]

[0086] As shown in FIG. 7, in its most basic configuration, a computing system 700 typically includes at least one hardware processing unit 702 and memory 704. The processing unit 702 may include a general-purpose processor, but may also include a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), or any other special purpose circuitry. The memory 704 may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" is also used herein to refer to non-volatile mass storage, such as a physical storage medium. When a computing system is distributed, the processing, memory, and / or storage functions may also be distributed.

[0075]

[0087] Computing system 700 also has a number of structures thereon, often referred to as "executable components." For example, memory 704 of computing system 700 is shown as including executable component 706. The term "executable component" refers to a structure familiar to those skilled in the computing arts as being a structure that may be software, hardware, or a combination thereof. Those skilled in the art will appreciate that when implemented, for example, by software, the executable component structure may include software objects, routines, methods, etc., that may be executed on the computing system, whether such executable component resides in the heap of the computing system or on a computer-readable storage medium.

[0076]

[0088] In that case, one skilled in the art will recognize that the structure of the executable component resides on a computer-readable medium such that, when interpreted by one or more processors (e.g., by processor threads) of a computing system, it causes the computing system to perform a function. Such structure may be directly computer readable by a processor (as would be the case if the executable component were binary). Alternatively, the structure may be configured to be interpretable and / or compiled (in a single or multiple stages) to generate such a binary that is directly interpretable by a processor. When the term "executable component" is used, such an understanding of examples of the structure of an executable component is well within the understanding of one skilled in the computing arts.

[0077]

[0089] The term "executable component" is further understood by those skilled in the art to include structures such as hard-coded or hard-wired logic gates that are implemented exclusively or nearly exclusively in hardware, such as in field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), or any other dedicated circuitry. Thus, the term "executable component" is a term for structures that are well understood by those skilled in the computing arts, whether implemented in software, hardware, or a combination. In this description, terms such as "component," "agent," "manager," "service," "engine," "module," "virtual machine," and the like may also be used. As used in this description and in this instance, these terms (whether or not presented with a qualifying clause) are intended to be synonymous with the term "executable component," and thus also include structures that are well understood by those skilled in the computing arts.

[0078]

[0090] In the above description, embodiments have been described with respect to actions performed by one or more computing systems. When such actions are implemented by software, one or more processors (of the associated computing systems performing the actions) direct the operation of the computing systems in response to executing the computer-executable instructions that constitute the executable components. For example, such computer-executable instructions may be embodied by one or more computer-readable media forming a computer program product. One example of such an operation includes the manipulation of data. When such actions are implemented exclusively or nearly exclusively in hardware, such as in an FPGA or ASIC, the computer-executable instructions may be hard-coded or hard-wired logic gates. The computer-executable instructions (and the data to be manipulated) may be stored in memory 704 of the computing system 700. The computing system 700 may also include a communication channel 708 that allows the computing system 700 to communicate with other computing systems, for example over a network 710.

[0079]

[0091] Although not all computing systems require a user interface, in some embodiments, the computing system 700 includes a user interface system 712 for use in interfacing with a user. The user interface system 712 may include an output mechanism 712A as well as an input mechanism 712B. The principles described herein are not limited to the exact output mechanism 712A or input mechanism 712B as they depend on the nature of the device. However, the output mechanism 712A may include, for example, a speaker, a display, a haptic output, a hologram, etc. Examples of input mechanism 712B may include, for example, a microphone, a touch screen, a hologram, a camera, a keyboard, a mouse or other pointer input, any type of sensor, etc.

[0080]

[0092] As discussed in more detail below, the embodiments described herein may include or utilize special purpose or general purpose computing systems including computer hardware such as one or more processors and system memory. The embodiments described herein also include physical and other computer readable media for carrying or storing computer executable instructions and / or data structures. Such computer readable media may be any available media that can be accessed by a general purpose or special purpose computing system. Computer readable media that store computer executable instructions are physical storage media. Computer readable media that carry computer executable instructions are transmission media. Thus, by way of example, and not limitation, embodiments of the present invention may include at least two distinctly different kinds of computer readable media: storage media and transmission media.

[0081]

[0093] Computer-readable storage media can be used to store desired program code means in the form of computer-executable instructions or data structures and include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other physical and tangible storage medium accessible by a general-purpose or special-purpose computing system.

[0082]

[0094] A "network" is defined as one or more data links that enable the transport of electronic data between computing systems and / or modules and / or other electronic devices. When information is transferred or provided to a computing system over a network or another communications connection (hardwired, wireless, or a combination of hardwired or wireless), the computing system properly regards the connection as a transmission medium. A transmission medium may be used to carry the desired program code means in the form of computer-executable instructions or data structures and may include a network and / or data links accessible by a general-purpose or special-purpose computing system. Combinations of the above are also intended to be included within the scope of computer-readable media.

[0083]

[0095] Furthermore, program code means in the form of computer-executable instructions or data structures may be automatically transferred from a transmission medium to a storage medium (or vice versa) upon arrival at various computing system components. For example, computer-executable instructions or data structures received over a network or data link may be buffered in RAM in a network interface module (e.g., a "NIC") and then gradually transferred to the computing system's RAM and / or to a less volatile storage medium in the computing system. Thus, it should be understood that storage media may be included in computing system components that also (or primarily) utilize transmission media.

[0084]

[0096] Computer-executable instructions include instructions and data that, when executed, for example, on a processor, cause a general-purpose computing system, special-purpose computing system, or special-purpose processing device to perform a certain function or group of functions. Alternatively, or in addition, computer-executable instructions may configure a computing system to perform a certain function or group of functions. Computer-executable instructions may be binaries or even instructions that have undergone some transformation (e.g., compilation) before being executed directly by a processor, for example intermediate format instructions such as assembly language or even source code.

[0085]

[0097] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the described features or acts described above. Rather, the described features and acts are disclosed as example forms of implementing the claims.

[0086]

[0098] Those skilled in the art will appreciate that the present invention may be practiced in networked computing environments having many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, cell phones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses), etc. The present invention may also be practiced in distributed system environments in which both local and remote computing systems perform tasks that are linked through a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links). In a distributed system environment, program modules may be located in both local and remote memory storage devices.

[0087]

[0099] Those skilled in the art will also appreciate that the present invention may be practiced within a cloud computing environment. A cloud computing environment may be distributed, but that is not required. If distributed, a cloud computing environment may be distributed internationally within an organization and / or may have components maintained across multiple organizations. In this description and the appended claims, "cloud computing" is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of "cloud computing" is not limited to any of the many other benefits that can be derived from such a model when properly deployed.

[0088]

[0100] The remaining figures may discuss various computing systems that may correspond to the computing system 700 described above. The computing systems of the remaining figures include various components or functional blocks that may implement various embodiments disclosed herein that are described. The various components or functional blocks may be implemented on a local computing system or on a distributed computing system that includes elements that are in the cloud or implements aspects of cloud computing. The various components or functional blocks may be implemented as software, hardware, or a combination of software and hardware. The computing systems of the remaining figures may include more or fewer components than shown, and some of the components may be combined depending on the circumstances. Although not necessarily shown, the various components of the computing system may access and / or utilize processors and memory, such as processor 702 and memory 704, as necessary to perform their various functions.

[0089]

[0101] In the processes and methods disclosed herein, the operations performed within the processes and methods may be implemented in differing orders. Moreover, the operations outlined are presented by way of example only, and some of the operations may be optional, combined into fewer steps and operations, supplemented with additional operations, or expanded into additional operations without departing from the essence of the disclosed embodiments.

[0090]

[0102] The present invention may be embodied in other specific forms without departing from the spirit or characteristics of the present invention. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is therefore indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced therein.

Claims

1. A computing system comprising: one or more processors; When executed by the one or more processors, executing a first hypervisor that creates one or more guest partitions; Creating a service partition, initializing a second hypervisor in the service partition with a static system state of the first hypervisor; Synchronizing at least a portion of a runtime state of the first hypervisor with the second hypervisor, wherein synchronizing at least a portion of the runtime state of the first hypervisor with the second hypervisor comprises: intercepting a request for a system invariant from the first hypervisor on a shared memory; and issuing a reverse hypercall to the second hypervisor to transmit the system invariants to the second hypervisor via the shared memory; Synchronizing, including devirtualizing the second hypervisor from the service partition to replace the first hypervisor; one or more computer-readable media having stored thereon computer-executable instructions configured to cause the computing system to A computing system comprising:

2. 2. The computing system of claim 1, wherein initializing the second hypervisor in the service partition further comprises granting read-only unique access to certain portions of the one or more computer-readable media of the computing system.

3. 2. The computing system of claim 1, wherein the runtime state includes at least one of: (1) one or more second level memory page tables that map guest physical memory of the guest partition to host physical memory of the computing system; (2) one or more page frame number databases; or (3) a list of physical devices attached to guest partitions and an allocation of the list of physical devices.

4. at least one of the one or more guest partitions includes a privileged parent partition; the privileged parent partition runs a host operating system or a virtualization service module; the host operating system or the virtualization service module includes an orchestrator configured to coordinate the initialization and synchronization of the second hypervisor; the first hypervisor enabling the orchestrator to intercept certain requests from the second hypervisor to the first hypervisor; The computing system of claim 1 .

5. 5. The computing system of claim 4, wherein when the orchestrator intercepts a request from the second hypervisor to the first hypervisor, the orchestrator issues a reverse hypercall to the second hypervisor and shares a portion of a memory page with the second hypervisor for communication purposes.

6. said initializing said second hypervisor generating, by the first hypervisor, a loader block for the second hypervisor, the loader block including a logical construct describing at least a portion of the static system state for initialization; obtaining, by the orchestrator, one or more system invariants of the computing system; sharing the one or more system invariants with the second hypervisor; and initializing the second hypervisor based on the one or more system invariants. The computing system of claim 5 .

7. The computing system of claim 6 , wherein the one or more system invariants include one or more features supported by one or more hardware resources of the computing system.

8. 8. The computing system of claim 7, wherein the orchestrator obtains the one or more features from at least one of: (1) a hypercall from the first hypervisor; or (2) directly from hardware resources.

9. The computing system of claim 8 , wherein the orchestrator obtains at least one of the characteristics through a binary interface that discovers or queries hardware capabilities.

10. The method of claim 1, further comprising: aggregating a runtime state of the first hypervisor by the orchestrator; sharing, by the orchestrator, the runtime state with the second hypervisor over a reverse hypercall; and causing the second hypervisor to replicate the runtime state; The computing system of claim 6 , comprising:

11. The method of claim 10, further comprising: receiving, by the first hypervisor, a hypercall invoked by a guest partition; servicing the hypercall by the first hypervisor; and recording, by the orchestrator, a log of the hypercall and actions that occurred during servicing of the hypercall by the first hypervisor; and sharing the runtime state comprises: feeding, by the orchestrator, the hypercall to the second hypervisor; and feeding, by the orchestrator, a log of the actions to the second hypervisor; The computing system of claim 10 , comprising:

12. The method of replicating the runtime state, comprising: receiving a reverse hypercall from the orchestrator in the second hypervisor; switching the context of the guest partition that invoked the reverse hypercall; processing the inverse hypercall in said second hypervisor to reconstruct the necessary software state or quiesced hardware state; and Informing the orchestrator of completion of the reverse hypercall. The computing system of claim 10 , comprising:

13. A parent partition comprising a host operating system and a virtualization service module; the virtualization services module is configured to provide virtualization services to at least one other guest partition; The computing system is further configured to hot restart the virtualization services module without restarting the host operating system. The computing system of claim 3.

14. Devirtualizing the second hypervisor comprises: transferring control of the physical hardware of the computing system from the first hypervisor to the second hypervisor; freezing each of the one or more guest partitions; communicating a final state of the first hypervisor to the second hypervisor; switching each guest partition to the second hypervisor; and Decompress each of the guest partitions, The computing system of claim 1 , comprising:

15. The computing system of claim 1, further configured to hot restart at least one of the one or more guest partitions.

16. A method for hot restart of a hypervisor of a host, implemented by a computing system, comprising: executing a first hypervisor that creates one or more guest partitions; Creating a service partition, initializing a second hypervisor in the service partition with a static system state of the first hypervisor; Synchronizing at least a portion of a runtime state of the first hypervisor with the second hypervisor, wherein synchronizing at least a portion of the runtime state of the first hypervisor with the second hypervisor comprises: intercepting a request for a system invariant from the first hypervisor on a shared memory; and issuing a reverse hypercall to the second hypervisor to transmit the system invariants to the second hypervisor via the shared memory; Synchronizing, including devirtualizing the second hypervisor from the service partition to replace the first hypervisor; A method comprising:

17. The method of claim 16, wherein the runtime state includes at least one of: (1) one or more second level memory page tables that map guest physical memory of the guest partition to host physical memory of the computing system; (2) one or more page frame number databases; or (3) a list of physical devices attached to a guest partition and an allocation of the list of physical devices.

18. The method of claim 17, wherein at least one of the one or more guest partitions includes a privileged parent partition; the privileged parent partition runs a host operating system or a virtualization service module; the host operating system or the virtualization service module includes an orchestrator configured to coordinate the initialization and synchronization of the second hypervisor; 17. The method of claim 16.

19. A computer program comprising computer-executable instructions that, when executed by one or more processors of a computing system, executing a first hypervisor that creates one or more guest partitions; Creating a service partition, initializing a second hypervisor in the service partition with a static system state of the first hypervisor; Synchronizing at least a portion of a runtime state of the first hypervisor with the second hypervisor, wherein synchronizing at least a portion of the runtime state of the first hypervisor with the second hypervisor comprises: intercepting a request for a system invariant from the first hypervisor on a shared memory; and issuing a reverse hypercall to the second hypervisor to transmit the system invariants to the second hypervisor via the shared memory; Synchronizing, including devirtualizing the second hypervisor from the service partition to replace the first hypervisor; A computer program product configured to cause the computing system to:

20. Devirtualizing the second hypervisor comprises: transferring control of the physical hardware of the computing system from the first hypervisor to the second hypervisor; freezing each of the one or more guest partitions; communicating a final state of the first hypervisor to the second hypervisor; switching each guest partition to the second hypervisor; and Decompress each of the guest partitions, 17. The method of claim 16, comprising:

Images