Hypervisor Hot Restart
The hypervisor hot restart method initializes a second hypervisor within a service partition, synchronizing states and devirtualizing it to replace the first hypervisor, thereby minimizing downtime and ensuring service continuity during hypervisor restarts.
Patent Information
- Application Number
- KR1020227037194
- Authority / Receiving Office
- KR · KR
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2020-04-15
- Filing Date
- 2021-03-15
- Publication Date
- 2026-11-06
- Estimated Expiration
- 2041-03-15
AI Technical Summary
When a hypervisor is reset or restarted, it interrupts all virtual machines running on top of it, significantly impacting the workload and services, leading to downtime and service disruption.
A hypervisor hot restart method that initializes a second hypervisor within a service partition with an identity mapping between guest and host physical address spaces, synchronizes the state of the first hypervisor with the second hypervisor, and devirtualizes the second hypervisor to replace the first, maintaining minimal interference to guest partitions.
Minimizes downtime and ensures continuity of services by smoothly transitioning from a first hypervisor to a second hypervisor, allowing guest partitions to operate uninterrupted during the restart process.
Smart Images

Figure R1020227037194_ABST
Abstract
Description
Background Technology
[0001] Virtualization in computing often refers to the abstraction of physical components into logical objects. A virtual machine (VM) can virtualize hardware resources, including processors, memory, storage, and network connections, and present virtualized resources to an operating host. The virtualization process of a VM comprises at least two parts: (1) the mapping of virtual resources or states, e.g., registers, memory, or files, to actual resources of underlying physical hardware, and (2) the use of machine instructions and / or system calls to perform actions specified by virtual machine instructions and / or system calls, e.g., by emulation of the virtual machine application binary interface (ABI) or instruction set architecture (ISA) interface.
[0002] A hypervisor is a software layer that provides the environment in which virtual machines operate (i.e., virtualized hardware partitions). The hypervisor sits between the physical resources on the physical hardware computing system and the VMs. Without a hypervisor, the operating system communicates directly with the underlying hardware; that is, disk operations would go directly to the disk subsystem, and memory calls would be fetched directly from physical memory. When multiple operating systems for multiple VMs are running simultaneously on a single machine, the hypervisor manages the interaction between each VM and the shared hardware to prevent simultaneous control of the shared hardware by multiple VMs.
[0003] If the hypervisor is reset or restarted (e.g., due to a software upgrade), all VMs running on top of the hypervisor are interrupted (e.g., restart interruption), which can significantly impact the workload and services running on the VMs.
[0004] The subject matter claimed herein is not limited to embodiments that address any disadvantages or operate only in environments as described above. Rather, this background art is provided only to illustrate one exemplary technical area in which some of the embodiments described herein may be practiced. Prior art literature
[65535] Spoorti Doddamani et al., "Fast and live hypervisor replacement," Virtual Execution Environments (VEE 2019), 2019.04.14.
[0005] This summary is provided to introduce, in a simplified form, the selection of concepts further explained in the detailed description below. This summary is not intended to identify the principal or essential features of the claimed subject matter, nor is it intended to be used as an aid in determining the scope of the claimed subject matter.
[0006] The embodiments described herein relate to a hypervisor hot restart that replaces a first hypervisor with a second hypervisor with minimal interference to a currently running guest partition. The embodiments described herein are implemented in a computing system. First, the computing system runs a first hypervisor on the computing system. The first hypervisor is configured to create one or more guest partitions. During a hypervisor soft restart, a service partition is created. The second hypervisor is initialized within the service partition. The service partition is created with an identity mapping between its guest physical address (GPA) space and its host physical address (HPA) space. Optionally, additional privileges may be granted to the existing partition / virtual machine to facilitate the initialization. Any execution environment capable of meeting the boot and security requirements of the platform may create and / or initialize the service partition. For example, a component of the trusted computing base (TCB) of a computing system, such as a first hypervisor, may receive a task to create or initialize a second hypervisor. During initialization, at least some of the state of the first hypervisor is initialized to the second hypervisor. The state of the first hypervisor includes, but is not limited to, (1) static system state, (2) dynamic system state, and (3) logical software state. The static system state includes, but is not limited to, the system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes, but is not limited to, a guest visible hardware architecture state, which includes, general-purpose registers and mode control registers.Dynamic system state also includes virtualization instruction set architecture (ISA) specific hardware state, including but not limited to the state of guest VMs such as virtual central processing unit (CPU) state, second-level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network cards). Logical software state includes, but is not limited to, a page frame number (PFN) database. Since dynamic system state and logical software state can continuously change during runtime, these states are also referred to as runtime state. Finally, the second hypervisor is devirtualized from the service partition to replace the first hypervisor, and the state of each VM is restored by the second hypervisor.
[0007] In some embodiments, at least one of one or more guest partitions includes a privileged parent partition, and the parent partition runs a host operating system that includes an orchestrator configured to orchestrate the initialization and synchronization of the second hypervisor. In some embodiments, the first hypervisor allows and supports the orchestrator to register and complete specific requests (e.g., hypercall intercepts, register intercepts) received from the second hypervisor. Since these requests (transmitted from the second hypervisor to the first hypervisor) are intercepted (e.g., registered and completed) by the orchestrator, these requests are also referred to as “intercepts.” In some embodiments, in response to receiving intercepts, the orchestrator issues an “inverse hypercall” to the second hypervisor that follows the Hypercall Application Binary Interface (ABI) to migrate the relevant state. A hypercall is a call from a guest partition to a hypervisor, and a “reverse hypercall” is a call from a privileged software or hardware component (e.g., an orchestrator) to a second hypervisor in a service partition. In some embodiments, the orchestrator may share a portion of memory pages with the first hypervisor and the second hypervisor for communication purposes (e.g., hypercalls, intercepts, and / or reverse hypercalls).
[0008] Exemplary embodiments are implemented in a special partition (e.g., a service partition) for servicing the hypervisor, but the entire technique of pre-initializing data structures in the virtual machine and / or migrating / synchronizing the runtime state within the VM / partition can be used for any part of the OS of any VM to reduce downtime and provide better uptime and continuity for service.
[0009] Additional features and benefits will be described in the following description, will become partially apparent from the description, or may become known through the practice of the teachings herein. Features and benefits of the invention may be realized and obtained by the tools and combinations specifically mentioned in the appended claims. Features of the invention will become more fully apparent from the following description and the appended claims, or may become known through the practice of the invention as described below. Brief explanation of the drawing
[0010] To explain how the above-described and other advantages and features can be obtained, a more specific description of the subject briefly described above will be provided with reference to specific embodiments illustrated in the accompanying drawings. It is understood that these drawings illustrate only general embodiments and should not be construed as limiting in scope, and embodiments will be described and explained with additional specificity and detail through the use of the accompanying drawings. FIGS. 1a and FIGS. 1b illustrate an exemplary computing system comprising a hypervisor that hosts one or more virtual machines. Figure 2 illustrates a schematic diagram of isomorphism between a guest partition and a host computing system. FIGS. 3a and FIGS. 3b illustrate exemplary embodiments of VM state management. FIG. 4 illustrates an exemplary embodiment (400) of memory virtualization using multiple-stage memory page tables. FIGS. 5A and FIGS. 5B illustrate an exemplary computing system in which hypervisor hot restart is enabled. Figure 6a illustrates a flowchart of an exemplary method for hypervisor hot restart. Figure 6b illustrates a flowchart of an exemplary method for initializing a service partition. FIG. 6c illustrates a flowchart of an exemplary method for synchronizing a portion of the runtime state of a first hypervisor with a second hypervisor. FIG. 6d illustrates a flowchart of an exemplary method for collecting the runtime state of a first hypervisor. FIG. 6e illustrates a flowchart of an exemplary method for sharing the runtime state of a first hypervisor with a second hypervisor. FIG. 6f illustrates a flowchart of an exemplary method for replicating the runtime state of a first hypervisor to a second hypervisor. FIG. 6g illustrates a flowchart of an exemplary method for devirtualizing a second hypervisor. FIG. 7 illustrates an exemplary computing system in which the principles described herein can be utilized. Specific details for implementing the invention
[0011] The embodiments described herein relate to a hypervisor hot restart that replaces a first hypervisor with a second hypervisor with minimal interference to a currently running guest partition. The embodiments described herein are implemented in a computing system. First, the computing system runs a first hypervisor on the computing system. The first hypervisor is configured to create one or more virtual machines / partitions, each hosting a guest operating system. During a hypervisor soft restart, a service partition with an identity-mapped second-level page table is created. Subsequently, the second hypervisor is initialized within the service partition and synchronized with at least a portion of the state of the first hypervisor. Any execution environment capable of meeting the boot and security requirements of the platform may create and / or initialize the second hypervisor. For example, a component of a Trusted Computing Base (TCB), such as the first hypervisor, may be tasked with creating and / or initializing the second hypervisor.
[0012] In some embodiments, the state of the first hypervisor includes, but is not limited to, (1) static system state, (2) dynamic system state, and (3) logical software state. The static system state includes, but is not limited to, the system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes, but is not limited to, general-purpose registers and mode control registers, the state of the guest visible hardware architecture. The dynamic system state also includes, but is not limited to, the state of the guest VM, such as the state of the virtual central processing unit (CPU), the second-level page table (also known as a nested page table), and the list of assigned hardware devices (e.g., network card). The logical software state includes, but is not limited to, a page frame number (PFN) database. Since the dynamic system state and the logical software state may continue to change during runtime, these states are also referred to as runtime states. Finally, the second hypervisor is devirtualized from the service partition and replaces the first hypervisor, and the state of each VM is restored by the second hypervisor.
[0013] In some embodiments, at least one of the one or more guest partitions includes a privileged parent partition, and the parent partition runs a host operating system that includes an orchestrator configured to coordinate the initialization and synchronization of the second hypervisor. In some embodiments, the first hypervisor is part of the system's Trusted Computing Base (TCB), and the orchestrator runs in a Trusted Execution Environment (TEE) to maintain the trust of inputs generated for the second hypervisor. In some embodiments, the first hypervisor allows and supports the orchestrator to register and complete specific requests received from the second hypervisor. Since these requests (transmitted from the second hypervisor to the first hypervisor) are intercepted by the orchestrator (e.g., registered and completed), these requests are also referred to as “intercepts.” In some embodiments, in response to receiving intercepts, the orchestrator issues a “reverse hypercall” to the second hypervisor that follows the Hypercall Application Binary Interface (ABI) to migrate the relevant state. A hypercall is generally a call that requests a suitable privileged operation from a guest partition to a hypervisor. The transition from the guest partition context to the hypervisor context is achieved using platform-specific instructions. Once the privileged operation is complete, control returns from the hypervisor context to the guest context, as opposed to using platform-specific instructions / mechanisms. A “reverse hypercall” uses the hypercall ABI to request an operation from a privileged component (e.g., an orchestrator) to the hypervisor in a service partition. The transition from the orchestrator to a second hypervisor context is achieved by authorizing the execution of the virtual processor in the service partition using registers formatted as required by the hypercall ABI.When the reverse hypercall is completed, the second hypervisor generates a very specific intercept that causes an intentional context switch from the second hypervisor of the service partition back to the orchestrator. In some embodiments, the orchestrator may share one or more memory pages with the first hypervisor and the second hypervisor for communication purposes (e.g., hypercall, intercept, and / or reverse hypercall).
[0014] In some embodiments, initializing the second hypervisor involves the first hypervisor generating a loader block for the second hypervisor. The loader block includes a logical configuration describing a portion of the static system state for initialization. The static system state includes various system invariants such as system topology, memory map, and layout. The memory map includes an identity map of substantially all system memory. In some embodiments, the identity map includes at least the memory visible to the first hypervisor being replaced. For example, in some cases, both the first (old) and second (new) hypervisors are aware of all the memory in the system. In some cases, for example, when some RAM containing errors needs to be taken offline, or when the second (new) hypervisor needs to be aware of new memory that is hot-added during a hot restart, at least one of the first and / or second hypervisors is not aware of all the memory. The orchestrator obtains one or more system invariants of a computing system, comprising at least one feature supported by a hardware resource of the computing system, (1) directly from the hardware or system resource(s) where a particular resource is not virtualized by the hypervisor, and / or (2) from the first hypervisor where the orchestrator's access to the underlying physical hardware resource is virtualized by the first hypervisor. In some embodiments, the orchestrator migrates the relevant system invariants via a reverse hypercall. The second hypervisor is then initialized based on one or more shared system invariants. In some embodiments, the second hypervisor initialization may also include providing the second hypervisor read-only specific access to a particular physical resource.
[0015] After the second hypervisor is initialized to a static system state in the service partition, the computing system subsequently migrates the runtime state of the first hypervisor to the second hypervisor and maintains synchronization. The runtime state includes at least dynamic system state and logical software state. Synchronizing the runtime state of the first hypervisor may include an orchestrator collecting the runtime state of the first hypervisor and sharing / migrating the runtime state with the second hypervisor via a reverse hypercall. As mentioned above, the dynamic system state includes guest visible hardware architecture state, including but not limited to general-purpose registers and mode control registers. The dynamic system state also includes virtualization instruction set architecture (ISA) specific hardware state, including but not limited to the state of the guest VM, such as virtual central processing unit (CPU) state, second-level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network cards). The logical software state includes but not limited to a page frame number (PFN) database. The second hypervisor then replicates the shared second-level memory page table and / or PFN database, and leaves them dormant until the second hypervisor finally devirtualizes them.
[0016] Additionally, during this synchronization, the guest partition running on top of the first hypervisor continues to operate and issues hypercalls to the first hypervisor. Therefore, after the initial synchronization of the computationally intensive state, additional incoming hypercalls to the first hypervisor that affect the previously transmitted state may also need to be synchronized to the second hypervisor. In some embodiments, these state changes are also transmitted using reverse hypercalls.
[0017] When the first hypervisor and the second hypervisor are fully or substantially synchronized, the second hypervisor is subsequently devirtualized. In some embodiments, devirtualization includes the first hypervisor “trampolining” to the second hypervisor and ceding all physical hardware control to the second hypervisor. “Trampoline” sometimes refers to a memory location that holds an address pointing to an interrupt service routine, I / O routine, etc. Here, “trampolining” refers to the transfer of control of all physical hardware from the first hypervisor to the second hypervisor. In some embodiments, trampolineing may be achieved by reusing the existing control transfer mechanism of the computing system for a Kernel Soft-Reboot. In some embodiments, other system or user software, excluding system firmware, is not permitted to execute during the trampoline process. However, direct memory access (DMA) programmed prior to devirtualization may continue to proceed and be completed. In some embodiments, the second hypervisor may undergo additional initialization, verify the hardware state, re-initialize the hardware, and / or initialize new hardware that has not been previously programmed by the first hypervisor. Devirtualization may also include freezing each guest partition currently running on top of the first hypervisor and transmitting the final state of the first hypervisor to the second hypervisor. Afterward, each guest partition is switched to the second hypervisor. The second hypervisor then thaws each guest partition. The memory footprint of the first hypervisor may be reclaimed eagerly or leisurely by the computing system depending on the total memory pressure.
[0018] Since the embodiments described herein are implemented in a VM environment capable of simultaneously supporting multiple guest partitions running their own operating systems and associated applications, some introductory discussion of virtualization and hypervisors will be explained with respect to FIGS. 1 to 4.
[0019] Hypervisor-based virtualization enables multiple guest operating systems (running in child partitions), as well as a privileged host operating system (often running in a "parent" partition), to simultaneously share access to the hardware of a single computing system, creating the illusion that each operating system has access to the entire set of system resources. To create this illusion, in some embodiments, the hypervisor in the computing system creates multiple partitions that function as virtual hardware machines (i.e., VMs), each running its own operating system and associated applications. Each operating system controls and manages a set of virtualized hardware resources.
[0020] FIG. 1a illustrates an exemplary computing system (100A) comprising a hypervisor (140A) that hosts one or more VMs (110A, 120A). The computing system (100A) comprises various hardware devices (150), such as one or more processors (151), one or more storage devices (152), and / or one or more peripheral devices (153). The peripheral devices (153) may be configured to perform input and output (I / O) for the computing system (100A). An ellipsis (154) indicates that there may be different or additional hardware devices included in the computing system (100A). Hereinafter, the physical hardware (150) running the hypervisor (140A) is also referred to as the “host computing system” (100A) (also referred to as the “host”), and the VM (110A, 120A) hosted on the hypervisor (140A) is also referred to as the “guest partition” (also referred to as the “guest”).
[0021] As illustrated in FIG. 1a, the hypervisor (140A) hosts one or more guest partitions (e.g., VM A (110A), VM B (120A)). The ellipsis (130) indicates that there may be any number of guest partitions included in the computing system (100A). The hypervisor (140A) allocates a portion of the physical hardware resources (150) of the computing system (100A) to each of the guest partitions (110A, 120A) and gives the guest partitions (110A, 120A) the illusion that they own the resources (114A, 124A). These “illusory” resources are called “virtual” resources. Each of the VMs (110A, 120A) uses its own virtual resources (114A, 124A) to run its own operating system (OS) (113A, 123A). The operating system (113A, 123A) can then allocate virtual resources (114A, 124A) to its various user applications (112A, 122A). For example, a guest partition (110A) runs the OS (113A) and user applications (112A). If the OS (113A) is a Windows® operating system, the user applications (112A) will be Windows® applications. As another example, a VM (120A) runs the OS (123A) and user applications (122A). If the OS (123A) is a Linux operating system, the user applications (122A) will be Linux applications.
[0022] Each virtual hardware resource (114A, 124A) may or may not have a corresponding physical hardware resource (150). If the corresponding physical hardware resource (150) is available, the hypervisor (140A) determines how access will be provided to the guest partition (110A, 120A) requiring its use. For example, the resource (150) may be partitioned or time-shared. If the virtual hardware resource (114A, 124A) does not have a matching physical hardware resource, the hypervisor (140A) may generally mimic the action of the required hardware resource through a combination of software and other hardware resources physically available on the host computing system (100A).
[0023] As illustrated in FIG. 1a, in some embodiments, the hypervisor (140A) may be the only software running at the highest privilege level defined by the system architecture. Such a system is called a native VM system. Conceptually, in a native VM system, the hypervisor (140A) is first installed on bare hardware, and then guest partitions VM A (110A) and VM B (120A) are installed on top of the hypervisor (140A). The guest operating systems (113A, 123A) and other less privileged applications (112A, 122A) run at a privilege level lower than the privilege level of the hypervisor. This generally implies that the privilege level of the guest OS (113A, 123A) may be mimicked by the hypervisor (140A).
[0024] Alternatively, in some embodiments, the hypervisor is installed on a host platform that is already running an existing OS. Such a system is called a hosted VM system. In a hosted VM system, the hypervisor controls and manages the resources required by each guest partition by leveraging capabilities already available in the host OS. In a hosted VM system, the hypervisor can be implemented at the user level or at a privilege level similar to the host operating system. Alternatively, part of the hypervisor is implemented at the user level, and other parts of the hypervisor are implemented at a privileged level.
[0025] In some embodiments, one of the guest partitions running on the same computing system may be considered to have more privileges than the others. FIG. 1b illustrates such an exemplary embodiment, where the more privileged guest partition may be called the parent partition (110B), and the remaining partitions are called child partitions (120B). In some embodiments, the parent partition (110B) includes an operating system (113B) and / or a virtualization service module (115B). Each of the operating system (113B) and / or the virtualization service module (115B) may have direct access to a hardware device (150) through a device driver (114B). Thus, the operating system (113B) may also be called the host operating system. Additionally, the virtualization service module (115B) may be able to create the child partition (120B) using a hypercall. Depending on the configuration, the virtualization service module (115B) may expose a subset of hardware resources to each child partition (140B) through the virtualization client module (125B). The child partition (120B) generally does not have direct access to the physical processor and / or cannot handle actual interrupts. Instead, the child partition (120B) may use the virtualization client module (125B) to obtain a virtual view of the virtual hardware resources (124B).
[0026] In some embodiments, the parent partition (110B) may also include a VM management service application (112B) that allows a user (e.g., a system administrator) to view and modify the configuration of the virtualization service module (115B). For example, the hypervisor (140B) may be a Microsoft® Hyper-V hypervisor, and the parent partition may run Windows® Server. The user interface of the parent partition may provide a window that displays the entire user interface of the child partition(s) (120B). Interaction with applications running in the child partition (120B) may occur within the window. If the host operating system (113B) is Windows®, a graphical window may be established on the desktop interface to interact with the child partition (120B) on the same platform. The elements of FIG. 1b (113B, 122B, 123B and / or 124B) are similar to the elements of FIG. 1a (113A, 122A, 123A and / or 124A) and will not be discussed further.
[0027] Regardless of whether it is a native VM system or a hosted VM system, the relationship between the hypervisor and the guest partition is generally similar to the relationship between an operating system and an application in conventional computing systems. In conventional computing systems, the operating system typically operates in kernel mode—for example, relative to user mode—at a level with higher privileges than that of applications. Similarly, in a VM environment, the hypervisor also operates in a mode with higher privileges than the guest partition's mode. If the guest partition needs to perform privileged operations, such as updating page tables, it will use hypercalls to request these operations, much like system calls for conventional operations.
[0028] Accordingly, the described embodiments of the invention are applicable to both native VM systems and hosted VM systems, and the term “hypervisor” herein refers to a hypervisor implemented in any type of VM system.
[0029] To better understand how a hypervisor operates, it is also necessary to understand how the hypervisor maintains the state of each guest partition. In a computing system, the system's architectured state is contained within and maintained by the system's hardware resources. Typically, there is an architectured hierarchy of state resources extending from registers at one end of the hierarchy to secondary storage (e.g., a hard drive) at the other end.
[0030] In a VM environment, each guest partition has its own architectured state information, and the host computing system may or may not have the appropriate physical resources to map each element of the guest's state to the natural level of the host's memory hierarchy. For example, the guest's register state can actually be stored in the host platform's main memory as part of a register context block.
[0031] In normal operation, the hypervisor (140A) periodically switches control between guest partitions (110A, 120A). As operations are performed on the state of the guest, the state maintained in the host computing system (100A) is modified as if it were in the guest operating system (113A, 123A). In some embodiments, the hypervisor (140A) configures an isomorphism that maps the state of the virtual guest operating system (113A, 123A) to the state of the physical host computing system (100A).
[0032] FIG. 2 illustrates a schematic diagram of isomorphism (200) between a guest partition (210) and a host computing system (220). The isomorphism (200) maps guest state(s) (211, 212) to host state(s) (221, 222) using a function (230), namely V(state A) = state A' and V(state B) = state B'. For a sequence of operations in the guest partition (210) that modifies the state of the guest partition (210) from state A (211) to state B (212), there is a corresponding sequence of operations in the host computing system (220) that modifies the state of the host computing system from state A' (221) to state B' (222). The isomorphism (200) between the guest partition (210) and the host computing system (220) is managed by a hypervisor (e.g., 140A, 140B).
[0033] In the embodiment, there are two basic methods for managing guest state to achieve this VM isomorphism. One method is to use a level of indirection by maintaining the state for each guest at a fixed location in the memory hierarchy of the host computing system using a hypervisor-managed pointer that indicates the currently active guest state. When the hypervisor switches between guest partitions, it changes the pointer to match the current guest. FIG. 3a illustrates such an exemplary embodiment (300A) of state management by indirection. Referring to FIG. 3a, the memory (320) of the host computing system managed by the hypervisor stores register values for each VM A and VM B in register context blocks (321, 322). The register block pointer (311A) of the processor (310A) points to the register context block (322) of the currently active guest partition (e.g., VM B). When a different guest partition is activated, the hypervisor changes the pointer (311A) stored in the processor to point to the register context block (321, 322) of the activated guest partition, and loads the program counter to point to the activated VM program and start execution.
[0034] Another method of managing guest state is to copy the guest state information to that natural level of the memory hierarchy whenever it is activated by the hypervisor, and to copy it again when a different guest is activated. FIG. 3b illustrates such an exemplary embodiment (300B) of guest state management by copying. As illustrated in FIG. 3b, host memory (320) managed by the hypervisor similarly stores register values for each guest partition VM A and VM B in register context blocks (321, 322). However, unlike in FIG. 3a, here the hypervisor copies the entire guest register content (322) into the register file (311B) of the processor (310B) when VM B is activated (after saving the registers of the previous guest back to memory (320).
[0035] The choice between indirection and copying may depend, for example, on the frequency of use and whether the guest state managed by the hypervisor is maintained on a type of hardware resource different from that of the native system. For frequently used state information, such as general-purpose registers, it may be desirable to swap the state of the virtual machine to a corresponding physical resource whenever the virtual machine is activated. However, as illustrated in FIGS. 3a and 3b, in either case, the register values (321, 322) of each VM are often maintained in the main memory (320) of the host platform as part of a register context block.
[0036] In addition to VM state management, memory management is also worth discussing. In a VM environment, each guest partition has its own set of virtual memory tables, also known as "Level 1" memory page tables. Address translation in each Level 1 memory page table translates addresses within its virtual address space into locations within guest physical memory. Here, guest physical memory does not correspond to host physical memory on the host computing system. Instead, guest physical addresses (GPAs) undergo an additional mapping to determine addresses within the physical memory of the host hardware, also known as host physical addresses (HPAs). This GPA-to-HPA mapping is performed by another set of virtual memory tables in the host computing system, also known as "Level 2" or nested memory page tables. Note that the combined total size of all guests' guest physical memory can be larger than the system's actual physical memory. In an embodiment, the hypervisor maintains its own swap space, distinct from the swap space of each guest, and the hypervisor manages physical memory by swapping guest physical pages into and out of its own swap space. Additionally, the status of all virtually or physically allocated pages and their corresponding attributes are stored in a list called the Page Frame Number (PFN) list. Tracks of virtually or physically allocated pages are stored in a database called the PFN database.
[0037] FIG. 4 illustrates an exemplary embodiment (400) of memory virtualization using multiple stage memory page tables (440 and 450). Each entry in the first level memory table (440) maps a location (e.g., PFN) in the virtual memory PFN database (410) to a location (e.g., PFN) in the guest physical memory PFN database (420). As illustrated in FIG. 4, part of the PFN database (410) tracks the virtualally allocated pages of a program running on VM A, and part of the PFN database (420) tracks the physical memory of the guest VM of VM A. Additionally, to convert GPA to HPA, the hypervisor also maintains a second level memory page table (450) that maps guest physical pages to host physical pages. There is also a PFN database (430) that tracks the physical memory of the host computing system.
[0038] As illustrated in FIG. 4, the physical page frame number (1500) is assigned to the guest physical page frame number (2500), which is subsequently assigned to the virtual memory page frame number (3000). Similarly, the physical page frame number (2000) is assigned to the guest physical page frame number (6000), which is subsequently assigned to the virtual memory page frame number (2000). The remaining physical memory pages may be assigned to either another VM or the hypervisor itself. These remaining physical memory pages, including the physical memory pages assigned to the hypervisor itself to record the register values of each VM (e.g., memory (320)), are also tracked by the PFN database (430).
[0039] FIG. 4 is merely a schematic diagram to illustrate a simplified concept of memory management using multiple stage page tables. Additional mechanisms may be implemented to achieve the same or similar memory management objectives. For example, in some embodiments, page translation is supported by a combination of a page table and a translation lookaside buffer (TLB).
[0040] A virtual environment and how a hypervisor manages and virtualizes various hardware resources have been explained with reference to FIGS. 1a through 4, and now a specific embodiment of a hypervisor hot restart will be explained with reference to FIGS. 5a and 5b.
[0041] FIG. 5a illustrates an exemplary computing system (500A) in which hypervisor hot restart is enabled. The computing system (500A) may include a native VM system as illustrated in FIG. 1a, where the hypervisor (520A) is the only software running at the highest privilege level defined by the system architecture. Alternatively, the computing system (500A) may include a hosted VM system, where the hypervisor (520A) is installed on a computing system that is already running a host OS.
[0042] Regardless of whether the computing system (500A) includes a native VM system or a hosted VM system, a new partition called a service partition (560) is created during a hypervisor hot restart. In an embodiment, the service partition (560) is treated differently from other partition types. When the service partition (560) is created, hardware resources including at least some processor resource(s) and memory resource(s) are allocated to the service partition (560). In some embodiments, the allocation of hardware resources may be based on user input(s). In an alternative embodiment, the hypervisor (520A) or the component that created the service partition (560) automatically allocates a predetermined portion of processor resources and / or memory resources to the service partition (560).
[0043] The allocation of processor resources may specify the total amount of processing power required by the service partition (560) and leave the allocation of available processors to workload management software, or the service partition (560) or the hypervisor (520A) may specify that a specific processor within the system must be dedicated for the use of the service partition (560). The service partition (560) or the hypervisor (520A) may specify that the service partition (560) requires a specific number of processors, but that the service partition (560) is willing to share these processors with other partitions. For example, if the service partition (560) requires a total of 8 processing units, the service partition (560) or the hypervisor (520A) may specify that the service partition (560) requires 8 processors exclusively for itself, or that the service partition (560) requires 16 processors but requires only half of the available compute power of each processor. The allocation of memory (including RAM and / or hard disk) can specify an amount of memory in chunks of a specific unit, for example, 1MB.
[0044] The service partition (560) is subsequently initialized. Any component of the Trust Computing Base (TCB) may receive a task to create and / or initialize the service partition (560). In some embodiments, the first hypervisor (520A) is part of the TCB, and the first hypervisor (520A) creates and / or initializes the service partition (560). The initialization process involves bootstrapping, which entails a series of actions in which each action enables the ability to perform the next action until the entire system is finally revived. In some embodiments, the hypervisor (520A) constructs a loader block for the second hypervisor (561). The execution of the initialization code allows other aspects of the service partition (560) to be initialized. As illustrated in FIG. 5a, unlike other ordinary guest partitions (540A, 550A) where a conventional OS (541A, 551A) is loaded, a hypervisor (561) is loaded within a service partition (560). For clarity, the hypervisors (520A and 561) are hereinafter referred to as the first hypervisor (520A) and the second hypervisor (561). In some embodiments, the initialization of the second hypervisor (561) may also include providing the second hypervisor read-only specific access to a specific physical resource (510).
[0045] The purpose of a hypervisor hot restart is to eventually replace the first hypervisor (520A) with the second hypervisor (561) with at least an undetectable interrupt to the guest virtual machine. Before the second hypervisor (561) replaces the first hypervisor (520A), the first hypervisor (520A) initializes the second hypervisor (561) using available system invariants (e.g., features supported by hardware resource(s) (510)) and then synchronizes the runtime state (512) with the second hypervisor (561). In some embodiments, the runtime state (512) is stored in memory (e.g., RAM) managed by the hypervisor (520A). As described with respect to FIGS. 3a, 3b and 4, the runtime state (512) may include hardware architecture states such as register values (321, 322) of each guest partition (e.g., Guest A (540A), Guest (550A)), virtualization hardware states such as second-level memory page table(s) (450) of each guest partition (e.g., Guest A (540A), Guest (550A)), and / or software-defined states such as a PFN database (430) associated with the first hypervisor (520A).
[0046] In some embodiments, communication between the first hypervisor (520A) and the second hypervisor (561) is coordinated through an orchestrator (530A) during initialization and synchronization. The first hypervisor (520A) allows and supports the orchestrator (530A) to register and complete specific requests received from the second hypervisor (561). The orchestrator (530A) is a software component of a host computing system (500A) configured to arrange communication between the first hypervisor (520A) and the second hypervisor (561). In some embodiments, the first hypervisor (520A) is part of the system's Trusted Computing Base (TCB), and the orchestrator (530A) runs in a Trusted Execution Environment (TEE) to maintain the reliability of inputs generated for the second hypervisor.
[0047] In some embodiments, the orchestrator (530A) may use a reverse hypercall to transmit state to the second hypervisor (561) and / or the second hypervisor (561) may use an intercept to request some service (e.g., characteristics of physical resources that the second hypervisor (561) does not have access to) from the orchestrator (530A). In some embodiments, the orchestrator (530A) shares a portion of the memory page (511) with the second hypervisor (561) so that data can be efficiently transferred between the orchestrator (530A), the first hypervisor (520A), and the second hypervisor (561). In some embodiments, the orchestrator (530A) issues a reverse hypercall to the second hypervisor that follows the hypercall application binary interface (ABI) to migrate the relevant state.
[0048] As described above, when the service partition (560) is created and loaded, the second hypervisor (561) first needs to obtain various system invariants, such as features of the hardware resources (510). In some embodiments, there is a strict coupling between the features seen by the first hypervisor (520A) and the features seen by the second hypervisor (561).
[0049] Alternatively, in another embodiment, there is loose coupling between the first hypervisor and the second hypervisor. The orchestrator (530A) can query the first hypervisor via a hypercall and / or directly query the hardware resource(s) to obtain features and characteristics of the hardware resource(s) regarding the initialization of the second hypervisor (561). For example, the second hypervisor (561) may attempt to read CPUID or MSR values. However, since it is in the virtual machine itself, it may only have access to virtualized values rather than physical values of the underlying physical computing system. In this case, the orchestrator (530A) may invoke a hypercall to the first hypervisor (520A) to obtain the corresponding physical values of the corresponding physical resources. For example, the second hypervisor (561) may want to query processor support for XSAVE features and instructions. If the first hypervisor (520A) does not support XSAVE virtualization, the orchestrator (530A) may query the characteristics of the underlying processor and determine that the processor supports XSAVE. The results of the query obtained may also be stored in a shared memory page (511), so that the second hypervisor (561) will have access to the results of the query. In some cases, the second hypervisor (561) may not be able to obtain all characteristics of the hardware resources (510). In some embodiments, these characteristics will be left unknown during initialization and will be obtained later after devirtualization.
[0050] In addition to obtaining system invariants, the runtime state of the first hypervisor (520A) is also synchronized with the second hypervisor (561). The runtime state includes at least dynamic system state and logical software state. The dynamic system state includes guest visible hardware architecture state, which includes but is not limited to general-purpose registers and control registers. The dynamic system state also includes virtualization instruction set architecture (ISA) specific hardware state, which includes but is not limited to the state of the guest VM, such as virtual central processing unit (CPU) state, a second-level page table (also known as a nested page table), and a list of assigned hardware devices (e.g., network cards). Synchronizing the runtime state (512) of the first hypervisor includes synchronizing at least one second-level memory page table and / or one or more PFN databases with the second hypervisor (561). The second-level memory table is a memory page table that maps the GPA of each guest partition to the HPA (e.g., the second-level memory page table (450) of FIG. 4). The synchronized PFN database may include a PFN database that tracks the physical memory of the host computing system (e.g., PFN database (430)).
[0051] However, after the initial synchronization of the runtime state between the two hypervisors (520A and 561) and before the second hypervisor (561) is devirtualized, the guest partitions (540A, 550A) are still running on top of the first hypervisor (520A), and these guest partitions (540A, 550A) can still invoke hypercalls. When an incoming hypercall is serviced by the first hypervisor (520A), the state of the guest partition (540A or 550A) that invoked the hypercall changes, and the previously synchronized state of the second hypervisor (561) is no longer accurate. To resolve this issue, the incoming hypercall (before the devirtualization of the second hypervisor) also needs to be recorded and synchronized with the second hypervisor (561).
[0052] In some embodiments, the orchestrator (530A) also receives the task of logging and synchronizing each incoming hypercall. For example, when the first hypervisor (520A) receives and services a hypercall invoked by guest partitions (540A, 540B), the orchestrator (530A) logs the hypercall and the actions that occurred during the service of the hypercall by the first hypervisor (520A). At the same time, the orchestrator (530A) supplies the hypercall to the second hypervisor (561) via a reverse hypercall. In some embodiments, the command point of the second hypervisor (561) is in the hypercall dispatch loop so that the second hypervisor (561) processes it when the reverse hypercall is supplied to the second hypervisor (561). When a reverse hypercall is received, the second hypervisor (561) switches the context of the guest partitions (540A, 540B) that invoked the hypercall and processes the reverse hypercall to rebuild the required software state and / or dormant hardware state. When the reverse hypercall is completed, the second hypervisor (561) then notifies the orchestrator (530A) of completion.
[0053] In some cases, a sequence of hypercalls is serviced within a short time period, and the orchestrator (530A) may transmit only to the state associated with the last or related action of the sequence of hypercalls. In such cases, the second hypervisor (561) may replay only a partial log of the action, i.e., perform a compressed replay. This process may be repeated as many times as necessary until the second hypervisor (561) is fully or at least substantially synchronized with the first hypervisor (520A). After that, the second hypervisor (561) is devirtualized from the service partition (560) and replaces the first hypervisor (520A). Devirtualization includes the first hypervisor (520A) trampolineing to the second hypervisor and transferring all physical hardware control to the second hypervisor (561). In some embodiments, the trampoline can be achieved by reusing the existing control transfer mechanism of the computing system (500A) for kernel soft reboot.
[0054] In some embodiments, the second hypervisor (561) may undergo additional initialization, verify the hardware state, re-initialize the hardware, and / or initialize new hardware that was not previously programmed by the first hypervisor. In some embodiments, devirtualization may include the first hypervisor (520A) freezing all guest partition(s) (540A, 550A), transmitting final state details to the second hypervisor (561), and transferring the guest partition(s) to the second hypervisor. The second hypervisor then thaws each of the guest partition(s). When the second hypervisor (561) initiates devirtualization, the first hypervisor (520A) is effectively terminated. Other system software, user software, and / or system firmware are not permitted to run during devirtualization. However, DMAs programmed prior to devirtualization may continue to proceed and be completed.
[0055] For example, when access to a physical device is provided to a guest partition (e.g., child A (540A), child (550A)), the guest partition may initiate a DMA using a GPA as a source or target for a DMA operation. A second-level page table (which may be programmed into an input-output memory management unit (IOMMU) by the first hypervisor) converts the GPA into an HPA and provides the HPA to the DMA engine, in addition to performing a required permission check. As mentioned above, the first hypervisor transmits / synchronizes not only the architecture guest state but also the architecture virtualization state to the second hypervisor. The architecture virtualization state includes, among other things, a second-level page table for CPU, IOMMU, and / or device domain information. Thus, when the second hypervisor devirtualizes and re-initializes the hardware, the second hypervisor carefully programs the hardware using a new page table configured itself during the initial synchronization stage. Since the effective address translation and permission of the new page table are the same despite being two different instances, all new translation requests from the DMA engine can continue and successfully use the same page table without loss of fidelity.
[0056] Additionally, in some embodiments, the memory footprint of the first hypervisor (520A) may be reclaimed by the computing system either eagerly or leisurely depending on the total memory pressure.
[0057] FIG. 5b illustrates another exemplary computing system (500B) in which hypervisor hot restart is enabled. The computing system (500B) corresponds to the computing system (100B) of FIG. 1b, in which one of the guest partitions (e.g., parent partition (110B, 530B)) has more privileges than the other guest partitions (e.g., child partitions (120B, 540B, 550B)). In some embodiments, the parent partition (530B) includes a host operating system and / or a virtualization service module (531B), and either of them may be configured to create and manage the child partitions (540B and 550B) and to handle various system management functions and device drivers. Similar to the embodiment illustrated in FIG. 5a, a service partition (560) is created, a second hypervisor (561) is initialized within the service partition (560), and finally, the first hypervisor (520B) is replaced by the second hypervisor (561) to complete the hypervisor hot restart.
[0058] In some embodiments, in the computing system (500B), since the parent partition (530B) has a higher level of privilege than the child partitions (540B, 550B), the orchestrator (532B) may be implemented in the parent partition (530B) as part of the host operating system of the parent partition (530B) or as a virtualization service module (531B). The orchestrator (532B), similar to the orchestrator (530B), functions to coordinate communication between the first hypervisor (520B) and the second hypervisor (561). The elements (541B, 551B) of FIG. 5b are similar to the elements (541A, 551A) of FIG. 5a and will not be discussed further.
[0059] Additionally, in some embodiments, a hot restart of the hypervisor (520A, 520B) may also be accompanied by a kernel soft reboot or reset of the guest partitions (540A, 550A, 530B, 540B, 550B). In a kernel soft reboot, the guest partition (540A, 550A, 530B, 540B, or 550B) may be recreated and initialized as a new partition, and the runtime state of the corresponding guest partition is synchronized with the new partition. Once all runtime states of the guest partition are synchronized with the new partition, the new partition may replace the corresponding guest partition to complete the kernel soft reboot of the corresponding guest partition. In some embodiments, only the parent partition (530B) is hot restarted along with the hot restart of the hypervisor (520B). Alternatively, or additionally, each of the guest partitions (540A, 550A) and / or child partitions (540B, 550B) is hot restarted along with the hot restart of the hypervisor (520A, 520B).
[0060] Some embodiments restart VM-related components within the parent partition (530B) without restarting the host operating system within the parent partition (530B). For example, some embodiments restart the virtualization service module (531B) in connection with the hypervisor hot restart without restarting the host operating system (113B). In this way, it is possible to upgrade and restart the hypervisor along with its operating system-level management components without restarting the host operating system (113B).
[0061] The hypervisor hot restart described herein, unlike a conventional hypervisor restart, substantially reduces the interrupt time caused to a running guest VM. Unlike a normal hypervisor restart, which can take several minutes, the second hypervisor (561) described herein is initialized and synchronized while the first hypervisor (520A or 520B) is still running, depending on the number of hosted VMs and the amount of hardware resources being managed. Thus, each running guest partition will only experience a short period of freezing (e.g., one second or less than a few seconds), which may not even be noticed by the user.
[0062] The following discussion refers to many methods and method actions that can now be performed. Method actions may be discussed in a specific order or exemplified as occurring in a specific order in a flowchart; however, since actions depend on another action being completed before they are performed, no specific order is required unless specifically stated or required.
[0063] FIG. 6a illustrates a flowchart of an exemplary method (600) for a hypervisor hot restart. The method (600) is implemented in a computing system capable of corresponding to a computing system (100A, 100B or 500A, 500B). The method (600) includes the step (610) of running a first hypervisor. The first hypervisor then creates one or more guest partitions, each capable of hosting a guest operating system (620). The purpose of the hypervisor hot restart is to replace the first hypervisor with a new hypervisor. When the hypervisor hot restart is performed, the computing system creates a service partition (630) and initializes a second hypervisor within the service partition (640). Next, at least a portion of the runtime state of the first hypervisor is synchronized with the second hypervisor (650). When synchronization is completed or substantially completed, the computing system devirtualizes the second hypervisor and replaces the first hypervisor with the second hypervisor (660).
[0064] FIG. 6b illustrates a flowchart of an exemplary method (640) for initializing a service partition, which corresponds to step (640) of FIG. 6a. The method (640) includes the step (641) of generating a loader block for a second hypervisor by the first hypervisor. The loader block is a logical configuration describing a portion of static system characteristics and resources for initialization. The method (640) also includes the step (642) of obtaining one or more system invariants and the step (643) of sharing one or more system invariants with the second hypervisor. The second hypervisor is then initialized based on one or more system invariants (644). In some embodiments, the system invariants include one or more features of the hardware resource(s) of the computing system (645). In some embodiments, there is tight coupling between the first hypervisor and the second hypervisor (646). In some embodiments, there is loose coupling between the first hypervisor and the second hypervisor. In this case, one or more feature(s) of the hardware resource(s) can be obtained from the first hypervisor via a hypercall (647) or directly from the hardware resource(s) via a system call(s) (648).
[0065] FIG. 6c illustrates a flowchart of an exemplary method (650) for synchronizing a portion of the runtime state of a first hypervisor with a second hypervisor, which corresponds to step (650) of FIG. 6a. The method (650) may be performed by the orchestrator (530A, 532B) of FIG. 5a or FIG. 5b. The method (650) includes a step (651) of collecting the runtime state of the first hypervisor. The method (650) also includes a step (652) of sharing the runtime state with the second hypervisor, which may be performed by the orchestrator (530A, 532B) via a reverse hypercall, which may or may not rely on a shared memory page to further facilitate signaling and / or message / data delivery. The shared runtime state of the first hypervisor is then replicated by the second hypervisor (653). Before the second hypervisor is devirtualized, guest partitions running on top of the first hypervisor continue to operate and can still make hypercalls, and the collected runtime state will soon become inaccurate. Therefore, this process can be repeated multiple times until the second hypervisor and the first hypervisor are fully or substantially synchronized.
[0066] FIG. 6d illustrates a flowchart of an exemplary method (651) for collecting the runtime state of a first hypervisor, which corresponds to step (651) of FIG. 6c. The method (651) includes a step (651-A) of collecting current data associated with one or more second-level memory page tables and one or more PFN(s). The method (651) also includes a step (651-B) of collecting data associated with an incoming hypercall. In particular, when a hypercall is invoked by a guest partition, the first hypervisor receives it (651-C) and will service the received hypercall (651-D). The computing system (e.g., an orchestrator (530A or 532B)) then logs the received hypercall and the actions that occurred during the service of the hypercall (651-E).
[0067] FIG. 6e illustrates a flowchart of an exemplary method (652) for sharing the runtime state of a first hypervisor with a second hypervisor, which corresponds to step (652) of FIG. 6c. The method (652) includes step (652-A) of supplying one or more second-level page table(s) and one or more PFN(s) to the second hypervisor. The method (652) also includes step (652-B) of supplying an incoming hypercall to the second hypervisor and step (652-C) of supplying a log of an action that occurred during the service of the hypercall (by the first hypervisor) to the second hypervisor.
[0068] FIG. 6f illustrates a flowchart of an exemplary method (653) for replicating the runtime state of a first hypervisor to a second hypervisor, which corresponds to step (653) of FIG. 6c. The method (653) includes the step (653-A) of receiving a reverse hypercall from an orchestrator in the second hypervisor. The second hypervisor then switches the context of the guest partition that invoked the hypercall (653-B) and processes the reverse hypercall to rebuild the essential software state and / or the dormant hardware state (653-C). When the processing of the reverse hypercall is complete, the second hypervisor notifies the orchestrator (653-D).
[0069] FIG. 6g illustrates a flowchart of an exemplary method (660) for devirtualizing a second hypervisor, which corresponds to step (660) of FIG. 6a. The method (660) includes the step (661) of freezing each guest partition currently running on top of the first hypervisor. The final state of the first hypervisor is then transmitted to the second hypervisor (662). Next, each guest partition is switched to the second hypervisor (663), and the second hypervisor thaws each guest partition (664). Finally, the first hypervisor is shut down (665).
[0070] The exemplary embodiment described above is implemented in a special partition (e.g., a service partition) for servicing the hypervisor, but note that the entire technique of pre-initializing data structures and / or migrating / synchronizing runtime state to save time can be used in any part of any VM.
[0071] Finally, since the principles described herein are implemented in the context of a computing system (e.g., the computing system of FIG. 1a, FIG. 1b (100A, 100B) and / or the computing system of FIG. 5a or FIG. 5b (500A, 500B)), some introductory discussion of the computing system will be explained with respect to FIG. 7.
[0072] Computing systems are now increasingly taking on a wide variety of forms. A computing system may be a device that is not conventionally considered a computing system, such as, for example, portable devices, consumer electronics, laptop computers, desktop computers, mainframes, distributed computing systems, data centers, or wearables (e.g., glasses). In this description and claims, the term “computing system” is broadly defined to include any device or system (or combination thereof) comprising at least one physical and type of processor and physical and type of memory capable of having computer-executable instructions that can be executed by the processor. Memory may take any form and may vary depending on the nature and form of the computing system. Computing systems may be distributed through a network environment and may include multi-configured computing systems.
[0073] As illustrated in FIG. 7, in its most basic configuration, the computing system (700) generally includes at least one hardware processing unit (702) and memory (704). The processing unit (702) may include a general-purpose processor and may also include a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. The memory (704) may be physical system memory that is volatile, non-volatile, or a combination of both. The term “memory” may also be used herein to refer to non-volatile mass storage, such as a physical storage medium. If the computing system is distributed, processing, memory, and / or storage capabilities may also be distributed.
[0074] The computing system (700) also has multiple structures, often referred to as “executable components.” For example, the memory (704) of the computing system (700) is exemplified as containing an executable component (706). The term “executable component” is a name for a structure well understood by those skilled in the art of computing as a structure that may be software, hardware, or a combination thereof. For example, when implemented in software, those skilled in the art will understand that the structure of the executable component may include software objects, routines, methods, etc., and that such executable component may be executed in the computing system regardless of whether it exists in the heap of the computing system or whether it exists on a computer-readable storage medium.
[0075] In such cases, those skilled in the art will recognize that the structure of an executable component exists on a computer-readable medium and enables the computing system to perform a function when interpreted by one or more processors of the computing system (e.g., by processor threads). This structure may be directly computer-readable by the processor (as in the case where the executable component is a binary). Alternatively, the structure may be structured to be interpretable and / or compiled (whether single-stage or multi-stage) to generate such a binary that is directly interpretable by the processor. This understanding of the exemplary structure of an executable component is within the scope of understanding of those skilled in the art of computing when using the term “executable component.”
[0076] The term “executable component” is also well understood by those skilled in the art to include structures such as hardcoded or hard-wired logic gates that are implemented exclusively or nearly exclusively in hardware, such as Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), or any other specialized circuits. Accordingly, the term “executable component” refers to structures well understood by those skilled in the art of computing, whether implemented in software, hardware, or a combination thereof. In this description, terms such as “component,” “agent,” “manager,” “service,” “engine,” “module,” “virtual machine,” etc., may also be used. As used in this description and in cases, these terms (with or without modification) also have structures intended to be synonymous with the term “executable component” and are therefore also well understood by those skilled in the art of computing.
[0077] In the above description, embodiments are described with reference to actions performed by one or more computing systems. If such actions are implemented in software, one or more processors (of the associated computing system performing the actions) direct the actions of the computing system in response to the execution of computer-executable instructions constituting executable components. For example, such computer-executable instructions may be implemented in one or more computer-readable media forming computer program products. Examples of such actions include the manipulation of data. If such actions are implemented exclusively or nearly exclusively in hardware, such as in an FPGA or ASIC, the computer-executable instructions may be hardcoded or wired logic gates. Computer-executable instructions (and manipulated data) may be stored in memory (704) of the computing system (700). The computing system (700) also includes a communication channel (708) that enables the computing system (700) to communicate with another computing system, for example, through a network (710).
[0078] Although not all computing systems require a user interface, in some embodiments, the computing system (700) includes a user interface system (712) for use in interfacing with a user. The user interface system (712) may include an output mechanism (712A) as well as an input mechanism (712B). The principles described herein are not limited to a precise output mechanism (712A) or input mechanism (712B), but will vary depending on the nature of the device. However, the output mechanism (712A) may include, for example, a speaker, a display, a haptic output, a hologram, etc. Examples of the input mechanism (712B) may include, for example, a microphone, a touchscreen, a hologram, a camera, a keyboard, a mouse or other pointer input, any type of sensor, etc.
[0079] The embodiments described herein may include or utilize a special-purpose or general-purpose computing system comprising computer hardware, such as one or more processors and system memory, as discussed in more detail below. The embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by a general-purpose or special-purpose computing system. A computer-readable medium storing computer-executable instructions is a physical storage medium. A computer-readable medium carrying computer-executable instructions is a transmission medium. Thus, as an example rather than a limitation, embodiments of the invention may include at least two distinctly different types of computer-readable media: a storage medium and a transmission medium.
[0080] Computer-readable storage media include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other physical and type of storage media that can be used to store desired program code means in the form of computer-executable instructions or data structures and can be accessed by a general-purpose or special-purpose computing system.
[0081] A “network” is defined as one or more data links that enable the transmission of electronic data between computing systems and / or modules and / or other electronic devices. When information is transmitted or provided to a computing system via a network or other communication connection (either wired or wireless, or a combination of wired or wireless), the computing system appropriately regards the connection as a transmission medium. A transmission medium may include a network and / or data link that can be used to carry a desired means of program code in the form of computer-executable instructions or data structures, and which can be accessed by a general-purpose or special-purpose computing system. The above combinations must also be included within the scope of a computer-readable medium.
[0082] Additionally, upon reaching various computing system components, means of program code in the form of computer-executable instructions or data structures may be automatically transferred from the transmission medium to the storage medium (or vice versa). For example, computer-executable instructions or data structures received via a network or data link may be buffered in RAM within a network interface module (e.g., “NIC”) and subsequently transferred to the computing system RAM and / or a less volatile storage medium in the computing system. Thus, it should be understood that the storage medium may be included in the computing system component that also (or even primarily) utilizes the transmission medium.
[0083] Computer-executable instructions include instructions and data that, for example, cause a general-purpose computing system, a special-purpose computing system, or a special-purpose processing device to perform a specific function or group of functions when executed on a processor. Alternatively or additionally, computer-executable instructions may configure a computing system to perform a specific function or group of functions. Computer-executable instructions may be intermediate-form instructions, such as binary or assembly language, or instructions that undergo some transformation (such as compilation) prior to direct execution by a processor, such as source code.
[0084] Although the subject matter has been described in language specific to structural features and / or methodological conduct, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the features or conduct described above. Rather, the described features and conduct are disclosed as exemplary forms embodying the claims.
[0085] Those skilled in the art will recognize that the present invention may be implemented in a network computing environment having many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, portable devices, multiprocessor systems, microprocessor-based or programmable home electronic devices, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses), etc. The present invention may also be implemented in a distributed system environment in which both local and remote computing systems are linked through a network (by either a wired data link or a wireless data link, or by a combination of wired and wireless data links) to perform tasks. In a distributed system environment, program modules may be located on both local and remote memory storage devices.
[0086] Those skilled in the art will also recognize that the present invention may be practiced in a cloud computing environment. A cloud computing environment may be distributed, but this is not mandatory. When distributed, a cloud computing environment may be distributed internationally within an organization and / or may have components owned across multiple organizations. In this description and in the following claims, “cloud computing” is defined as a model that enables on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any other numerous benefits that may be obtained from such a model when properly deployed.
[0087] The remaining drawings may discuss various computing systems that correspond to the previously described computing system (700). The computing system of the remaining drawings includes various components or functional blocks that can implement various embodiments disclosed herein as described. Various components or functional blocks may be implemented in a local computing system, or may be implemented in a distributed computing system that includes elements residing in the cloud or implements aspects of cloud computing. Various components or functional blocks may be implemented in software, hardware, or a combination of software and hardware. The computing system of the remaining drawings may include more or fewer components than those illustrated in the drawings, and some of the components may be combined depending on the circumstances. Although not necessarily illustrated, various components of the computing system may access and / or utilize processors and memory, such as a processor (702) and memory (704), as needed to perform various functions.
[0088] With respect to the processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in different orders. Additionally, the described operations are provided by way of example only, and some of the operations may be optional, combined into fewer steps and operations, supplemented by additional operations, or extended by additional operations without compromising the essence of the disclosed embodiments.
[0089] The present invention may be embodied in other specific forms without departing from its spirit or characteristics. The described embodiments are to be considered in all respects as illustrative only and not limiting. Accordingly, the scope of the invention is defined by the appended claims rather than by the foregoing description. Any modifications occurring within the meaning and scope of the equivalence of the claims shall be included within their scope.
Claims
Claim 1 A computing system comprises: one or more processors; and one or more computer-readable media storing computer-executable instructions, wherein, when executed by the one or more processors, the computer-executable instructions cause the computing system to execute a first hypervisor that creates one or more guest partitions; create a service partition; initialize a second hypervisor within the service partition to the static system state of the first hypervisor; and synchronize at least a portion of the runtime state of the first hypervisor to the second hypervisor—synchronizing at least a portion of the runtime state of the first hypervisor to the second hypervisor includes intercepting a request for a system invariant from the first hypervisor via shared memory; and issuing an inverse hypercall to the second hypervisor to transmit the system invariant to the second hypervisor via shared memory—; A computing system structured to perform devirtualization of the second hypervisor from the service partition in order to replace the first hypervisor. Claim 2 A computing system according to claim 1, wherein initializing the second hypervisor within the service partition further comprises granting specific read-only access to specific portions of the one or more computer-readable media of the computing system. Claim 3 A computing system according to claim 1, wherein the runtime state comprises at least one of (1) one or more second-level memory page tables that map the guest physical memory of the guest partition(s) to the host physical memory of the computing system, (2) one or more page frame number databases, and (3) a list of physical devices connected to the guest partition and an assignment of the list of physical devices. Claim 4 A computing system according to paragraph 3, wherein the parent partition comprises a host operating system and a virtualization service module, the virtualization service is configured to provide virtualization services to at least one other guest partition, and the computing system also enables the virtualization service module to hot restart without restarting the host operating system. Claim 5 A computing system according to claim 1, wherein at least one of the one or more guest partitions comprises a privileged parent partition; said parent partition operates a host operating system or a virtualization service module; said host operating system or said virtualization service module comprises an orchestrator configured to orchestrate the initialization and synchronization of said second hypervisor; and said first hypervisor allows said orchestrator to intercept specific request(s) from said second hypervisor to said first hypervisor. Claim 6 A computing system according to claim 5, wherein when the orchestrator intercepts a request from the second hypervisor to the first hypervisor, the orchestrator issues a reverse hypercall to the second hypervisor and shares a portion of a memory page with the second hypervisor for communication purposes. Claim 7 A computing system according to claim 6, wherein initializing the second hypervisor comprises, by the first hypervisor, generating a loader block for the second hypervisor—the loader block comprising a logical configuration describing at least a portion of the static system state for initialization—; obtaining one or more system invariant(s) of the computing system by the orchestrator; sharing the one or more system invariant(s) with the second hypervisor; and initializing the second hypervisor based on the one or more system invariant(s). Claim 8 A computing system according to claim 7, wherein the one or more system invariant(s) include one or more features supported by one or more hardware resources of the computing system. Claim 9 A computing system according to claim 8, wherein the orchestrator obtains at least one of the above features from at least one of (1) the first hypervisor via a hypercall and (2) directly from a hardware resource. Claim 10 In claim 9, the above orchestrator is a computing system that acquires at least one of the above features through a binary interface that discovers or queries hardware capabilities. Claim 11 A computing system according to claim 7, wherein synchronizing the runtime state of the first hypervisor comprises: collecting the runtime state of the first hypervisor by the orchestrator; sharing the runtime state with the second hypervisor via a reverse hypercall by the orchestrator; and causing the second hypervisor to replicate the runtime state. Claim 12 A computing system according to claim 11, wherein collecting the runtime state comprises receiving a hypercall invoked by a guest partition by the first hypervisor; servicing the hypercall by the first hypervisor; and recording a log of an action that occurred during the hypercall and the servicing of the hypercall by the first hypervisor by the orchestrator, and sharing the runtime state comprises supplying the hypercall to the second hypervisor by the orchestrator; or supplying a log of the action to the second hypervisor by the orchestrator. Claim 13 A computing system according to claim 11, wherein replicating the runtime state comprises: receiving a reverse hypercall from the orchestrator in the second hypervisor; switching to the context of the guest partition that invoked the hypercall; processing the reverse hypercall to reconstruct the essential software state or the dormant hardware state in the second hypervisor; and notifying the orchestrator of the completion of the reverse hypercall. Claim 14 A computing system according to claim 1, wherein devirtualizing the second hypervisor comprises: transferring control of the physical hardware of the computing system from the first hypervisor to the second hypervisor; freezing each of the one or more guest partitions; transmitting the final state of the first hypervisor to the second hypervisor; switching each guest partition to the second hypervisor; and defreezing each guest partition. Claim 15 A computing system according to claim 1, wherein the computing system also enables at least one of the one or more guest partitions to hot restart. Claim 16 A method implemented in a computing system for hot restart of a hypervisor on a host, comprising: executing a first hypervisor that creates one or more guest partitions; creating a service partition; initializing a second hypervisor within the service partition to the static system state of the first hypervisor; synchronizing at least a portion of the runtime state of the first hypervisor to the second hypervisor—synchronizing at least a portion of the runtime state of the first hypervisor to the second hypervisor comprises: intercepting a request for a system invariant from the first hypervisor via shared memory; and issuing a reverse hypercall to the second hypervisor to transmit the system invariant to the second hypervisor via the shared memory—; and devirtualizing the second hypervisor from the service partition to replace the first hypervisor. Claim 17 A method implemented in a computing system for hot restart of a hypervisor on a host, wherein the runtime state comprises at least one of (1) a second-level memory page table mapping the guest physical memory of the guest partition(s) to the host physical memory of the computing system, (2) a page frame number database, and (3) a list of physical devices connected to the guest partition and the assignment of the list of physical devices. Claim 18 A method implemented in a computing system for hot restart of a hypervisor on a host, wherein, in claim 16, at least one of the one or more guest partitions comprises a privileged parent partition, said parent partition operates a host operating system or a virtualization service module, and said host operating system or said virtualization service module comprises an orchestrator configured to coordinate the initialization and synchronization of said second hypervisor. Claim 19 A method implemented in a computing system for hot restart of a hypervisor on a host, wherein, in claim 16, devirtualizing the second hypervisor comprises transferring control of the physical hardware of the computing system from the first hypervisor to the second hypervisor; freezing each of the one or more guest partitions; transmitting the final state of the first hypervisor to the second hypervisor; switching each guest partition to the second hypervisor; and defreezing each guest partition. Claim 20 A computer-readable storage medium having computer-executable instructions stored therein, wherein the computer-executable instructions, when executed by one or more processors of a computing system, the computer-executable instructions cause the computing system to execute a first hypervisor that creates one or more guest partitions; create a service partition; initialize a second hypervisor within the service partition to the static system state of the first hypervisor; synchronize at least a portion of the runtime state of the first hypervisor to the second hypervisor—synchronizing at least a portion of the runtime state of the first hypervisor to the second hypervisor includes intercepting a request for a system invariant from the first hypervisor via shared memory; and issuing a reverse hypercall to the second hypervisor to transmit the system invariant to the second hypervisor via the shared memory—; and structured to perform devirtualization of the second hypervisor from the service partition to replace the first hypervisor.
Citation Information
Patent Citations
Hypervisor replacing method and information processing device
US20120254865A1
