Hypervisor hot restart

Abstract

To provide a computing system that achieves hot restart of a hypervisor by replacing a first hypervisor in execution with a second hypervisor with minimum down time for a guest partition.SOLUTION: A computing system executes a first hypervisor, creates one or more virtual partitions, generates a service partition during a hot restart, initializes a second hypervisor, migrates and synchronizes at least part of a runtime state of the first hypervisor to the second hypervisor using a reverse hypercall, and de-virtualizes the second hypervisor from the service partition in order to replace the first hypervisor after synchronization. De-virtualization includes transferring control of hardware resources from the first hypervisor to the second hypervisor using a previously migrated and synchronized runtime state.SELECTED DRAWING: Figure 6A

Description

Background Art

[0001] Background

[0001] Virtualization in computing often refers to abstracting physical components into logical objects. A virtual machine (VM) can virtualize hardware resources including processors, memory, storage, and network connectivity, and present the virtualized resources to the host operating system. The process of virtualizing a VM includes at least two parts: (1) mapping virtual resources or states, such as registers, memory, or files, to the actual resources within the underlying physical hardware, and (2) using machine instructions and / or system calls to execute actions specified by virtual machine instructions and / or system calls, such as emulating the application binary interface (ABI) or instruction set architecture (ISA) interface of the virtual machine.

[0002]

[0002] A hypervisor is a software layer that provides an environment in which virtual machines operate (i.e., a virtualized hardware partition). The hypervisor is located between the physical resources on a physical hardware computing system and the VMs. Without a hypervisor, the operating system communicates directly with the hardware beneath it, i.e., disk operations go directly to the disk subsystem, and memory calls are fetched directly from physical memory. When multiple operating systems of multiple VMs are running simultaneously on a single machine, the hypervisor manages the interaction between each VM and the shared hardware to prevent simultaneous control of the shared hardware by multiple VMs.

[0003]

[0003] When the hypervisor is reset or restarted (e.g., by a software upgrade), all VMs running on the hypervisor are interrupted (e.g., restarted, paused, etc.), which can significantly affect the workload and the services running on the VMs.

[0004]

[0004] The claims made herein are not limited to embodiments that resolve any disadvantages or embodiments that operate only in the environment described above. Rather, this background is provided only to illustrate an exemplary technical field in which some of the embodiments described herein can be put into practice. [Overview of the project] [Means for solving the problem]

[0005] Brief Overview

[0005] This summary is provided to introduce a set of concepts that will be further explained in the following detailed description in a simplified form. This summary is not intended to identify any important or essential features of the claims, nor to be used to help determine the scope of the claims.

[0006]

[0006] Embodiments described herein relate to a hypervisor hot restart that replaces a first hypervisor with a second hypervisor with minimal interference to the currently running guest partition. Embodiments described herein are implemented in a computing system. First, the computing system runs a first hypervisor on the computing system. The first hypervisor is configured to create one or more guest partitions. A service partition is created during a soft restart of the hypervisor. The second hypervisor is initialized within the service partition. The service partition is created along with an identity mapping between its guest physical address (GPA) space and host physical address (HPA) space. Optionally, the service partition may be granted additional privileges beyond those of conventional partitions / virtual machines to facilitate its initialization. Any execution environment that can meet the platform's boot and security requirements can create and / or initialize the service partition. For example, a component within the Computing Control Center (TCB) of the computing system, such as the first hypervisor, may be tasked with creating or initializing the second hypervisor. During initialization, at least a portion of the state of the first hypervisor is initialized in the second hypervisor. The state of the first hypervisor includes, but is not limited to, (1) static system states, (2) dynamic system states, and (3) logical software states. The static system states include, but are not limited to, the system topology, memory maps, and memory layouts of the first hypervisor. The dynamic system states include, but are not limited to, the hardware architecture states visible to the guest, including general-purpose registers and mode control registers. The dynamic system states also include, but are not limited to, the state of the guest VM, such as the state of the virtual central processing unit (CPU), the second-level page table (also known as a nested page table), and the list of assigned hardware devices (e.g., network cards), which are specific to the virtualization instruction set architecture (ISA).The logical software state includes, but is not limited to, a page frame number (PFN) database. Since the dynamic system state and logical software state can constantly change during runtime, these states are also called runtime states. Finally, the second hypervisor is devirtualized from the service partition to replace the first hypervisor, and the state of each VM is restored by the second hypervisor.

[0007]

[0007] In some embodiments, at least one of one or more guest partitions includes a privileged parent partition, which runs a host operating system including an orchestrator configured to coordinate the initial setup and synchronization of a second hypervisor. In some embodiments, the first hypervisor supports the orchestrator by enabling and completing certain requests (e.g., hypercall intercepts, register intercepts) received from the second hypervisor. These requests (sent from the second hypervisor to the first hypervisor) are intercepted (e.g., registered and completed) by the orchestrator, and are therefore also called “intercepts”. In some embodiments, in response to receiving an intercept, the orchestrator issues a “reverse hypercall” to the second hypervisor, which follows the application binary interface (ABI) of the hypercall to transition to the associated state. A hypercall is a call from a guest partition to the hypervisor, and a “reverse hypercall” is a call from a privileged software or hardware component (e.g., the orchestrator) to the second hypervisor in a service partition. In some embodiments, the orchestrator may share a portion of memory pages with the first and second hypervisors for communication purposes (e.g., hypercalls, intercepts, and / or reverse hypercalls).

[0008]

[0008] While examples of embodiments are implemented within a dedicated partition (e.g., a service partition) for providing services to the hypervisor, the overall technique of pre-initializing data structures within a virtual machine and / or migrating / synchronizing runtime states within a VM / partition can be used in any part of the OS of any VM to reduce service downtime and provide better uptime and continuity.

[0009]

[0009] Further features and advantages are described below, partially apparent from the description, or can be known by practicing the teachings herein. The features and advantages of the present invention can be realized by the apparatus and combinations specifically indicated in the appended claims. The features of the present invention are more fully apparent from the following description and the appended claims, or can be known by practicing the invention as described below.

[0010] Brief explanation of the drawing

[0010] In order to explain the advantages and features listed above, as well as how to obtain other advantages and features, a more detailed explanation of the contents briefly described above will be given by referring to specific embodiments shown in the attached drawings. It should be understood that these drawings only show typical embodiments and should therefore not be considered limiting in scope, and embodiments will be described and explained with further specificity and detail using the attached drawings. [Brief explanation of the drawing]

[0011] [Figure 1A]

[0011] An example of a computing system including a hypervisor that hosts one or more virtual machines is shown. [Figure 1B]

[0011] An example of a computing system including a hypervisor that hosts one or more virtual machines is shown. [Figure 2]

[0012] A schematic diagram of the isomorphism between the guest partition and the host computing system is shown. [Figure 3A]

[0013] An example of an embodiment of VM state management is shown. [Figure 3B]

[0013] An example of an embodiment of VM state management is shown. [Figure 4]

[0014] An example of an embodiment 400 of memory virtualization using a multi-level memory page table is shown. [Figure 5A]

[0015] An example of a computing system in which hot restart of a hypervisor is enabled is shown. [Figure 5B]

[0015] An example of a computing system in which hot restart of a hypervisor is enabled is shown. [Figure 6A]

[0016] A flowchart of an example of a method for hot restart of a hypervisor is shown. [Figure 6B]

[0017] A flowchart of an example of a method for initializing a service partition is shown. [Figure 6C]

[0018] A flowchart of an example of a method for synchronizing a part of the runtime state of a first hypervisor with a second hypervisor is shown. [Figure 6D]

[0019] A flowchart of an example of a method for collecting the runtime state of a first hypervisor is shown. [Figure 6E]

[0020] A flowchart of an example of a method for sharing the runtime state of a first hypervisor with a second hypervisor is shown. [Figure 6F]

[0021] A flowchart of an example of a method for replicating the runtime state of a first hypervisor to a second hypervisor is shown. [Figure 6G]

[0022] A flowchart of an example of a method for de-virtualizing a second hypervisor is shown. [Figure 7]

[0023] An example of a computing system that can employ the principles described herein is shown.

Embodiments of the Invention

[0012] Detailed explanation

[0024] The embodiments described herein relate to a hot restart of a hypervisor that replaces a first hypervisor with a second hypervisor with minimal interference to the currently running guest partition. The embodiments described herein are implemented in a computing system. First, the computing system runs a first hypervisor on the computing system. The first hypervisor is configured to create one or more virtual machines / partitions, each hosting a guest operating system. During the soft restart of the hypervisor, a service partition is created with a second-level page table to which identities are mapped. The second hypervisor is then initialized within the service partition and synchronized with at least some of the state of the first hypervisor. Any execution environment that can meet the platform's boot and security requirements can create and / or initialize the second hypervisor. For example, a component within a highly reliable computing framework (TCB), such as the first hypervisor, may be tasked with creating and / or initializing the service partition.

[0013]

[0025] In some embodiments, the state of the first hypervisor includes, but is not limited to, (1) a static system state, (2) a dynamic system state, and (3) a logical software state. The static system state includes, but is not limited to, the system topology, memory map, and memory layout of the first hypervisor. The dynamic system state includes, but is not limited to, the hardware architecture state visible to a guest, including general-purpose registers and mode control registers. The dynamic system state also includes, but is not limited to, the state of a guest VM such as the state of a virtual central processing unit (CPU), a second-level page table (also known as a nested page table), and a list of allocated hardware devices (e.g., a network card), which are virtualization instruction set architecture (ISA)-specific hardware states. The logical software state includes, but is not limited to, a page frame number (PFN) database. Since the dynamic system state and the logical software state can change continuously during runtime, these states are also referred to as runtime states. Finally, to replace the first hypervisor, the second hypervisor is virtualization-released from the service partition, and the state of each VM is restored by the second hypervisor.

[0014]

[0026] In some embodiments, at least one of one or more guest partitions includes a privileged parent partition, which runs a host operating system including an orchestrator configured to coordinate the initial setup and synchronization of a second hypervisor. In some embodiments, the first hypervisor is part of the system's High Reliability Computational Processing (TCB), and the orchestrator runs within a High Reliability Execution Environment (TEE) to maintain the reliability of the inputs generated for the second hypervisor. In some embodiments, the first hypervisor enables and supports the orchestrator to register and complete certain requests received from the second hypervisor. These requests (sent from the second hypervisor to the first hypervisor) are intercepted (e.g., registered and completed) by the orchestrator, and are therefore also called “intercepts.” In some embodiments, in response to receiving an intercept, the orchestrator issues an “inverse hypercall” to the second hypervisor, which follows the application binary interface (ABI) of hypercalls to transition to the associated state. A hypercall is typically a call from a guest partition to the hypervisor requesting an operation with appropriate privileges. Switching from the guest partition context to the hypervisor context is achieved using platform-specific instructions. With the use of platform-specific instructions / mechanisms, control is returned from the hypervisor context to the guest context upon completion of the privileged operation. A "reverse hypercall" is a request from a privileged component (e.g., an orchestrator) to an operation in the hypervisor within a service partition that uses the hypercall ABI. Switching from the orchestrator to the second hypervisor context is achieved by allowing the execution of the virtual processor in the service partition, which has registers formatted as required by the hypercall ABI.Upon completion of the inverse hypercall, the second hypervisor generates a very specific intercept that causes a deliberate context switch from the second hypervisor in the service partition back to the original orchestrator. In some embodiments, the orchestrator may share one or more memory pages with the first and second hypervisors for communication purposes (e.g., hypercalls, intercepts, and / or inverse hypercalls).

[0015]

[0027] In some embodiments, initializing the second hypervisor includes the first hypervisor generating a loader block for the second hypervisor. The loader block includes a logical construct that describes part of the static system state for initialization. The static system state includes various system invariants such as the system topology, memory map, and layout. The memory map includes an identity map of almost all system memory. In some embodiments, the identity map includes at least the memory visible to the first hypervisor being replaced. For example, in some cases, both the first (old) hypervisor and the second (new) hypervisor know about all the memory in the system. In some cases, for example, when some RAM containing errors must be taken offline, or when the second (new) hypervisor must know about new memory that has been hot-added during a hot restart, at least one of the first and / or second hypervisors does not know about all the memory. The orchestrator obtains one or more system invariants of the computing system, including at least one or more features supported by the computing system's hardware resources, (1) directly from the hardware or system resources if the particular resources are not virtualized by the hypervisor, and / or (2) from the first hypervisor if the orchestrator's access to the underlying physical hardware resources is virtualized by the first hypervisor. In some embodiments, the orchestrator migrates the relevant system invariants by inverse hypercalls. The second hypervisor is then initialized based on the shared one or more system invariants. In some embodiments, initializing the second hypervisor may also include providing the second hypervisor with read-only unique access to certain physical resources.

[0016]

[0028] After the second hypervisor is initialized using static system state within the service partition, the computing system migrates the runtime state of the first hypervisor to the second hypervisor and keeps it synchronized. The runtime state includes at least dynamic system state and logical software state. Synchronizing the runtime state of the first hypervisor may involve the orchestrator collecting the runtime state of the first hypervisor and sharing / migrating the runtime state with the second hypervisor via inverse hypercall. As mentioned above, the dynamic system state includes, but is not limited to, the hardware architecture state visible to the guest, including general-purpose registers and mode control registers. The dynamic system state also includes, but is not limited to, the state of the guest VM, such as the state of the virtual central processing unit (CPU), the second-level page table (also known as a nested page table), and the virtualization instruction set architecture ISA-specific hardware state, including a list of assigned hardware devices (e.g., network cards). The logical software state includes, but is not limited to, the page frame number (PFN) database. Next, the second hypervisor replicates the shared second-level memory page tables and / or PFN database, and keeps them dormant until the second hypervisor is finally de-virtualized.

[0017]

[0029] In addition, during this synchronization, guest partitions running on the first hypervisor continue to operate and issue hypercalls to the first hypervisor. Therefore, after the initial synchronization of computationally intensive states, additional inbound hypercalls to the first hypervisor that affect previously transmitted states may also need to be synchronized to the second hypervisor. In some embodiments, such state changes are also transmitted using inverse hypercalls.

[0018]

[0030] When the first and second hypervisors are fully or nearly synchronized, the second hypervisor is devirtualized. In some embodiments, devirtualization involves the first hypervisor "trampoline" the second hypervisor, relinquishing all physical hardware control to the second hypervisor. "Trampoline" may refer to a memory location that holds addresses indicating interrupt service routines, I / O routines, etc. Here, "trampoline" means transferring all physical hardware control from the first hypervisor to the second hypervisor. In some embodiments, trampolining can be achieved by reusing an existing control transfer mechanism of the computing system for kernel soft reboot. In some embodiments, no other system or user software, or burling system firmware, is permitted to run during the trampolining process. However, direct memory access (DMA) programmed before devirtualization can continue to run and complete. In some embodiments, the second hypervisor may undergo additional initialization to verify the hardware state, reinitialize the hardware, and / or initialize new hardware that has not been previously programmed by the first hypervisor. Devirtualization may also include freezing each of the guest partitions currently running on the first hypervisor and transmitting the final state of the first hypervisor to the second hypervisor. Each of the guest partitions is then switched onto the second hypervisor. The second hypervisor then decompresses each of the guest partitions. The memory footprint of the first hypervisor may be rapidly or slowly reclaimed by the computing system that is subject to the overall memory load.

[0019]

[0031] Since the embodiments described herein are implemented within a VM environment capable of simultaneously supporting multiple guest partitions, each running its own operating system and associated application programs, some introductory explanations of virtualization and hypervisors will be given with reference to Figures 1 to 4.

[0020]

[0032] Hypervisor-based virtualization often allows a privileged host operating system (running in the "parent" partition) and multiple guest operating systems (running in the "child" partitions) to simultaneously share access to the hardware of a single computing system, while each operating system is given the illusion of having access to a complete set of system resources. To create this illusion, in some embodiments, the hypervisor in the computing system creates multiple partitions, each acting as a virtual hardware machine (i.e., VM) running its own operating system and associated application programs. Each operating system controls and manages a set of virtualized hardware resources.

[0021]

[0033] Figure 1A shows an example of a computing system 100A that includes a hypervisor 140A hosting one or more VMs 110A, 120A. The computing system 100A includes various hardware devices 150, such as one or more processors 151, one or more storage devices 152, and / or one or more peripheral devices 153. Peripheral devices 153 may be configured to provide input and output (I / O) for the computing system 100A. Omitted symbols 154 indicate that different or additional hardware devices may be included in the computing system 100A. Hereinafter, the physical hardware 150 running the hypervisor 140A will also be referred to as the “host computing system” 100A (or “host”), and the VMs 110A, 120A hosted on the hypervisor 140A will also be referred to as “guest partitions” (or “guests”).

[0022]

[0034] As shown in Figure 1A, the hypervisor 140A hosts one or more guest partitions (e.g., VM A110, VM B120). The ellipsis 130 indicates that any number of guest partitions may be included in the computing system 100A. The hypervisor 140A allocates a portion of the physical hardware resources 150 of the computing system 100A to each of the guest partitions 110A and 120A, giving them the illusion of owning resources 114A and 124A. These "illusory" resources are called "virtual" resources. Each of the VMs 110A and 120A uses its own virtual resources 114A and 124A to run its own operating system (OS) 113A and 123A. The operating systems 113A and 123A can allocate the virtual resources 114A and 124A to their various user applications 112A and 122A. For example, guest partition 110A runs OS113A and user application 112A. If OS113A is the Windows® operating system, then user application 112A will be a Windows® application. As another example, VM120A runs OS123A and user application 1122A. If OS123A is the Linux operating system, then user application 122A will be a Linux application.

[0023]

[0035] Each virtual hardware resource 114A, 124A may or may not have a corresponding physical hardware resource 150. If the corresponding physical hardware resource 150 is available, the hypervisor 140A determines how to provide access to the guest partitions 110A, 120A that request its use. For example, resource 150 can be partitioned or time-sharing. If the virtual hardware resources 114A, 124A do not have a matching physical hardware resource, the hypervisor 140A can typically emulate the actions of the desired hardware resource by combining physically available software and other hardware resources on the host computing system 100A.

[0024]

[0036] As shown in Figure 1A, in some embodiments, the hypervisor 140A may be the only software running within the highest privilege level defined by the system architecture. Such a system is called a native VM system. Conceptually, in a native VM system, the hypervisor 140A is first installed on bare hardware, and then guest partitions VM A110A and VM B120A are installed on top of the hypervisor 140A. Guest operating systems 113A, 123A and other lower-privileged applications 112A, 122A run within a privilege level lower than that of the hypervisor. This generally means that the privilege levels of the guest OSs 113A, 123A may be emulated by the hypervisor 140A.

[0025]

[0037] Alternatively, in some embodiments, the hypervisor is installed on a host platform that is already running an existing OS. Such a system is called a hosted VM system. In a hosted VM system, the hypervisor utilizes functions already available on the host OS to control and manage the resources desired by each guest partition. In a hosted VM system, the hypervisor may be implemented at the user level or at the privileged level, similar to the host operating system. Alternatively, part of the hypervisor may be implemented at the user level, and another part at the privileged level.

[0026]

[0038] In some embodiments, one of the guest partitions running on the same computing system may be considered to be more privileged than the other guest partitions. Figure 1B shows an example of such an embodiment, where the more privileged guest partition can be called the parent partition 110B, and the remaining partitions are called child partitions 120B. In some embodiments, the parent partition 110B includes an operating system 113B and / or a virtualization service module 115B. Each of the operating system 113B and / or the virtualization service module 115B may have direct access to hardware devices 150 by a device driver 114B. Thus, the operating system 113B can also be called the host operating system. Furthermore, the virtualization service module 115B may be able to create child partitions 120B using hypercalls. Depending on the configuration, the virtualization service module 115B may expose a subset of hardware resources to each child partition 140B by a virtualization client module 125B. Child partitions 120B generally do not have direct access to physical processors and / or cannot handle real interrupts. Alternatively, the child partition 120B can use the virtualization client module 125B to obtain a virtual view of the virtual hardware resource 124B.

[0027]

[0039] In some embodiments, the parent partition 110B may also include a VM management service application 112B that allows a user (e.g., a system administrator) to view and modify the configuration of the virtualization service module 115B. For example, the hypervisor 140B could be the Microsoft® Hyper-V hypervisor, and the parent partition could run Windows® Server. The user interface of the parent partition may provide a window that displays the full user interface of the child partition 120B. Interaction with applications running on the child partition 120B may take place within this window. If the host operating system 113B is Windows®, a graphical window can be established on the desktop interface to interact with the child partition 120B on the same platform. Elements 113B, 122B, 123B, and / or 124B in Figure 1B are similar to elements 113A, 122A, 123A, and / or 124A in Figure 1A, and therefore will not be discussed further.

[0028]

[0040] Whether it's a native VM system or a hosted VM system, the relationship between the hypervisor and the guest partition is generally similar to the relationship between an operating system and an application program in a traditional computing system. In a traditional computing system, the operating system generally operates within a privilege level higher than that of the application, for example, within kernel mode relative to user mode. Similarly, in a VM environment, the hypervisor also operates within a privilege mode higher than that of the guest partition. When a guest partition needs to perform a privileged operation, such as updating a page table, the guest partition uses hypercalls to request such an operation, just as it would with system calls in a traditional operation.

[0029]

[0041] Accordingly, the embodiments of the present invention described herein are applicable to both native VM systems and hosted VM systems, and the term "hypervisor" herein refers to a hypervisor implemented by any type of VM system.

[0030]

[0042] To further understand how a hypervisor works, we also need to understand how it maintains the state of each guest partition. In a computing system, the constructed state of the computing system is contained within and maintained by the computing system's hardware resources. Typically, there is a constructed hierarchy of state resources, ranging from registers at one end of the hierarchy to secondary storage (e.g., hard drives) at the other end of the hierarchy.

[0031]

[0043] In a VM environment, each guest partition has its own pre-built state information, and there may or may not be sufficient physical resources in the host computing system to map each element of the guest state to its natural level within the host's memory hierarchy. For example, the guest's register state may actually be preserved in the host platform's main memory as part of a register context block.

[0032]

[0044] In normal operation, the hypervisor 140A periodically switches control between guest partitions 110A and 120A. When an operation is performed on the guest state, the state maintained on the host computing system 100A is modified to match the state on the guest operating systems 113A and 123A. In some embodiments, the hypervisor 140A constructs an isomorphism that maps the states of the virtual guest operating systems 113A and 123A to the states of the physical host computing system 100A.

[0033]

[0045] Figure 2 shows a schematic diagram of the isomorphism 200 between the guest partition 210 and the host computing system 220. The isomorphism 200 maps guest states 211, 212 to host states 221, 222 using the function 230, namely V(state A) = state A' and V(state B) = state B'. For a set of operations within the guest partition 210 that corrects the state of the guest partition 210 from state A211 to state B212, there is a corresponding set of operations within the host computing system 220 that corrects the state of the host computing system from state A'221 to state B'222. The isomorphism 200 between the guest partition 210 and the host computing system 220 is managed by the hypervisor (e.g., 140A, 140B).

[0034]

[0046] In this embodiment, there are two basic ways of managing guest states so that the isomorphism of the VMs is achieved. One way is to use a degree of indirection by holding the state of each guest in a fixed location in the memory hierarchy of the host computing system, with a pointer managed by the hypervisor indicating the currently active guest state. When the hypervisor switches between guest partitions, the hypervisor changes the pointer to match the current guest. Figure 3A shows an example of an embodiment 300A in which state management by indirection is performed. Referring to Figure 3A, the memory 320 of the host computing system managed by the hypervisor stores the register values ​​of VMs A and B in register context blocks 321 and 322. The register block pointer 311A ​​of the processor 310A points to the register context block 322 of the currently active guest partition (e.g., VM B). If a different guest partition is activated, the hypervisor modifies the pointer 311A ​​stored in the processor to point to register context blocks 321 and 322 of the activated guest partition, loads the program counter to point to the activated VM program, and starts execution.

[0035]

[0047] Another way to manage guest state is for the hypervisor to always replicate the guest's state information to its natural level in the memory hierarchy when activated, and then replicate it back when a different guest is activated. Figure 3B shows an example of such an embodiment 300B of guest state management by replication. As shown in Figure 3B, the host memory 320 managed by the hypervisor similarly stores the register values ​​of VMs A and B of each guest partition in register context blocks 321 and 322. However, unlike in Figure 3A, here the hypervisor replicates the entire guest register content 322 into the register file 311B of processor 310B when VM B is activated (after saving the previous guest's registers back into memory 320).

[0036]

[0048] The choice between indirect referencing and replication may depend, for example, on usage frequency and whether the guest state managed by the hypervisor is held in a different type of hardware resource than on the native system. For frequently used state information such as general-purpose registers, it may be preferable to swap the virtual machine state to the corresponding physical resource each time the virtual machine is activated. However, as shown in Figures 3A and 3B, in both cases, the register values ​​321 and 322 of each VM are often kept in the host platform's main memory 320 as part of a register context block.

[0037]

[0049] In addition to VM state management, memory management is also worth discussing. In a VM environment, each guest partition has its own set of virtual memory tables, also called "first-level" memory page tables. Address translation in each first-level memory page table transforms addresses in its virtual address space to locations in guest physical memory. Here, guest physical memory does not correspond to host physical memory on the host computing system. Instead, guest physical addresses (GPAs) are subjected to further mapping to determine addresses in the physical memory of the host hardware, also called host physical addresses (HPAs). This mapping from GPA to HPA is performed by another set of virtual memory tables on the host computing system, also called "second-level" or nested memory page tables. Note that the combined total size of guest physical memory for all guests may be larger than the actual physical memory on the system. In some embodiments, the hypervisor maintains its own swap space separate from each guest's swap space, and manages physical memory by swapping guest physical pages in and out of its own swap space. Furthermore, the status of all virtually or physically allocated pages and their corresponding attributes is stored in a list called the Page Frame Number (PFN) list. The track of virtually or physically allocated pages is stored in a database called the PFN database.

[0038]

[0050] Figure 4 shows an example of an embodiment 400 of memory virtualization using multi-level memory page tables 440 and 450. Each entry in the first-level memory table 440 maps a location in the PFN database 410 of virtual memory (e.g., PFN) to a location in the PFN database 420 of guest physical memory (e.g., PFN). As shown in Figure 4, part of the PFN database 410 tracks the virtually allocated pages of a program running on VM A, and part of the PFN database 420 tracks the physical memory of the guest VM of VM A. Furthermore, to convert GPA to HPA, the hypervisor also maintains a second-level memory page table 450 that maps guest physical pages to host physical pages. There is also a PFN database 430 that tracks the physical memory of the host computing system.

[0039]

[0051] As shown in Figure 4, a physical page frame numbered 1500 is allocated to a guest physical page frame numbered 2500, which in turn is allocated to a virtual memory page frame numbered 3000. Similarly, physical page frame 2000 is allocated to a guest physical page frame numbered 6000, which in turn is allocated to a virtual memory page frame numbered 2000. The remaining physical memory pages can be allocated to other VMs or to the hypervisor itself. These remaining physical memory pages (e.g., memory 320), including those allocated to the hypervisor itself to record the register values ​​of each VM, are also tracked by the PFN database 430.

[0040]

[0052] Figure 4 is merely a schematic diagram illustrating a simplified concept of memory management using a multi-level page table. Additional mechanisms may be implemented to achieve the same or similar memory management objectives. For example, in some embodiments, page conversion is supported by a combination of a page table and a conversion index buffer (TLB).

[0041]

[0053] Referring to Figures 1A to 4, we will describe how the virtual environment and the hypervisor manage and virtualize various hardware resources. Next, we will explain specific embodiments of the hypervisor's hot restart with reference to Figures 5A and 5B.

[0042]

[0054] Figure 5A shows an example of a computing system 500A in which hypervisor hot restart is enabled. Computing system 500A may include a native VM system in which hypervisor 520A is the only software running within the highest privilege level defined by the system architecture, as shown in Figure 1A. Alternatively, computing system 500A may include a hosted VM system in which hypervisor 520A is installed on a computing system that is already running a host OS.

[0043]

[0055] Whether the computing system 500A includes a native VM system or a hosted VM system, a new partition called a service partition 560 is created during a hypervisor hot restart. In some embodiments, the service partition 560 is treated differently from other partition types. When the service partition 560 is created, hardware resources, including at least some processor resources and memory resources, are allocated to the service partition 560. In some embodiments, the allocation of hardware resources may be based on user input. In alternative embodiments, the hypervisor 520A or the component that created the service partition 560 automatically allocates a predetermined portion of the processor resources and / or memory resources to the service partition 560.

[0044]

[0056] Processor resource allocation can specify the total amount of processing power required by the service partition 560, leaving the allocation of available processors to the workload management software, or the service partition 560 or hypervisor 520A can specify that specific processors in the system be allocated to use by the service partition 560. The service partition 560 or hypervisor 520A can specify that the service partition 560 requires a certain number of processors, but is willing to share those processors with other partitions. For example, if the service partition 560 requires a total of 8 processors, the service partition 560 or hypervisor 520A can specify that the service partition 560 requires 8 dedicated processors, or that the service partition 560 requires 16 processors, but only half of the available computing power of each processor. Memory allocation (including RAM and / or hard disks) can specify the amount of memory in chunks of a specific granularity, for example, 1 MB.

[0045]

[0057] Next, the service partition 560 is initialized. Any component within the High Reliability Computing Block (TCB) may be tasked with creating and / or initializing the service partition 560. In some embodiments, the first hypervisor 520A is part of the TCB and generates and / or initializes the service partition 560. The initialization process includes bootstrapping, which involves a series of actions, each activating a function that allows subsequent actions to be performed until the entire system is finally activated. In some embodiments, the hypervisor 520A constructs a loader block for the second hypervisor 561. Executing initialization code allows for the initialization of other aspects of the service partition 560. As shown in Figure 5A, unlike the other normal guest partitions 540A and 550A where the conventional OS 541A and 551A are loaded, the hypervisor 561 is loaded into the service partition 560. For clarity, hypervisors 520A and 561 will hereafter be referred to as the first hypervisor 520A and the second hypervisor 561. In some embodiments, initializing the second hypervisor 561 may include providing the second hypervisor with read-only unique access to certain physical resources 510.

[0046]

[0058] The purpose of a hypervisor hot restart is to ultimately replace the first hypervisor 520A with the second hypervisor 561 with minimal to imperceptible disruption to the guest virtual machine. Before the second hypervisor 561 replaces the first hypervisor 520A, the first hypervisor 520A initializes the second hypervisor 561 using available system invariants (e.g., features supported by hardware resource 510) and then synchronizes the runtime state 512 with the second hypervisor 561. In some embodiments, the runtime state 512 is stored in memory (e.g., RAM) managed by the hypervisor 520A. As described with respect to Figures 3A, 3B, and 4, the runtime state 512 may include hardware architecture states such as register values ​​321, 322 for each guest partition (e.g., guests A 540A, 550A), virtualization hardware states such as the second-level memory page table 450 for each guest partition (e.g., guests A 540A, 550A), and / or software-defined states such as the PFN database 430 associated with the first hypervisor 520A.

[0047]

[0059] In some embodiments, communication between the first hypervisor 520A and the second hypervisor 561 during initial setup and synchronization is coordinated by an orchestrator 530A. The first hypervisor 520A enables and supports the orchestrator 530A to register and complete certain requests received from the second hypervisor 561. The orchestrator 530A is a software component of the host computing system 500A configured to coordinate communication between the first hypervisor 520A and the second hypervisor 561. In some embodiments, the first hypervisor 520A is part of the system's High Reliability Computational Processing (TCB), and the orchestrator 530A runs within a High Reliability Execution Environment (TEE) to maintain the reliability of the inputs generated for the second hypervisor.

[0048]

[0060] In some embodiments, the orchestrator 530A may use inverse hypercalls to transmit state to the second hypervisor 561, and / or the second hypervisor 561 may use intercepts to request certain services (e.g., characteristics of physical resources that the second hypervisor 561 does not have access to) from the orchestrator 530A. In some embodiments, the orchestrator 530A shares a portion of memory pages 511 with the second hypervisor 561 to enable efficient data transfer between the orchestrator 530A, the first hypervisor 520A, and the second hypervisor 561. In some embodiments, the orchestrator 530A issues inverse hypercalls to the second hypervisor, which follows the application binary interface (ABI) of the hypercall to transfer the relevant state.

[0049]

[0061] As discussed above, once the service partition 560 is created and loaded, the second hypervisor 561 first needs to obtain various system invariants, such as the characteristics of the hardware resource 510. In some embodiments, there is a tight coupling between the characteristics recognized by the first hypervisor 520A and the characteristics recognized by the second hypervisor 561.

[0050]

[0062] Alternatively, in other embodiments, there is loose coupling between the first hypervisor and the second hypervisor. The orchestrator 530A can query the first hypervisor by hypercall to obtain the characteristics and properties of hardware resources related to the initial setup of the second hypervisor 561, and / or query the hardware resources directly. For example, the second hypervisor 561 may attempt to read the CPUID or MSR value. However, these may be located within the virtual machine itself, and the second hypervisor 561 may only be able to access virtualized values ​​rather than the physical values ​​of the underlying physical computing system. In that case, the orchestrator 530A can make a hypercall to the first hypervisor 520A to obtain the corresponding physical values ​​of the corresponding physical resources. For example, the second hypervisor 561 may want to query processor support for the XSAVE function and instructions. If the first hypervisor 520A does not support XSAVE virtualization, the orchestrator 530A can query the characteristics of the underlying processor and determine that the processor supports XSAVE. The obtained query results can also be stored in the shared memory page 511 so that the second hypervisor 561 has access to the query results. In some cases, the second hypervisor 561 may not be able to obtain all the features of the hardware resource 510. In some embodiments, such features are left unknown during initialization and obtained later after devirtualization.

[0051]

[0063] In addition to obtaining system invariants, the runtime state of the first hypervisor 520A is also synchronized with the second hypervisor 561. The runtime state includes at least dynamic system state and logical software state. The dynamic system state includes, but is not limited to, hardware architecture state visible to the guest, including general-purpose registers and control registers. The dynamic system state also includes, but is not limited to, virtualization instruction set architecture ISA-specific hardware state, including guest VM state such as the state of the virtual central processing unit (CPU), second-level page tables (also known as nested page tables), and a list of assigned hardware devices (e.g., network cards). Synchronizing the runtime state 512 of the first hypervisor includes at least synchronizing one or more second-level memory page tables and / or one or more PFN databases with the second hypervisor 561. The second-level memory table is a memory page table that maps the GPA of each guest partition to the HPA (e.g., second-level memory page table 450 in Figure 4). The synchronized PFN database may include a PFN database (e.g., PFN database 430) that tracks the physical memory of the host computing system.

[0052]

[0064] However, after the initial synchronization of runtime states between the two hypervisors 520A and 561, and before the second hypervisor 561 is devirtualized, guest partitions 540A and 550A are still running on the first hypervisor 520A, and these guest partitions 540A and 550A can still invoke hypercalls. When an incoming hypercall is serviced by the first hypervisor 520A, the state of the guest partition 540A or 550A that invoked the hypercall changes, and the previously synchronized state of the second hypervisor 561 is no longer accurate. To resolve this issue, incoming hypercalls (before the second hypervisor is devirtualized) also need to be recorded and synchronized with the second hypervisor 561.

[0053]

[0065] In some embodiments, the orchestrator 530A is also tasked with recording and synchronizing each incoming hypercall. For example, if the first hypervisor 520A receives and services hypercalls invoked by guest partitions 540A and 540B, the orchestrator 530A logs the hypercalls and any actions that occurred during the service of the hypercalls by the first hypervisor 520A. Simultaneously, the orchestrator 530A feeds hypercalls to the second hypervisor 561 via inverse hypercalls. In some embodiments, the instruction point of the second hypervisor 561 is within the hypercall dispatch loop, and therefore the second hypervisor 561 processes inverse hypercalls when they are supplied to it. Upon receiving an inverse hypercall, the second hypervisor 561 processes the inverse hypercall to switch the context of the guest partitions 540A and 540B that invoked the hypercall and to reconstruct the necessary software and / or hibernation hardware states. Upon completion of the reverse hypercall, the second hypervisor 561 notifies the orchestrator 530A of the completion.

[0054]

[0066] In some cases, a series of hypercalls are served within a short period of time, and the orchestrator 530A may only transmit the state related to the last or relevant action of the series of hypercalls. In that case, the second hypervisor 561 can only replay a partial log of the actions, i.e., perform a condensed replay. This process may be repeated as many times as necessary until the second hypervisor 561 is fully or at least substantially synchronized with the first hypervisor 520A. The second hypervisor 561 is then devirtualized from the service partition 560 to replace the first hypervisor 520A. Devirtualization involves the first hypervisor 520A trampolineing to the second hypervisor, relinquishing all physical hardware control to the second hypervisor 561. In some embodiments, the trampoline may be achieved by reusing an existing control transition mechanism of the computing system 500A for kernel soft reboot.

[0055]

[0067] In some embodiments, the second hypervisor 561 may undergo additional initialization, verify the hardware state, reinitialize the hardware, and / or initialize new hardware that has not been previously programmed by the first hypervisor. In some embodiments, devirtualization may include the first hypervisor 520A freezing all guest partitions 540A, 550A, transmitting the details of the final state to the second hypervisor 561, and migrating the guest partitions onto the second hypervisor. The second hypervisor then decompresses each of the guest partitions. Once the second hypervisor 561 begins devirtualization, the first hypervisor 520A is effectively terminated. No other system software, user software, and / or system firmware are permitted to run during devirtualization. However, DMA programmed before devirtualization may continue to run and complete.

[0056]

[0068] For example, if a guest partition (e.g., child A 540A, 550A) is provided with access to a physical device, the guest partition can initiate DMA using its own GPA as the source or target of the DMA operation. The second-level page table (which may be programmed within the I / O Memory Management Unit (IOMMU) by the first hypervisor) translates the GPA to the HPA and provides the HPA to the DMA engine, in addition to performing the necessary authorization checks. As previously mentioned, the first hypervisor transmitted / synchronized the architecture guest state and architecture virtual state to the second hypervisor. The architecture virtual state includes, among other things, a second-level page table for CPU, IOMMU, and / or device domain information. Therefore, when the second hypervisor devirtualizes and reinitializes the hardware, it carefully programs the hardware using the new page table it constructed during the previous synchronization phase. Since the translation and authorization of effective addresses in the new page table are identical despite being two different instances, all new translation requests from the DMA engine can continue to be executed without loss of fidelity and successfully use the same page table.

[0057]

[0069] In addition, in some embodiments, the memory footprint of the first hypervisor 520A may be rapidly or slowly reclaimed by the computing system that bears the overall memory load.

[0058]

[0070] Figure 5B shows another example of compute system 500B in which hypervisor hot restart is enabled. Computer system 500B corresponds to compute system 100B in Figure 1B, where one of the guest partitions (e.g., parent partitions 110B, 530B) is privileged over the remaining guest partitions (e.g., child partitions 120B, 540B, 550B). In some embodiments, parent partition 530B may contain the host operating system and / or virtualization service module 531B, one of which may be configured to create and manage child partitions 540B and 550B and handle various system management functions and device drivers. Similar to the embodiment shown in Figure 5A, a service partition 560 is created, a second hypervisor 561 is initialized within the service partition 560, and finally, the first hypervisor 520B is replaced by the second hypervisor 561 to complete the hypervisor hot restart.

[0059]

[0071] In some embodiments, the parent partition 530B has a higher level of privileges than the child partitions 540B and 550B within the computing system 500B, so the orchestrator 532B can be implemented within the parent partition 530B as part of the host operating system or virtualization service module 531B of the parent partition 530B. The orchestrator 532B functions similarly to the orchestrator 530B in coordinating communication between the first hypervisor 520B and the second hypervisor 561. Elements 541B and 551B in Figure 5B are the same as elements 541A and 551A in Figure 5A, and therefore do not require further discussion.

[0060]

[0072] In addition, in some embodiments, a hot restart of hypervisors 520A and 520B may be accompanied by a kernel soft reboot or reset of guest partitions 540A, 550A, 530B, 540B, and 550B. In a kernel soft reboot, guest partitions 540A, 550A, 530B, 540B, or 550B may be recreated as new partitions and reinitialized, and the runtime state of the corresponding guest partitions is synchronized with the new partitions. Once all runtime states of the guest partitions are synchronized with the new partitions, the new partitions can replace the corresponding guest partitions in order to complete the kernel soft reboot of the corresponding guest partitions. In some embodiments, only the parent partition 530B is hot-restarted in conjunction with the hot restart of hypervisor 520B. Alternatively, in addition, guest partitions 540A, 550A, and / or child partitions 540B, 550B are each hot-restarted along with the hot restart of hypervisors 520A and 520B.

[0061]

[0073] Some embodiments restart VM-related components within the parent partition 530B without restarting the host operating system within the parent partition 530B. For example, some embodiments restart the virtualization service module 531B in connection with a hot restart of the hypervisor without restarting the host operating system 113B. In this way, the hypervisor can be upgraded and restarted along with its operating system-level management components without restarting the host operating system 113B.

[0062]

[0074] Unlike conventional hypervisor restarts, the hypervisor hot restart described herein significantly reduces the downtime inflicted on running guest VMs. Unlike a normal hypervisor restart, which can take several minutes depending on the number of hosted VMs and the amount of hardware resources being managed, the second hypervisor 561 described herein is initialized and synchronized while the first hypervisor 520A or 520B is still running, resulting in only a very short freeze period (e.g., less than one or a few seconds) in each running guest partition, which may even go unnoticed by the user.

[0063]

[0075] Next, the following explanation refers to several possible methods and actions of the methods. The actions of the methods may be discussed in a certain order, or shown in flowcharts as being performed in a specific order, but no specific ordering is required unless otherwise specified or required because an action depends on another action that has been completed before it is performed.

[0064]

[0076] Figure 6A shows a flowchart of an example of method 600 for a hot restart of a hypervisor. Method 600 is implemented on a computing system that may correspond to computing systems 100A, 100B, or 500A, 500B. Method 600 includes running a first hypervisor (610). The first hypervisor then creates one or more guest partitions, each capable of hosting a guest operating system (620). The purpose of a hot restart of a hypervisor is to replace the first hypervisor with a new hypervisor. When a hot restart of a hypervisor is performed, the computing system creates a service partition (630) and initializes a second hypervisor within the service partition (640). Next, at least part of the runtime state of the first hypervisor is synchronized with the second hypervisor (650). Once synchronization is complete or nearly complete, the computing system devirtualizes the second hypervisor to replace the first hypervisor with the second hypervisor (660).

[0065]

[0077] Figure 6B shows a flowchart of an example of method 640 for initializing a service partition, corresponding to step 640 in Figure 6A. Method 640 includes generating a loader block for a second hypervisor by the first hypervisor (641). The loader block is a logical construct that describes some of the static system characteristics and resources for initialization. Method 640 also includes obtaining one or more system invariants (642) and sharing one or more system invariants with the second hypervisor (643). The two hypervisors are then initialized based on one or more system invariants (644). In some embodiments, the system invariants include one or more characteristics of the hardware resources of the computing system (645). In some embodiments, there is a tight coupling between the first and second hypervisors (646). In some embodiments, there is a loose coupling between the first and second hypervisors. In that case, one or more features of the hardware resource can be obtained from the first hypervisor by hypercall (647), or directly from the hardware resource by system call (648).

[0066]

[0078] Figure 6C shows a flowchart of an example of method 650 for synchronizing a portion of the runtime state of a first hypervisor with a second hypervisor, corresponding to step 650 in Figure 6A. Method 650 may be performed by orchestrators 530A, 532B in Figure 5A or Figure 5B. Method 650 includes collecting the runtime state of the first hypervisor (651). Method 650 also includes sharing the runtime state with the second hypervisor (652), which may be done by inverse hypercalls by orchestrators 530A, 532B and may or may not further rely on shared memory pages to assist with signaling and / or message / data passing. The shared runtime state of the first hypervisor is then replicated by the second hypervisor (653). Before the second hypervisor is devirtualized, the guest partition running on the first hypervisor continues to function and can still make hypercalls, and the collected runtime state quickly becomes inaccurate. Therefore, this process may be repeated several times until the second and first hypervisors are fully or nearly synchronized.

[0067]

[0079] Figure 6D shows a flowchart of an example of method 651 for collecting the runtime state of the first hypervisor, corresponding to step 651 in Figure 6C. Method 651 includes collecting current data relating to one or more second-level memory page tables and one or more PFNs (651-A). Method 651 also includes collecting data relating to incoming hypercalls (651-B). Specifically, when a hypercall is invoked by a guest partition, the first hypervisor receives the incoming hypercall (651-C) and services it (651-D). The computing system (e.g., orchestrator 530A or 532B) then logs the received hypercall and any actions that occurred during the service of the hypercall (651-E).

[0068]

[0080] Figure 6E shows a flowchart of an example of method 652 for sharing the runtime state of the first hypervisor with the second hypervisor, corresponding to step 652 in Figure 6C. Method 652 includes feeding one or more second-level page tables and one or more PFNs to the second hypervisor (652-A). Method 652 also includes feeding incoming hypercalls to the second hypervisor (652-B) and feeding the second hypervisor a log of actions that occurred during the servicing of the hypercalls (by the first hypervisor) (652-C).

[0069]

[0081] Figure 6F shows a flowchart of an example of method 653 for replicating the runtime state of a first hypervisor with a second hypervisor, corresponding to step 653 in Figure 6C. Method 653 includes receiving an inverse hypercall from the orchestrator within the second hypervisor (653-A). The second hypervisor then switches the context of the guest partition that invoked the hypercall (653-B) and processes the inverse hypercall to reconstruct the necessary software state and / or hibernation hardware state (653-C). Upon completion of processing the inverse hypercall, the second hypervisor notifies the orchestrator (653-D).

[0070]

[0082] Figure 6G shows a flowchart of an example of method 660 for devirturing the second hypervisor, corresponding to step 660 in Figure 6A. Method 660 includes freezing each guest partition currently running on the first hypervisor (661). Then, the final state of the first hypervisor is transmitted to the second hypervisor (662). Next, each guest partition is switched onto the second hypervisor (663), and the second hypervisor decrypts each guest partition (664). Finally, the first hypervisor is terminated (665).

[0071]

[0083] Even if the above examples of embodiments are implemented within a dedicated partition (e.g., a service partition) for providing services to the hypervisor, it should be noted that the overall technique of pre-initializing data structures and / or migrating / synchronizing runtime state to save time can be used for any part of any VM.

[0072]

[0084] Finally, since the principles described herein are implemented in the context of a computing system (for example, computing systems 100A, 100B in Figures 1A and 1B, and / or 500A, 500B in Figures 5A or 5B), some introductory explanation of the computing system will be given with respect to Figure 7.

[0073]

[0085] Currently, computing systems are taking on an increasingly diverse range of forms. These may include, for example, portable devices, instruments, laptop computers, desktop computers, mainframes, distributed computing systems, data centers, and even wearables (e.g., glasses), devices that were not previously considered computing systems. In this description and the claims, the term “computing system” is broadly defined as any device or system (or combination thereof) comprising at least one physical and tangible processor and physical and tangible memory capable of having computer-executable instructions that can be executed by the processor. The memory can take any form and may depend on the nature and form of the computing system. A computing system can be distributed across a network environment and may comprise multiple configuration computing systems.

[0074]

[0086] As shown in Figure 7, in its most basic configuration, the computing system 700 typically includes at least one hardware processing unit 702 and memory 704. The processing unit 702 may include a general-purpose processor, a rewritable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other dedicated circuit. The memory 704 may be physical system memory, which may be volatile, non-volatile, or any combination of both. The term “memory” may also be used herein to refer to non-volatile mass storage, such as a physical storage medium. If the computing system is distributed, the processing, memory, and / or storage functions may also be distributed.

[0075]

[0087] The computing system 700 also has several structures on it, often referred to as “executable components.” For example, the memory 704 of the computing system 700 is shown as containing an executable component 706. The term “executable component” is a name for a structure that is well known to those skilled in the field of computing, and may be software, hardware, or a combination thereof. For example, when implemented in software, those skilled in the field will understand that the structure of an executable component may include software objects, routines, methods, etc., that can be executed on the computing system, whether such executable component resides in the computing system’s heap or on a computer-readable storage medium.

[0076]

[0088] In that case, a person skilled in the art will recognize that the structure of the executable component exists on a computer-readable medium such that it causes the computing system to perform functions when interpreted by one or more processors of the computing system (e.g., by processor threads). Such a structure may be directly computer-readable by the processor (as it would be if the executable component were a binary). Alternatively, this structure may be configured and / or compiled (in one or more stages) to produce such a binary that is directly interpretable by the processor. When using the term “executable component,” such an understanding of examples of the structure of an executable component is well within the understanding of a person skilled in the art of computing.

[0077]

[0089] The term “executable component” is more commonly understood by those skilled in the art to include structures such as hardcoded or hardwired logic gates that are implemented exclusively or substantially exclusively within hardware, such as in a rewritable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other dedicated circuit. Thus, whether implemented in software, hardware, or a combination of these, the term “executable component” is a term for structures commonly understood by those skilled in the art of computing. In this explanation, terms such as “component,” “agent,” “manager,” “service,” “engine,” “module,” and “virtual machine” may also be used. When used in this explanation and in these examples, these terms (whether with a modifying clause or not) are intended to be synonymous with the term “executable component” and therefore also to include structures commonly understood by those skilled in the art of computing.

[0078]

[0090] The above description has described embodiments relating to actions performed by one or more computing systems. When such actions are implemented in software, one or more processors (of the computing system performing the action) direct the operation of the computing system in response to the execution of a computer executable instruction that constitutes an executable component. For example, such a computer executable instruction may be embodied in one or more computer-readable media that form a computer program product. An example of such action involves the manipulation of data. When such actions are implemented exclusively or nearly exclusively in hardware, such as within an FPGA or ASIC, the computer executable instruction may be hardcoded or a hardwired logic gate. The computer executable instruction (and the data being manipulated) may be stored in the memory 704 of the computing system 700. The computing system 700 may also include a communication channel 708 that enables the computing system 700 to communicate with other computing systems, for example, over a network 710.

[0079]

[0091] While not all computing systems require a user interface, in some embodiments, the computing system 700 includes a user interface system 712 for use when interfacing with a user. The user interface system 712 may include an output mechanism 712A and an input mechanism 712B. The principles described herein are not limited to the output mechanism 712A or the input mechanism 712B, as they depend on the nature of the device. However, the output mechanism 712A may include, for example, a speaker, display, haptic output, hologram, etc. Examples of the input mechanism 712B may include, for example, a microphone, touchscreen, hologram, camera, keyboard, mouse, or other pointer input, any type of sensor, etc.

[0080]

[0092] As will be discussed in more detail below, embodiments described herein may include or utilize a dedicated or general-purpose computing system, which may include, for example, one or more processors and system memory, among other computer hardware. Embodiments described herein also include physical and other computer-readable media for carrying or storing computer-executable instructions and / or data structures. Such computer-readable media may be any available media accessible by the general-purpose or dedicated computing system. A computer-readable medium that stores computer-executable instructions is a physical storage medium. A computer-readable medium that carries computer-executable instructions is a transmission medium. Thus, embodiments of the present invention may include, but are not limited to, at least two distinctly different types of computer-readable media: storage media and transmission media.

[0081]

[0093] Computer-readable storage media can be used to store desired program code means in the form of computer-executable instructions or data structures, and include RAM, ROM, EEPROM, CD-ROM, or other optical disk storage, magnetic disk storage, or other magnetic storage devices, or any other physical and tangible storage media accessible by a general-purpose or dedicated computing system.

[0082]

[0094] "Network" is defined as one or more data links that enable the transport of electronic data between computing systems and / or modules and / or other electronic devices. When information is transferred to or provided to a computing system over a network or another communication connection (hardwired, wireless, or a combination of hardwired and wireless), the computing system appropriately considers that connection as a transmission medium. A transmission medium can be used to carry desired program code means in the form of computer-executable instructions or data structures and may include networks and / or data links accessible by a general-purpose computing system or a dedicated computing system. Any combination of the above shall also be included in the scope of computer-readable medium.

[0083]

[0095] Furthermore, program code, in the form of computer-executable instructions or data structures, can be automatically transferred from a transmission medium to a storage medium (or vice versa) upon reaching various computing system components. For example, computer-executable instructions or data structures received over a network or data link can be buffered in RAM within a network interface module (e.g., a "NIC") and then gradually transferred to the computing system's RAM and / or to less volatile storage media within the computing system. Therefore, it should be understood that storage media can be included in computing system components that also (or primarily) utilize transmission media.

[0084]

[0096] Computer executable instructions include instructions and data that, when executed, for example in a processor, cause a general-purpose computing system, a dedicated computing system, or a dedicated processing unit to perform a certain function or set of functions. Alternatively, computer executable instructions can configure a computing system to perform a certain function or set of functions. Computer executable instructions may be binary or instructions that undergo some transformation (such as compilation) before being directly executed by the processor, such as intermediate form instructions in assembly language or source code.

[0085]

[0097] While this content has been described in language specific to structural features and / or methodological actions, it should be understood that the content defined in the attached claims is not necessarily limited to the described features or actions. Rather, the described features and actions are disclosed as examples of forms that implement the claims.

[0086]

[0098] Those skilled in the art will understand that the present invention can be implemented in network computing environments having many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, portable devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile phones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses), etc. The present invention can also be implemented in distributed system environments where both local and remote computing systems linked over a network (by hardwired data links, wireless data links, or a combination of hardwired and wireless data links) perform tasks. In a distributed system environment, program modules may reside in both local and remote memory storage devices.

[0087]

[0099] Those skilled in the art will also understand that the present invention can be practiced within a cloud computing environment. While a cloud computing environment may be distributed, it is not mandatory. If distributed, the cloud computing environment may be internationally distributed within an organization and / or may have components maintained across multiple organizations. In this description and the accompanying claims, “cloud computing” is defined as a model for enabling on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services). The definition of “cloud computing” is not limited to any of the many other benefits that can be obtained from such a model when properly implemented.

[0088]

[0100] The remaining drawings may discuss various computing systems that may correspond to the computing system 700 described above. The computing systems of the remaining drawings include various components or functional blocks that can implement the various embodiments disclosed herein described herein. The various components or functional blocks may be implemented on a local computing system or on a distributed computing system that includes elements in the cloud or implements aspects of cloud computing. The various components or functional blocks may be implemented as software, hardware, or a combination of software and hardware. The computing systems of the remaining drawings may include more or fewer components than those shown, and some components may be combined as needed. Although not necessarily shown, the various components of the computing system may access and / or utilize processors and memory, such as processor 702 and memory 704, when necessary to perform their various functions.

[0089]

[0101] In the processes and methods disclosed herein, the operations performed within the processes and methods may be implemented in a different order. Furthermore, the operations briefly described are merely examples, and some of the operations can be selectively performed, combined with fewer steps and operations, supplemented with further operations, or extended to additional operations without deviating from the essence of the disclosed embodiments.

[0090]

[0102] The present invention can be implemented in other specific forms without departing from the spirit or characteristics of the invention. The embodiments described should be considered in all respects as illustrative and not limiting. Accordingly, the scope of the invention is indicated not by the above description but by the appended claims. Any changes that fall within the meaning and scope of equivalence of the claims are incorporated into the claims.

Claims

1. At least one processor, A computing system including at least one hardware memory device, The at least one hardware storage device is Initialize the service partition. The first hypervisor loads the second hypervisor into the service partition. While the second hypervisor is loaded within the service partition, the state of the first hypervisor is synchronized with the state of the second hypervisor, and Transferring control of hardware resources from the first hypervisor to the second hypervisor, wherein the transfer includes devirtualizing the second hypervisor from the service partition. The system stores instructions that can be executed by at least one processor in order to cause the computing system to perform the following: Computation system.

2. The computing system according to claim 1, wherein the second hypervisor acquires control of the hardware resources only after complete synchronization has been achieved between the state of the first hypervisor and the state of the second hypervisor.

3. The computing system according to claim 1, wherein the second hypervisor acquires control of the hardware resources before complete synchronization is achieved between the state of the first hypervisor and the state of the second hypervisor.

4. The computing system according to claim 1, wherein memory resources are allocated to the service partition and processor resources are allocated to the service partition.

5. The computing system according to claim 1, wherein the first hypervisor initializes the service partition.

6. The computing system according to claim 1, wherein the service partition is initialized by a highly reliable computing processing mechanism.

7. The computing system according to claim 6, wherein the first hypervisor is included as part of the high-reliability computing processing mechanism.

8. The computing system according to claim 1, wherein the second hypervisor is devirtualized from the service partition.

9. The computing system according to claim 1, wherein the first hypervisor is configured to generate a loader block for the second hypervisor, which is a logical construct describing static system characteristics and resources for initial setup of the second hypervisor.

10. One or more processors of the computing system The first hypervisor is instructed to perform the initial setup of the service partition. Loading a second hypervisor into the aforementioned service partition, While the second hypervisor is loaded onto the service partition, the state of the first hypervisor is synchronized with the state of the second hypervisor, and Transferring control of hardware resources from the first hypervisor to the second hypervisor, wherein the transfer includes devirtualizing the second hypervisor from the service partition. Methods that include...

11. The method according to claim 10, wherein the first hypervisor delegates control of all physical hardware to the second hypervisor.

12. The method according to claim 10, wherein the state of the virtual machine associated with the first hypervisor is restored by the second hypervisor.

13. The method according to claim 10, wherein the state of the first hypervisor includes one or more of a static system state, a dynamic system state, or a logical software state.

14. The method according to claim 10, wherein the state of the first hypervisor is the runtime state.

15. The method according to claim 10, wherein the state of the second hypervisor is the runtime state.

16. The method according to claim 10, wherein synchronizing the state of the first hypervisor with the state of the second hypervisor involves the orchestrator collecting the runtime state of the first hypervisor and sharing the runtime state with the second hypervisor on one or more inverse hypercalls.

17. At least one processor, A computer system including at least one hardware storage device, The at least one hardware storage device is Initialize the service partition. The first hypervisor loads the second hypervisor into the service partition. While the second hypervisor is loaded within the service partition, the state of the first hypervisor is synchronized with the state of the second hypervisor, and By devirtualizing the second hypervisor from the service partition, the second hypervisor will perform operations that were previously performed by the first hypervisor. The computer system stores instructions that can be executed by at least one processor in order to perform the above. Computer system.

18. The computer system according to claim 17, wherein the service partition is a partition included as part of the computer system.

19. The computer system according to claim 17, wherein, after the synchronization of the state of the first hypervisor and the state of the second hypervisor, additional incoming hypercalls to the first hypervisor are synchronized to the second hypervisor.

20. The computer system according to claim 17, wherein the second hypervisor is devirtualized before complete synchronization is achieved between the state of the first hypervisor and the state of the second hypervisor.

21. At least one processor, A computing system including at least one hardware memory device, The at least one hardware storage device is Initialize the service partition within the computing system using the first hypervisor of the computing system. Initialize the second hypervisor in the service partition using the system invariants of the first hypervisor and the computing system. Synchronizing the runtime state of the first hypervisor with the second hypervisor, and By devirturing the second hypervisor from the service partition, the first hypervisor is replaced with the second hypervisor. To cause the computing system to perform the above, it stores computer executable instructions that can be executed by at least one processor. Computation system.

22. The computing system according to claim 21, wherein the first hypervisor is executed at the highest privilege level defined by the system architecture of the computing system.

23. The computing system according to claim 21, wherein the computing system includes a host virtual machine system on which the first hypervisor is installed, and when the second hypervisor is initialized, the computing system runs a host operating system.

24. The computing system according to claim 21, wherein the processor resources of the at least one processor and the memory resources of the at least one hardware storage device are allocated to the service partition.

25. The computing system according to claim 24, wherein allocating the processor resources includes specifying the total amount of processing power available to the service partition.

26. The computing system according to claim 24, wherein allocating the processor resources includes designating a specific processor as a processing resource.

27. The computing system according to claim 24, wherein allocating the memory resources includes specifying the amount of memory available to the service partition.

28. The computing system according to claim 21, wherein initializing the service partition includes a bootstrapping operation.

29. The computing system according to claim 21, wherein the highly reliable computing processing mechanism of the computing system facilitates the initial setup of the service partition.

30. The computing system according to claim 21, wherein an operating system (OS) is loaded into a guest partition, and the guest partition is also initialized using the first hypervisor.

31. The computing system according to claim 21, wherein initializing the second hypervisor includes providing the second hypervisor with read-only access to physical resources.

32. The computing system according to claim 21, wherein the runtime state of the first hypervisor includes the hardware architecture state of the computing system.

33. The computing system according to claim 21, wherein communication between the first hypervisor and the second hypervisor is coordinated via an intermediate computing entity of the computing system.

34. A method for implementing a hypervisor hot restart, wherein one or more processors in a computing system, The first hypervisor is instructed to perform the initial setup of the service partition within the computing system. The first hypervisor is instructed to construct a loader block for the second hypervisor within the service partition. The first hypervisor is instructed to perform the initial setup of the second hypervisor within the service partition. The first hypervisor is instructed to transfer the runtime state of the first hypervisor to the second hypervisor, and Replacing the first hypervisor with the second hypervisor, wherein replacing the first hypervisor with the second hypervisor includes devirtualizing the second hypervisor from the service partition. Methods that include...

35. The method according to claim 34, wherein the runtime state includes at least one of a hardware architecture state, a virtualization hardware state, or a software-defined state related to the first hypervisor.

36. The method according to claim 34, wherein the runtime state includes at least one of a dynamic system state and a logical software state.

37. The method according to claim 36, wherein the runtime state includes the dynamic system state, the dynamic system state includes the hardware architecture state visible to the guest, and the hardware architecture state includes information relating to general-purpose registers and control registers.

38. The method according to claim 34, wherein transferring the runtime state of the first hypervisor to the second hypervisor includes synchronizing at least one of the second-level memory pages of the computing system or the page frame number database of the computing system with the second hypervisor.

39. A method for implementing the replacement of a first hypervisor with a second hypervisor in a computing system, The first hypervisor is instructed to perform the initial setup of the service partition within the computing system, wherein the service partition omits the operating system. The first hypervisor is instructed to construct a loader block for the second hypervisor within the service partition. The first hypervisor is instructed to perform the initial setup of the second hypervisor within the service partition, wherein the initial setup uses the information contained in the loader block. The first hypervisor synchronizes the runtime state of the first hypervisor with the second hypervisor, and By devirturing the second hypervisor from the service partition, the first hypervisor is replaced with the second hypervisor. Methods that include...

40. The method according to claim 39, wherein the first hypervisor is part of the highly reliable computing processing mechanism of the computing system.

Images