Multicore architecture and virtual machines

Abstract

A computer system having a multi-kernel architecture comprising multiple operating system microkernels is described. The computer system comprises multiple processing cores, a memory comprising multiple private memory regions and a shared memory region, and multiple compute nodes each comprising one or more processing cores and further comprising one of the multiple operating system microkernels and an associated user space for non-kernel operations, wherein only the operating system microkernels run at the highest privilege level of the one or more processing cores of the compute nodes. Each of the multiple compute nodes has an associated private memory region from the multiple private memory regions for use by the operating system microkernel of the each of the multiple compute nodes, and for non-kernel operations, each of the multiple compute nodes accesses the shared memory region. Associated methods and computer systems are also described.

Description

Technical Field

[0001] This disclosure relates to a multi-core architecture and virtual machine. Embodiments particularly relate to server architectures suitable for providing services (especially cloud services) over a network connection. Background Technology

[0002] Cloud computing involves the on-demand availability of computing system resources without requiring direct user management—cloud services are services provided to users using the cloud computing model. Cloud services are typically provided to users through cloud servers—this usually involves providing and using virtual machines to deliver cloud services. Some cloud services contain sensitive information and may need to operate in network environments with potential for compromise. For such cloud services, provable security and protection of user data privacy are particularly desirable. It is also desirable that the user space of the cloud server operate dynamically, rather than being completely statically defined at compile time. This presents challenges to cloud services requiring provable security. Summary of the Invention

[0003] In a first aspect, the present invention provides a computer system with a multi-kernel architecture, the multi-kernel architecture including multiple operating system microkernels, the computer system comprising: multiple processing cores; memory including multiple private memory regions and shared memory regions; and multiple computing nodes, each computing node including one or more of the processing cores, and further including one of the multiple operating system microkernels and an associated user space for non-kernel operations, wherein only the operating system microkernel runs at the highest privilege level of one or more processing cores of the computing node; wherein each of the multiple computing nodes has an associated private memory region from the multiple private memory regions for use by the operating system microkernel of each of the multiple computing nodes, and wherein, for non-kernel operations, each of the multiple computing nodes accesses the shared memory region.

[0004] In one embodiment, the computer system may be adapted to statically partition the system hardware into nodes at startup.

[0005] In some embodiments, one of the plurality of computing nodes may be a management node for managing user nodes, and one or more other computing nodes may be user nodes for running user workloads. In some embodiments, the plurality of computing nodes may be management nodes. In some embodiments, the computer system may be adapted to provide one or more virtual machines to users. In some cases, no more than one virtual machine is implemented in any user node. In other cases, two or more mutually trusted virtual machines may be implemented in one or more user nodes. One or more virtual machines may be implemented using multiple user nodes. In some embodiments, the computer system may be adapted to operate as a cloud server.

[0006] The management node may include a manager component, and each user node may include an agent, wherein the node management component is adapted to instruct the agent of the user node to create, destroy, or modify virtual machines within that user node. In some embodiments, communication between user nodes is not privileged—that is, not at the privilege level between privileged components—but communication between the manager component and one or more agents is privileged, i.e., communication between privileged components. Direct communication between nodes may be limited to direct communication between the user spaces of these nodes.

[0007] In this embodiment, each user node may have a virtual machine manager component for managing virtual machines within that user node through interaction with the manager component. In some cases, a virtual machine manager is provided for each virtual machine in the user node. Each user node may include a console driver, and the management node may include a console relay for communication between the management node and the virtual machine manager. Each virtual machine manager component may be statically defined at startup for subsequent configuration.

[0008] In some embodiments, the computer system further includes one or more network devices, wherein the management node includes a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; and wherein one or more of the user nodes include a virtual function device driver for a virtual function network device, corresponding to one of the one or more network devices, wherein the virtual function device driver provides virtual network device functionality in the user node. Such a physical function device driver may be provided by an SR-IOV mechanism. In some cases, the computer system may also include one or more network devices, wherein one or more of the network devices are storage devices, and data may then be encrypted for communication between the virtual machine and the storage device.

[0009] In one embodiment, the computer system further includes a remote management interface that includes a dedicated network link. In another embodiment, each microkernel may have proven implementation correctness. In yet another embodiment, each microkernel may be a third-generation microkernel, such as the seL4 microkernel.

[0010] In a second aspect, the present invention provides a method for operating a computer system including multiple processing cores and memory to implement a multi-core architecture, the method comprising: defining a management computing node having one or more processing cores; starting the management computing node and statically partitioning the processing cores into multiple additional computing nodes, each computing node including one or more processing cores and having an operating system microkernel and an associated user space, wherein only the operating system microkernel runs at the highest privilege level of one or more processing cores of the computing node; and starting and provisioning the additional computing nodes as user nodes.

[0011] In an embodiment, the method may further include: establishing the memory as a plurality of private memory regions and a shared memory region, wherein each of the plurality of computing nodes is provided with an associated private memory region from the plurality of private memory regions for use by the operating system microkernel of each of the plurality of computing nodes, and wherein some or all of the plurality of computing nodes can access the shared memory region for non-kernel operations. Such a user node may be adapted to provide one or more virtual machines to a user. In some cases, no more than one virtual machine may be implemented in any user node. In other cases, two or more virtual machines may be implemented in one or more user nodes. One or more virtual machines may be implemented using multiple user nodes.

[0012] In one embodiment, the computer system may be adapted to operate as a cloud server.

[0013] In an embodiment, the management node may include a manager component, and each user node may include an agent, wherein the method may further include: the node management component instructing the agent of the user node to create, destroy, or modify a virtual machine in the user node.

[0014] Each user node may be configured with a virtual machine manager component for managing virtual machines within that user node through interaction with the manager component. In one embodiment, the method may further include: statically defining each virtual machine manager component at startup. In another embodiment, the method may further include: configuring the virtual machine manager component of the user node when a virtual machine is created in the user node.

[0015] The computer system may further include one or more network devices, wherein the management node may include a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; and wherein one or more of the user nodes may include a virtual function device driver for interacting with a virtual function network device, the virtual function network device corresponding to one of the one or more network devices, and the method may further include: the virtual function device driver providing virtual network device functionality in the user node. In a further embodiment, one or more of the network devices may be a storage device, and the method may further include: encrypting data for communication between the virtual machine and the storage device.

[0016] In this embodiment, all communication between nodes can be performed by the node's userspace component. Any privileged communication between nodes can occur between the manager node and another node, where privileged communication includes messages and commands that cause some components within the node to reconfigure.

[0017] In a third aspect, a method is provided for providing a dynamic user space in a computer system, wherein the computer system is adapted for virtualization, the method comprising: statically defining a plurality of computing nodes, each of the computing nodes including an operating system kernel, wherein one of the plurality of computing nodes is a management node, and one or more other computing nodes of the plurality of computing nodes are user nodes; wherein the management node includes a manager component for managing the user nodes, and wherein the user nodes or each user node includes an agent; and wherein the manager component instructs the agent of the user node to provide the creation, destruction, or modification of virtual machines in the user node, thereby providing a dynamic user space in the user node.

[0018] In this embodiment, the operating system kernel of the compute node may be a microkernel. In this embodiment, only the operating system kernel runs at the highest privilege level within the compute node.

[0019] In this embodiment, multiple management nodes may exist. In this embodiment, the manager component may be adapted to receive management requests from a remote management interface including a dedicated network link. In this embodiment, the manager component may determine the allocation of all memory for the virtual machine.

[0020] In an embodiment, the method may further include: creating a virtual machine in a user node via instructions from a manager component, the instructions including an identifier of memory to be used by the virtual machine and a physical device to be passed through to the virtual machine. The virtual machine may be created in a stopped state. The method may further include: the manager component instructing an agent to start the virtual machine.

[0021] In one embodiment, the user node may include a virtual machine manager component, and the method may further include configuring the virtual machine manager component for use with the virtual machine to be created. Such an embodiment may also include statically creating the virtual machine manager component as a shell component at system startup for configuration during virtual machine creation.

[0022] In this embodiment, each user node may include a console driver, and the management node may include a console relay for communication between the management node and the virtual machine manager.

[0023] Communication between user nodes can be unprivileged, but communication between the manager component and one or more agents can be privileged (i.e., between privileged components). Direct communication between nodes can be limited to direct communication between the user spaces of those nodes.

[0024] In an embodiment, the computer system may further include one or more network devices, wherein the management node may include a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; the method may then further include providing a virtual function device driver for a virtual function network device at one or more user nodes, the virtual function network device corresponding to one of the one or more network devices, wherein the virtual function device driver then provides virtual network device functionality in the user nodes. The physical function device driver may be provided by an SR-IOV mechanism. One or more of the network devices may be storage devices, and the method may further include encrypting data for communication between the virtual machine and the storage device.

[0025] In a fourth aspect, the present invention provides a computer system suitable for virtualization using dynamic user space, the computer system comprising: a plurality of computing nodes, each of the computing nodes including an operating system kernel, wherein one of the plurality of computing nodes is a management node, and one or more other computing nodes of the plurality of computing nodes are user nodes; the management node includes a manager component for managing user nodes, and wherein the user node or each user node includes an agent; and wherein the computer system is adapted to instruct the agent of the user node through the manager component to provide the creation, destruction or modification of virtual machines in the user node, providing dynamic user space in the user node.

[0026] Each compute node may include one or more processing cores. The operating system kernel may be a microkernel. In some embodiments, only the operating system kernel runs at the highest privilege level.

[0027] Communication between user nodes can be unprivileged, but communication between the manager component and one or more agents can be privileged. Direct communication between nodes can be limited to direct communication between the user-space components of those nodes.

[0028] In a fifth aspect, the present invention provides a computer system with a multi-kernel architecture, the multi-kernel architecture including multiple operating system kernels, wherein the computer system is adapted to provide and manage virtual machines, the computer system comprising: multiple processing cores; multiple computing nodes, wherein each computing node is assigned one or more processing cores, and wherein each computing node includes an operating system kernel, and wherein one of the multiple computing nodes is a management node for managing user nodes, and one or more other computing nodes of the multiple computing nodes are user nodes; wherein the management node includes a manager component, and wherein each user node includes a virtual machine manager component for managing virtual machines in the user node through interaction with the manager component.

[0029] In this embodiment, each virtual machine manager component can be statically defined at startup for subsequent configuration. The virtual machine manager component can also be statically defined as a shell to be populated when virtual machines are created on user nodes.

[0030] In one embodiment, a virtual machine manager may be provided for each virtual machine in a user node. In another embodiment, the computing system may include multiple management nodes. In yet another embodiment, each user node may include a console driver, and each management node may include a console relay for communication between the management node and the virtual machine manager.

[0031] In one embodiment, the management node may include a manager component, and each user node may include an agent, wherein the node management component is adapted to instruct the user node's agent to create, destroy, or modify virtual machines within that user node. The user node's agent may then be adapted to configure the virtual machine manager component for that user node when a virtual machine is created.

[0032] In this embodiment, the operating system kernel of the compute node is a microkernel. In this embodiment, only the microkernel can run at the highest privilege level of the CPU cores. In this case, the virtual machine manager component may not run at the highest privilege level.

[0033] In a sixth aspect, the present invention provides a computer system suitable for virtualization using network devices, the computer system comprising: a plurality of processing cores; a plurality of computing nodes, wherein each computing node is assigned one or more processing cores and wherein each computing node includes an operating system kernel, and wherein one of the plurality of computing nodes is a management node, and one or more other computing nodes of the plurality of computing nodes are user nodes; and one or more network devices; wherein the management node includes a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; and wherein one or more of the user nodes includes a virtual function device driver for a virtual function network device, the virtual function network device corresponding to one of the one or more network devices, wherein the virtual function device driver provides virtual network device functionality in the user node.

[0034] In this embodiment, the physical function device driver may be provided by the SR-IOV mechanism. In some cases, one or more of the network devices may be storage devices, and data may be encrypted for communication between the virtual machine and the storage device.

[0035] In some embodiments, the operating system kernel of a computing node may be a microkernel. In this case, in some embodiments, only the microkernel can run at the highest privilege level of the CPU cores.

[0036] In some embodiments, the computing system includes multiple management nodes. Attached Figure Description

[0037] One or more embodiments of this disclosure will now be described by way of example only with reference to the accompanying drawings, in which:

[0038] Figure 1 compares monolithic architecture and microkernel architecture; Figure 2 illustrates existing operating systems using a multi-kernel architecture; and Figure 3 An operating system according to an embodiment of the present invention is shown; Figure 4 This shows the result after a single VM has been created on node 1. Figure 3 system; Figure 5 This demonstrates how to create a VM using two nodes. Figure 3 system; Figure 6 It shows the use of Figure 3 The startup strategy of the example; Figure 7 The cross-node communication link used in some embodiments is shown; Figure 8 It shows Figure 3 Network device management in the embodiments; Figure 9 It shows Figure 3 External storage usage in the embodiments; Figure 10 It shows Figure 3 The remote management structure in the embodiment; Figure 11 The combination is shown Figure 10 The deployment uses PKI for remote management; Figure 12 An operating system according to an alternative embodiment of the present invention is shown; Figure 13 It shows that it can be used Figure 12 The server architecture shown implements a hybrid static / dynamic software stack; and Figure 14 This demonstrates the management of virtual machines spanning two nodes. Detailed Implementation

[0039] General and specific embodiments of this disclosure will now be described with reference to the accompanying drawings.

[0040] The inventors recognize that cloud server designs based on a multi-kernel architecture using a microkernel are particularly effective for achieving provably secure cloud services and ensuring user data privacy. First, a microkernel will be described, with particular reference to the seL4 operating system microkernel used in embodiments of the invention. Then, a multi-kernel architecture using a microkernel will be discussed, along with a hypervisor designed for creating and managing virtual machines on such an architecture. Specific embodiments of the invention will then be described, relating to architectural implementation options and associated hypervisor design. As described below, these are particularly suitable for use with cloud servers providing cloud services, as will be described below.

[0041] Throughout this specification, the following terms are used for definition—for each term, the first sentence is the working definition, and subsequent sentences provide the necessary explanations: The kernel is the core of a computer system's operating system (OS)—it typically controls everything within that system, mitigates conflicts between processes, and facilitates interactions between components. The kernel is low-level system software—executing in a higher-privilege CPU mode—that controls the computer system's resources and enforces security. In most systems, the kernel controls all hardware resources (such as I / O, memory, and cryptographic services) through device drivers, arbitrates conflicts between processes regarding these resources, and optimizes the utilization of shared resources (such as CPU and cache usage, file management, and network sockets).

[0042] Resources - This includes anything that can be used by or is used within a computer system, including CPU cores, memory regions, and devices, unless otherwise stated.

[0043] Node - This is a set of resources exclusively controlled by a single OS kernel.

[0044] Multi-kernel – This is an OS architecture that includes multiple kernels running concurrently. In different architectures (or different arrangements), these kernels will have varying degrees of independence from each other.

[0045] Microkernel - This is an OS architecture that comprises a minimal amount of software that executes at a high CPU privilege level to implement the OS, typically including address space management, thread management, and inter-process communication. Many components that are usually hosted in the kernel (such as device drivers, file systems, and networking) are instead implemented in user space and executed at a low CPU privilege level.

[0046] User space – This includes a set of OS components that run at low CPU privilege levels (e.g., x86 ring 3, ARM EL0), in contrast to the OS kernel, which is the only component that runs at high CPU privilege levels (e.g., x86 ring 0, ARM EL1 / EL2).

[0047] Symmetric Multiprocessing (SMP) - This is an OS architecture where a single OS runs on multiple CPU cores.

[0048] Virtual machine (VM) - A virtualized computer system managed by a virtual machine monitor (VMM).

[0049] A Virtual Machine Monitor (VMM) is a system and process that supports the management and execution of virtual machines (part of the virtualization stack). A VMM emulates hardware components that cannot virtualize themselves—for example, by implementing virtual network interfaces in software. This contrasts with self-virtualized hardware, such as the SR-IOV network interface, which presents an independent, isolated personality for each VM.

[0050] Hypervisor - The system and processes used to manage the kernel space functions (part of the virtualization stack) of a virtual machine system. Hypervisors typically execute at a higher privilege level than the OS kernel. Their primary task is to configure those parts of the system capable of self-virtualization—for example, by partitioning memory among virtual machines through two-stage transitions. By extension, the term "hypervisor" is also used to describe the complete virtualization stack.

[0051] Private resources - Resources that can only be accessed by a single node.

[0052] Shared resources - Resources that are accessible to one or more nodes.

[0053] Privileged communication - Communication between privileged components.

[0054] Unlike applications, the kernel has exclusive access to a more privileged execution mode of the processor—kernel mode—which allows it direct access to the hardware. Applications execute only in user mode and can only access hardware if the kernel allows it. Therefore, the kernel's role is particularly important for secure operation. Traditional operating systems (such as Windows or Linux) use large and complex monolithic kernels. While controlling so much functionality through the kernel has some advantages for application programming—meaning the ways applications can access computer system resources are well-defined—it also has significant drawbacks. Such kernels can be difficult to verify and error-prone, and the presence of these errors can easily lead to complete system corruption. Another approach involves using microkernels. Microkernels implement only the minimum basic primitives that allow an operating system to be built on top of them, minimizing the amount of code that executes with higher privileges.

[0055] Figure 1 illustrates a comparison of these architectures. The traditional kernel architecture has a large kernel 1 located above hardware layer 3, with stacked functionality (from device drivers 1a at the bottom of the stack to the virtual file system at the top 1d), and applications are built on application layer 2, accessing kernel 1 via system calls—only applications are in user mode 7; a large amount of functionality is handled by the kernel running in kernel mode 8. The microkernel architecture has a much smaller kernel 10 handling virtual memory and inter-process communication—all other functionality, such as not only applications 12, but also functionality in the traditional OS kernel (like device drivers 13 and servers like Unix servers 14 and file servers 15), is built on top of the microkernel 10, communicates with it using inter-process communication, and these functions run in user mode 7, not in kernel mode 8 or with kernel-mode privileges.

[0056] The embodiments disclosed herein use the seL4 operating system microkernel—seL4 is available as open-source code from GitHub and is supported by the seL4 Foundation—seL4 and its development are further described at https: / / sel4.systems. seL4 was developed as a third-generation microkernel to form the basis of highly secure and reliable systems. The characteristics of a third-generation microkernel are: a security-oriented API; resource access controlled through capabilities; virtualization as a primary concern; novel kernel resource management methods; and suitability for formal analysis. seL4 has been formally proven to be correctly implemented (with ongoing machine-checked proofs), and if configured correctly, the following properties are guaranteed: • Confidentiality – seL4 does not allow any entity without read access to read or infer the data; • Integrity – seL4 does not allow any entity to modify data unless that entity has write access to that data; and • Availability – seL4 does not allow one entity to block another entity from authorizing the use of a resource.

[0057] seL4 provides a small set of primitives for securely reusing hardware resources—these primitives embody as few policies as possible. The kernel provides mechanisms to allow operations requiring elevated privileges on a given machine to be invoked according to policies defined by non-privileged components. The kernel's security mechanisms are designed to allow mutually untrusted components to manage disjoint sets of resources without interference.

[0058] Each resource in the system is managed as an object, named by capabilities that represent permissions to it. seL4 uses "capabilities" for access control. Capabilities are access tokens that support very fine-grained control over which entity can access a specific resource in the system—capabilities both name the object of an operation and establish the permissions of the caller to perform the operation. Capabilities support strong security based on the principle of least privilege—users are granted the minimum access or permission required to perform their job functions. seL4 identifies the unit of permission and the unit of scheduling—the thread. Each thread has a defined scope of permissions, and threads typically initiate kernel operations by invoking their capabilities. This core design principle of a security system is not feasible in practice for access control types that require traditional operating systems such as Linux or Windows.

[0059] seL4 is a particularly suitable choice for use as a microkernel in the context of cloud service provision because it is capability-based and formally verified (and therefore secure), and also fast. The main machine resources managed by the kernel are CPU time and RAM. Access to the CPU is abstracted into kernel-visible threads, and RAM is abstracted into frames. The seL4 kernel is designed for lightweight execution with minimal kernel state (outside of explicit kernel objects). The kernel itself uses an atomic single-stack design. Calls either fail without modifying user-visible state or complete before any user thread is scheduled (with very limited exceptions for some long-running operations, but even there, state is stored in explicit kernel objects). Therefore, from the user space perspective, kernel execution simply appears as a series of atomic steps, without evaluating partially completed operations or storing the kernel execution context. The kernel uses a single, statically allocated stack that is simply overwritten on each call. This architecture relies on single-threaded execution within the kernel itself.

[0060] Although the seL4 microkernel was initially designed for single-core systems, it was subsequently developed to support multi-core systems. As multi-core CPUs became more available, support for symmetric multiprocessing (SMP) systems was later introduced. To achieve this in a single-threaded execution environment with a kernel, a locking strategy was required. seL4 employs a so-called Big Kernel Lock (BKL) strategy for SMP, rather than the finer-grained locking used in monolithic kernels like Linux or BSD. Fine-grained locking mechanisms attempt to maximize performance by identifying all critical sections, narrowing them down to shorter code paths, and protecting them with dedicated locks such as mutexes and spinlocks. BKL, on the other hand, attempts to minimize the complexity of managing execution concurrency by acquiring a global lock at the entry point of each system call and releasing it when the system call exits.

[0061] It has been found that for microkernels like seL4, the simple BKL mechanism is sufficient to scale the system to 4 or even 8 cores, depending on the load. This makes it suitable for embedded systems, but it doesn't seem to work well for cloud server applications. Modern cloud servers use CPUs with far more cores—an order of magnitude more—and on such systems, the BKL mechanism performs poorly because most CPU cores will compete for the global lock.

[0062] Another challenge is that the verification framework used to verify the seL4 kernel currently does not support concurrent execution—meaning it requires that system memory can only be modified by a single thread of execution. On SMP systems, memory can be modified independently by all concurrently running threads of execution (each CPU core), making this configuration unverifiable under the current framework. While seL4's BKL mechanism appears to satisfy the non-concurrent requirements within the kernel, the code for acquiring and releasing locks is itself unverifiable. Furthermore, the proof still requires modification to support multi-core systems.

[0063] This leaves two configuration options for seL4: • Single-core configuration, which can run a formally verified kernel, but can only use one core.

[0064] • Multi-core configuration, which cannot run formally verified kernels, but can be reasonably scaled to 4 cores or even 8 cores.

[0065] Neither of these configurations is considered suitable for building secure and high-performance cloud servers: it requires a configuration that can scale to more cores and reuse formal proofs of single-core kernels. To effectively utilize multi-core systems, it is desirable not only to use a multi-core configuration of kernels, but also to use multiple kernels across different cores or sets of kernels. This constitutes a multi-core architecture. In a multi-core architecture, common and performance-critical operations are typically handled locally without communication or synchronization between computing units (such as discrete CPU cores or devices like network interface cards (NICs)). A separate OS instance runs on each device—for example, the seL4 microkernel. State consistency—such as memory allocation—is then handled by a distributed protocol running outside the kernel.

[0066] This effective partitioning of the OS across kernels does not prevent user-level applications or virtual machines from spanning multiple kernels. Coordination protocols can be used to provide separate instantiation of applications on different kernels, and then the applications execute as if on any OS, with their threads coordinated via shared memory and inter-kernel signals. OS-level inter-kernel communication is typically not used; communication between user nodes is performed solely by user-space tasks.

[0067] Existing multi-core architectures include Barrelfish and clustered multi-core architectures. Barrelfish, as shown in Figure 2, is an OS architecture that implements the multi-core OS architecture proposed by Timothy Roscoe's research group at ETH Zurich. Barrelfish is designed to solve the scalability problem of common SMP systems by treating the underlying hardware as a distributed system, where each CPU core runs its own kernel instance and communicates with each other using explicit messages.

[0068] As shown in Figure 2, Barrelfish implements a single kernel instance 21 running on each CPU core 22. Each kernel 21 manages the tasks running on its CPU core 22. User-space tasks can communicate with other user-space tasks on all other kernels 21 to maintain the illusion of a single unified system—CPU cores 22 are connected via physical interconnect 20 and communicate via asynchronous messages 23. Complex algorithms run on each OS node to efficiently maintain a local copy of the state 25. The application programming model of this type of OS is broadly similar to that of traditional SMP systems such as Linux: applications 24 are dynamically created, threads can migrate between cores, and multithreaded applications can even have threads running concurrently on different cores.

[0069] Clustered Multikernel (CMK) was proposed by Michael von Tessin in his 2013 paper, using a modified version of the seL4 kernel. In the CMK model, the machine is divided into multiple clusters, each consisting of one or more CPU cores and a fixed amount of system memory, with one seL4 instance generated in each cluster. All inter-node communication occurs outside the seL4 kernel—it is directly managed by user-space tasks using memory shared between the nodes and inter-processor interrupts (IPIs). This implementation assumes a static system where hardware partitioning for clusters, communication channels, and user-space applications are all defined at compile time. This implementation is much simpler than Barrelfish, with its scope primarily focused on verification.

[0070] The primary function of the cloud server software stack is to enable large physical machines to be dynamically partitioned into multiple virtual machines (VMs) that can run end-user operating systems as workloads. VMs are allocated resources such as storage regions, dedicated physical devices (passthrough), and virtual devices such as virtual network links and virtual storage devices. Therefore, the basic unit of computing is the VM, a much simpler management object than traditional user applications: its resources and connections are defined at creation time and fixed for the duration of its lifecycle. Cloud servers are devices that are remotely managed using application programming interfaces (APIs). Cloud operators typically manage a large number of such cloud servers in data centers and manage them as clusters through a management console.

[0071] Such VMs will be managed by a suitable hypervisor—embodiments of the invention relate to providing an architecture and hypervisor suitable for providing highly secure cloud services. Such a secure cloud hypervisor can be designed to run mutually untrusted VMs on bare-metal cloud-scale hardware. Threats to be addressed include VM escape, hardware timing channels, and network compromise. Preferably, formal verification methods can be applied to demonstrate the correctness of the implementation and the required strong security properties, particularly the strict isolation of VMs.

[0072] The seL4 microkernel is particularly well-suited for this purpose because it comes with formal, mathematical, machine-checked proofs of implementation correctness. Specifically, seL4 provides specifications constituting correct behavior, a simplified hardware model, a C language implementation, and machine-checkable proofs (against the hardware model) demonstrating that the C language implementation corresponds to the specifications of correct behavior. It should be noted that embodiments of the invention are not limited to the use of the seL4 microkernel, but these features make implementations using seL4 particularly suitable.

[0073] The seL4 microkernel and its ecosystem have historically focused on embedded and network-physical devices, which are small systems based on ARM or RISC-V architectures, with a few central processing unit cores, a few gigabytes of memory, and statically defined at compile time. The architecture of this embodiment of the invention, implemented using seL4, extends the applicability and formal guarantees of seL4 to large cloud server machines running multiple virtual machines for cloud tenants.

[0074] The hypervisor according to embodiments of the present invention does not attempt to scale the seL4 kernel and its proofs to a large number of cores. Instead, it employs a multi-core system architecture in which multiple single-core seL4 instances run concurrently, each instance bound to a single dedicated CPU core and using a small amount of private memory. Furthermore, a large pool of shared memory is available for client VMs. In this configuration, the kernels do not need to coordinate or interact directly with each other. This multi-core architecture allows the system to scale to a very large number of cores without compromising the validity of its formal verification proofs. Strict partitioning of system resources also provides protection against side-channel and Rowhammer-type attacks.

[0075] Regarding this architecture, a distinction is made between the term hypervisor and the term virtual machine monitor. The hypervisor refers to the full kernel-mode system required to support the VM, while the virtual machine monitor is a non-privileged task running on a kernel and supporting exactly one VM running on the same kernel. This design enables the construction of secure systems with strong isolation between VMs using untrusted and isolated VMMs. The memory regions supporting the VMs are allocated from memory shared across all kernels. Multi-core guest VMs are implemented using a distributed VMM, where the virtual CPU of each VM is processed by separate VMM tasks running on different physical CPU cores.

[0076] Such a management procedure can be remotely managed using an API over a dedicated network link. Cloud operators typically manage a large number of such cloud servers in data centers and manage them as a cluster through a management console. The management procedure according to embodiments of the invention can, for example, interface with the storage server using the Internet Small Computer System Interface (iSCSI) protocol over a dedicated network link. In this embodiment, all data sent to the storage server is transparently encrypted by the management procedure. Network access can be provided using a dedicated network interface that supports virtualization capabilities such as Single Root Input / Output Virtualization (SR-IOV). The VMM can then implement the network interface VF device driver and the iSCSI initiator, exposing VirtIO block devices, network devices, and console devices to clients.

[0077] The primary function of this hypervisor is to enable large physical machines to be dynamically partitioned into multiple virtual machines (VMs) that can run end-user operating systems as workloads. VMs are allocated resources such as CPU cores, private memory regions, exclusive physical devices (passthroughs), and virtual devices (such as virtual network links and virtual storage devices). Therefore, the basic unit of computing is the VM, a much simpler management object than traditional user applications: its resources and interfaces with system components are defined at creation time and do not change until the VM stops running. The hypervisor according to embodiments of the invention uses multiple virtual machine controllers (VMMs), each running as a non-privileged user-space task and managing only the virtual CPUs (vCPUs) of a single VM. This design enables the construction of secure systems with strong isolation between VMs using untrusted and isolated VMMs.

[0078] In embodiments of the invention, a node-based approach can be used to provide a system architecture particularly well-suited for cloud server implementations. This requires providing a suitable virtualization platform or hypervisor. Using this approach, system hardware can be dynamically or statically partitioned into nodes at startup. In the simplest version of this approach, a single-core instance of the seL4 microkernel runs on each CPU core. Each instance—the kernel and its user space running on a single CPU core—is referred to as a node in a particular embodiment. More generally, a node can be considered a set of resources exclusively controlled by a single OS kernel. Other methods of node construction can be employed—a node may, for example, comprise one or more CPU cores, or a particular node may be individually located (e.g., the management node, as described below with respect to a particular embodiment, may be provided on another device, such as a PCIe card).

[0079] Nodes are granted exclusive access to a subset of system resources (such as physical memory and devices), shared access to memory regions used to implement communication channels between nodes, and the ability to share data between software components located on different nodes. The set of resources allocated to a node at creation can be fixed for the node's lifetime: in such an arrangement, a node cannot gain access to additional resources during its lifetime. In other arrangements, nodes may be able to grant or share resources, provided this can be accomplished through formally verifiable kernel capabilities.

[0080] This approach requires minimal changes to the seL4 microkernel and has a limited impact on provability. It does require a suitable hypervisor to manage the virtualization process, and a virtual machine manager (which can be incorporated into the hypervisor) to manage the virtual machines. Before discussing specific implementations, general principles for a suitable hypervisor will be described.

[0081] A suitable hypervisor, based on the sel4 microkernel and the guarantees provided by its verification, provides secure virtualization. This achieves isolation between untrusted virtual machines in a cloud-hosted environment. To this end, the hypervisor runs on virtualization-capable hardware maintained by the hosting provider. The hypervisor enables system administrators to control the allocation of virtual CPUs to physical CPUs. It provides the lowest level of system software, allowing untrusted VMs to coexist on the same physical hardware with the highest available isolation. Therefore, the hypervisor is a "bare-metal" hypervisor, reusing VMs on shared hardware only in its implementation (without a traditional operating system running at a higher privilege level). Thus, the guarantees provided by using sel4 are directly related to VM isolation.

[0082] In the embodiments described in detail below, the following choices are made. A system with a distributed Virtual Machine Monitor (VMM) exists, each VMM instantiating a single VM (but each VM can have multiple VMMs—one per virtual CPU core). Treating these as a single entity for security purposes mitigates the risk of the VMM being compromised by untrusted VMs. Memory is divided into system memory—partitioned across nodes and potentially containing seL4 objects—and shared memory, accessible from each node but only usable as a mappable frame. Both types of memory are represented by seL4 capabilities. System memory capabilities exist only on the node to which they belong—memory allocation is described in more detail below for specific embodiments. System memory is used to create all kernel objects required for system operation, while shared memory is provided for other uses.

[0083] For the use of physical resources, Single Root Input / Output Virtualization (SR-IOV) can be used to allow physical PCIe devices to present themselves multiple times, thereby achieving isolation between VMs. Interaction with external storage servers can be performed via the iSCSI protocol, with the content exposed to the VM as VirtIO devices. The hypervisor can handle the encryption and decryption of data hosted on an external storage server—which should be considered untrusted—on behalf of the VM, while the VM itself does not need to implement encryption or the iSCSI protocol. Remote management can be provided via an API, as described in the examples below.

[0084] In this way, the management program can provide the following functions: The front-end network provides general network access to guest VMs. The hypervisor uses a NIC that supports virtualization via SR-IOV. Guest VMs obtain network access through virtualization network devices mapped to dedicated physical network ports.

[0085] The backend network provides access to the storage server. The hypervisor implements the iSCSI protocol on a dedicated physical network port to communicate with the external storage server.

[0086] • Management network on a dedicated physical network port. The management program controls the network via an API accessed over an IP network. Authentication and authorization are delegated to a Public Key Infrastructure (PKI) managed by the hosting provider. The Atoll management program implements x509 certificates with specific extensions. The server's backplane management controller (BMC) can be exposed on a dedicated NIC or a shared management network NIC port.

[0087] As described below, in this embodiment, one node is used for system management, while other nodes serve as user nodes with user spaces running user workloads. Although the discussion below considers it as a single management node, in other embodiments there may be multiple management nodes—for example, physical hardware assets may be split across different management nodes, or different management nodes may be used for different purposes. Such management nodes either need to use different sets of resources or need to provide mechanisms to determine which management node is responsible for a specific resource. It is also possible that the system of one or more management nodes and user nodes described only describes a portion of the system—for example, additional nodes running alternative operating systems (such as single-core Linux) may be provided, or additional nodes may be hardware nodes—while these nodes do not have the properties described in the embodiments below, they can be integrated into a system that does indeed operate as described below.

[0088] Implementation examples of the architecture, along with related processes and operational methods, will now be described. These utilize hypervisors of the type described above to implement VMs in a virtualization platform using a multi-kernel architecture based on the seL4 microkernel. Such systems are particularly well-suited for implementing relatively long-running VMs that run network services such as email servers, DNS servers, file servers, application servers, or web servers.

[0089] Figures 3 to 11 and Figure 14 A specific embodiment of this architecture and related components and processes is illustrated. In this approach, the server hardware is statically partitioned into nodes at startup, and this partitioning remains fixed throughout the software execution. Services such as networking and storage are provided through appropriate hardware functionalities, such as the use of PCIe SR-IOV virtualization. Dynamism is provided by an agent, as described below.

[0090] In this approach, each node is started up at startup and remains active until the system shuts down. There are only two types of nodes: the root node (node ​​0) and all other nodes used for user load.

[0091] Node 0 runs on the boot processor core 30. It is a special node that performs system administration duties. It hosts a remote API implementation 305 bound to a dedicated network interface and receives administration commands from the system administrator. It also hosts a privilege manager component 302, which communicates with agents 312, 322, and 332 running on other nodes 31, 32, and 33 to execute requests from the system administrator. Node 0 does not perform user workloads. Node 0 also includes one or more physical function network drivers 304 for driving network devices and a console relay function 303 for out-of-band access.

[0092] All other nodes 31, 32, and 33 run on the remaining processor cores and are dedicated to running user workloads: VMs and their supporting tasks. Each regular node hosts privileged agent tasks 312, 322, and 332, which maintain communication with node 0 (here via circular buffer 38) and perform configuration requirements such as creating and tearing down VMs. Each node 31, 32, and 33 also hosts trusted console driver tasks 313, 323, and 333, which provide out-of-band access to VMs (here using circular buffer 37). The agent and console driver components are statically defined, while other components such as VMMs 315, 325, and 335, as well as supporting components, are dynamically created or recycled within the node.

[0093] Each node is provided with a small, exclusive amount of memory—this comes from a private memory region 401 within system memory 40. For simplicity and convenience, the size of this memory region is fixed and statically defined when the system image is assembled, so a node may not grow or shrink during its lifetime—unlike in other types of embodiments where resources can be dynamically transferred from one node to another. This memory is used to allocate the seL4 kernel itself (e.g., its text and data segments) and all kernel objects that the node will create during its lifetime. This memory is guaranteed to be node-private and exclusive: by design, no other place in the system has the ability to point to it; it exists only on the single node that owns it.

[0094] Each regular node also has access to a small memory region labeled Device Memory 402, dedicated to implementing the communication mechanism between that node and Node 0. These memory regions are fixed-size, statically created, and managed by the shared capabilities of the manager and console relay components on Node 0, as well as the local agents and console components on the node, effectively making these memory regions shared memory. The communication mechanism is bidirectional and point-to-point: agents communicate only with the manager on Node 0; they do not communicate with each other. Communication is entirely managed by user-space tasks, without any direct kernel involvement.

[0095] The remaining memory (except for the memory mapped to specific device 404), that is, memory not yet allocated as node-private memory or for fixed communication buffers between all nodes and node 0, is provided to all nodes as device memory 403. Memory marked as device memory for the seL4 microkernel can only be mapped into the address space as frames: it can never be used to create kernel objects. To enforce consistency in memory mapping attributes and sizes across all cores, memory is broken down into frame capabilities that cannot be further subdivided on any node. This large memory region is reserved for dynamically allocating memory for guest VMs, for any additional buffers required for inter-node communication, and for supporting emulated devices. Allocation is performed by the manager component on node 0 and by the agent component on the appropriate node. The manager does not own the capabilities of these memory regions; it only makes policy decisions, which are executed by the agent component by invoking capabilities within its node. Both the manager and the agent are responsible for the proper allocation and mapping of pages and are therefore highly trusted.

[0096] Figure 3 The memory allocation and node decomposition of a small exemplary system with four cores after startup are shown. Figure 3 This represents the base state reached at startup, where all components are statically defined, all resources are statically allocated, and all communication mechanisms are fixed. Because it is completely static, the base state system is built using a modified version of the Microkit framework (described at https: / / docs.sel4.systems / projects / microkit / ), which has been modified to build a multi-kernel-based system.

[0097] Nodes are allocated to a fixed physical memory area called private memory. As described above, Node 0 is a special system management node that hosts the remote management API and the manager components that centralize system management. Nodes 1 through 3 are started at boot and initially host only static components, such as the agent responsible for populating the node. The manager communicates with the agent using a circular buffer located in memory shared by the communicating nodes. All remaining physical memory (general memory and device memory) is initially allocated to Node 0. In some embodiments, the implementation of Node 0 may differ from the other nodes because its functionality is different. For example, for better isolation, some or all of Node 0's functionality may be moved to a dedicated PCIe device—which could be an external device used to run system management tasks.

[0098] In this architecture, it is assumed that at most one VMM task runs per node: each virtual CPU (vCPU) in this embodiment is exclusively mapped to a single physical CPU core (but it should be noted that a VM can be associated with multiple CPU cores). This design increases isolation between guest VMs because they are constructed to ensure they never share a CPU. This also makes the total number of VMM components known at compile time, allowing them to be allocated at image creation time. Note that the VMM components initially created in ground state are actually empty shells: they are not powered on, and their memory regions are not filled.

[0099] VMs are created and destroyed based on requests from system administrators via an application programming interface (API) component located on node 0 and accessible through a dedicated network interface. All system administration is arranged by the manager component on node 0 and executed by an agent on the selected target node.

[0100] The manager component running on node 0 maintains a consistent view of the entire system across multiple nodes and always acts as the custodian of all device memory in the system. It tracks all resource allocations and can grant and revoke access to these resources to the VMM running on all nodes via locally running agents. The agent components act as local representatives and execute requests from the manager—they are not involved in policy decisions.

[0101] When a system administrator requests the creation of a VM, the manager component performs the following operations: • Select one or more target nodes to host the newly created VM. When a multi-core guest VM is requested, the manager selects a node for each virtual CPU core, where each VMM task emulates a single virtual CPU core. The CPU core allocation for hosting the multi-core guest VM should follow the underlying NUMA (Non-Uniform Memory Access) topology of the hardware, such that all selected cores are preferably allocated close to each other.

[0102] • Allocate a memory region (labeled device memory) from the large pool of available memory to be used as guest memory for the VMs. This memory is also used to store VM state data that needs to be shared between cooperating VMMs (in the case of multi-core VMs) and any additional buffers required for their communication. The allocation of guest memory should follow the underlying NUMA topology of the hardware to minimize access time from the selected CPU cores.

[0103] • Assign PCIe devices to be passed through to the VMs. In this type of implementation, network devices supporting virtual functions and SR-IOV are relied upon. The physical function driver is a trusted component running on node 0, while the virtual function driver is untrusted and implemented in the VMM. The virtual device is directly assigned to the VMM and passed through to the guest VM. The VMM exposes the VirtIO device to the guest VM.

[0104] • Prepare requests for the agent components in the selected node to populate and configure the local VMM and its supporting tasks. This request includes references to all physical resources allocated to the VM to be created. Capabilities are not transferred between nodes: all nodes already have access to all shared memory regions; the manager and agent components are jointly responsible for their proper allocation and mapping.

[0105] After receiving the VM creation request from the manager, the agent performs the following operations: • Maps available memory regions marked as device memory into the VMM's address space. This region can be primarily used to create client address spaces.

[0106] • Map the memory associated with virtual functional devices into the VMM's address space. These areas will be used by the VMM to emulate VirtIO networks and block devices for guest VMs.

[0107] • Load the VMM binary code into the VMM's address space. Populate the regions containing data segments with VM metadata received from the manager, including the addresses and sizes of all memory regions mapped into its address space (guest memory, devices, etc.), references to I / O interrupt notification objects, etc.

[0108] • Start all VMM threads and notify the manager of the successful execution of the request.

[0109] Figure 4 Showing Figure 3 The system requests and dynamically creates dual-core VMs on Node 1 and Node 2. Following the method described above, the manager component allocates memory for the VMs from a large pool of unused memory marked as device memory. The physical network devices are partitioned into virtual functions using SR-IOV. The manager sends VM creation requests, along with references to all resources allocated to the VMs, to the agents on Node 1 and Node 2. The agents map these regions into the VMM's address space. The VMM ultimately maps these regions into the address space of the VMM components, populates their contents, and then starts the fully operational VMM components 315a and 325a.

[0110] Therefore, the ground state (such as Figure 3 The system shown is purely static; dynamic behavior is only revealed through a proxy. The following aspects are static and configured at image creation time: • Decompose large systems into multi-core configurations with multiple nodes, particularly the number of nodes and the amount of memory allocated to each node for use as private storage.

[0111] • The communication channels between the manager and the agent, the number of channels, the memory regions allocated as shared memory, and the inter-processor interrupt objects (IPIs) used for signaling.

[0112] • All components running on node 0.

[0113] • The root component, console, and agent component that run on all regular nodes.

[0114] • VMM shell components created on all regular nodes.

[0115] Figure 5 This process is illustrated. System administrator 51 requests 505 the creation of a VM, and the manager component 52 at node 0 accordingly allocates 510 global resources, in this case, two nodes, as well as storage and devices. The manager component then uses agent 53 at node 1 to "create" 515 the node 1 VMM (as described above, this involves activating the statically created "shell" VMM). Node 1 agent 53 configures 520 the node 1 VMM task and initializes 525 the node 1 VMM 54, which is reported back to node 1 agent 53 and subsequently to manager 52. A similar process occurs at node 2: manager 52 "creates" 530 the node 2 VMM at node 2 agent 56, configures 535 it, and initializes 540 the node 2 VMM 57, which is also reported back to manager 52. The VM hosting the client can then be started by manager 52—this begins by sending instruction 545 to node 1 agent 53, which similarly instructs 550 to node 1 VMM 54, thus starting node 1 VMM 54 to start 555 the node 1 client 55. Its first action is to start the second CPU—that is, the CPU allocated to the second node (node ​​2) of the VM. Node 1 client 55 passes this start command 560 to node 1 VMM 54, node 1 VMM 54 instructs node 2 VMM 57 to start node 2 client 58, and then node 2 VMM 57 executes 570. While the client start process is in progress, the manager 52 at node 0 is informed and notifies the system administrator.

[0116] Figure 14 This diagram illustrates an arrangement where a set of VMMs 141 and 142 running on different nodes can collaboratively manage the same customer VM instance by allocating the virtual memory of the customer VM in a shared memory region between nodes 147 and 148. Each VMM emulates a separate vCPU for the customer VM, and each node has its own microkernel 140. VMMs 141 and 142 communicate with each other using a dedicated region of shared memory 143 and IPI 144 for synchronization. Figure 14This demonstrates a split VMM that exposes a VM with two vCPUs, each virtualized by VMM tasks running on different multi-core nodes. Node A hosts the VMM tasks for the boot processor (CPU #0) of the emulated guest VM. Node B hosts the VMM tasks for the first application processor (AP) (CPU #1) of the emulated guest VM. Both nodes execute additional auxiliary tasks 145, 146 to service their local VMMs (e.g., the backend of emulated devices). The guest OS 149 runs on system memory shared between the two nodes. The two VMM components communicate with each other and maintain synchronization using inter-processor interrupts.

[0117] During initialization, the VMM simulating the boot processor assumes the role of the primary VMM. It configures the guest memory layout, loads the guest OS image and boot code, and waits for all other VMMs to check in. The primary VMM then communicates details of the guest VM settings (specifically memory layout and mapping) to the secondary VMMs so they can build the same configuration for the same VM. Once all VMMs are configured and ready, the primary VMM begins simulating the booting of a guest vCPU by executing the guest OS boot code, while all other VMMs start their vCPUs from an initial stopped state. The VMMs collectively implement the Symmetric Multiprocessing (SMP) specification, allowing the guest OS to request each vCPU to come online.

[0118] Therefore, it can be seen that such a distributed VMM can relatively easily emulate a typical SMP system. However, access to pass-through devices and emulated devices does require additional attention.

[0119] Pass-through devices are implemented by directly mapping the physical device memory into the address space of the guest VM, so once the memory mapping is configured, the VMM requires almost no work. In a distributed VMM configuration, the physical memory region of the pass-through device should be exposed to all multi-core nodes of the VMM component hosting the guest VM, and this memory region should be mapped to the same guest physical address in each VMM.

[0120] The emulation of a device is implemented through special calls (hypercalls, page faults) that trap the VMM upon access, allowing the VMM to emulate the behavior of the specific device and return control to the client. In a distributed VMM configuration, the hypercall should be configured identically on all VMMs participating in the emulation of a particular VM. All VMMs should be able to respond to the hypercall and emulate the device. When synchronization between VMMs is required to correctly emulate the device, the VMMs can use a dedicated communication channel established over shared memory and IPI.

[0121] The different elements and related processes of this architecture will now be described in more detail, starting with the boot process. This architecture differs from traditional sel4 systems during boot, which follow a three-stage process (hardware discovery and configuration, kernel initialization, and root task creation). This different boot behavior is referenced... Figure 6 The first phase 610 of the boot process is essentially the same as in an existing SEL4 system, starting with the booting processor core while all other processor cores are powered down. For this type of implementation, a new boot phase 615 is now inserted, where the kernel allocates physical memory regions for each node, fills these regions with a copy of the kernel itself, the node's root task, and a memory structure describing a subset of the system resources available to the node, and powers on all remaining processor cores. The second and third phases of the boot process (kernel state initialization 620, root task creation 630) then proceed on each processor core essentially as in a traditional SEL4 system, the main difference being that they are all started in parallel and do not share state. Typically, all SEL4 instances are unaware of each other's existence, and all cross-node communication is managed by user-space tasks.

[0122] The details of this new phase of allocating multi-core nodes are statically defined at compile time: the number of nodes to be allocated, the size of their private memory regions, the list of devices to be exposed to each node, etc. Therefore, this multi-core configuration is purely static.

[0123] from Figure 6 As can be seen, unlike the more traditional symmetric multiprocessing (SMP) configuration, nodes do not share kernel state; they run their own root tasks and have the ability to use different resource sets.

[0124] The root task executed on each node is statically constructed, and the details of the nodes on which the root task will run are known at compile time. The root task receives the following resources: • A memory region specifically allocated to the node, used as regular untyped objects, excluding the region used by the node kernel for its text segments and internal state. No other node has access to these regions. All future kernel objects created within this node will be specifically retyped from these untyped objects.

[0125] • Memory regions shared across all nodes, marked as device typeless. All nodes are given the ability to access these regions, enabling them to be shared effectively. Manager and agent components work together to ensure consistent and secure use of these regions across nodes. These regions are marked as device memory and can never be used to create kernel objects; they can only be used to create frames mapped into the address space.

[0126] • Device memory maps to input / output memory and is labeled as untyped device. Multi-core is statically configured to present a subset of devices present on the system to all nodes. Typically, only devices intended for VMs need to be assigned to all nodes, allowing them to be mapped into the VMM's address space (for emulating devices) and into the guest VM's address space (for passing through devices).

[0127] In this type of arrangement, the user space of each node is therefore statically built at compile time. Here, the user space of a node is a hybrid of static and dynamic, with the agent task providing the dynamism.

[0128] The communication mechanism connecting the manager component in node 0 and the agent components in all other nodes is statically defined at compile time. The root task of each node is responsible for creating all statically defined kernel objects within the node, including those assigned to the communication mechanism. These are typically device-untyped memory used as shared memory buffers and IPIs used for synchronization.

[0129] All communication between nodes is strictly implemented outside the kernel by user-space tasks, as discussed further below: the kernel does not participate in the communication mechanism—this contrasts with traditional SMP software architectures, where IPI is used internally by the kernel to synchronize execution across multiple cores.

[0130] Running independent kernels on multiple cores of the same CPU presents some challenges for memory management—the following features should be implemented in appropriate implementations: • Physical memory regions shared across multiple cores should use the same page size on all cores when mapped to virtual memory. These memory regions serve as guest physical memory and are used to create communication channels between nodes. One solution is to map these regions specifically using 2MB pages—these pages are supported by many 64-bit architectures and offer a good trade-off between memory allocation granularity and memory management overhead. Node-specific physical memory regions are not a problem because, by design, they are not mapped to virtual memory on any other core.

[0131] • Address space identifiers—On ARM, address space identifiers (ASIDs) and virtual machine identifiers (VMIDs); on x86, process context identifiers (PCIDs)—should be globally unique across all cores of the same CPU. Identifiers are determined independently by each node kernel, and seL4 does not expose mechanisms that allow policies to be defined on each kernel instance. User space should then allocate identifiers and assign them to VSpace objects, making tasks unschedulable until their VSpaces are assigned address space identifiers. The manager component centralizes the allocation of address space identifiers and guarantees their global uniqueness. It is assumed that the total number of globally running tasks does not exceed the number of address space identifiers, so that address space identifier reclamation is only needed during VM creation and demolition.

[0132] For x86 implementations, whenever the mapping changes, the system software should manually invalidate the TLB entries for each core; this is typically implemented using IPI. Here, a multi-core VM is implemented by multiple VMMs running on different nodes and sharing memory regions. The mapping does not change during the VM's lifetime. When the VM is demolished, the system should carefully invalidate the corresponding entries in the TLBs of all participating nodes. Instead of using IPI between selected nodes (which requires the nodes to know each other), synchronization is arranged by a manager component running on node 0 and coordinated by an agent on each participating node.

[0133] Special consideration needs to be given to the I / O memory management unit, as it is a critical resource shared across all cores—therefore, a method that goes beyond strict resource partitioning, which is used in other cases as a key method to prevent unwanted concurrency, is required.

[0134] The System Memory Management Unit (SMMU) on modern ARM devices has multiple contexts, each with its own dedicated root table and a PCIe requester ID (RID) that ultimately serves as the context descriptor. Virtual function (VF) devices derive their RIDs from the RIDs of their physical function (PF) devices—this simply follows the SRIOV specification. Therefore, each VF device can be assigned its own I / O MMU root page table, which is not shared with any other device. The virtual function set is then strictly partitioned for the kernel, guaranteeing non-concurrent access to the I / O MMU configuration.

[0135] As mentioned earlier, all communication between nodes is implemented outside the kernel by user-space tasks. Cross-node communication is crucial for the manager. A proxy communication channel, as well as a backend for the VirtIO console device running the VM, are required. The communication mechanism employed uses pages from memory regions marked as device memory and shared between nodes. These pages are mapped into the address spaces of the two components that act as communication endpoints in their respective nodes, effectively making them shared memory. Synchronization between the two endpoints relies on periodically polling the shared memory for new data at a fixed rate. In principle, IPIs can be used in addition to polling—and in some architectures, they may be more efficient.

[0136] It is assumed that the two endpoints participating in the cross-node communication link trust each other: their cooperation is necessary for the proper implementation of the mechanism.

[0137] An optional communication system used in some embodiments is such as Figure 7 As shown, the pages are organized in a ring, with the current read and write indices stored in different pages and configured as write-through caches. This configuration supports a lock-free implementation relying solely on memory barriers and atomic operations. While this mechanism would work as described by periodically polling changes to the read and write indices from two endpoints, a notification mechanism would be more efficient. The traditional mechanism for signaling to remote CPU cores is the inter-processor interrupt (IPI). These interrupts are typically used by SMP kernels to synchronize local processor state with global system state, such as updating the memory management unit (MMU) configuration. To this end, the traditional sel4 kernel is modified to provide a mechanism for sending and receiving IPIs from user-space components. These IPIs are used as notifications between two communicating endpoints in their respective nodes to indicate changes to the read and write indices.

[0138] Figure 7 An example of a cross-node communication link is provided. Eight shared data pages are organized in a ring. Read page indices and write page indices are stored in different pages. The transmitter component compares the values ​​of the read page index and the write page index to determine the number of pages that can be written before overwriting unread pages (2 in this example), writes data to the pages starting from the write index, and eventually updates the write index to point to the next page. The receiver component compares the values ​​of the read page index and the write page index to determine the number of pages remaining to be read (5 in this example), reads page data starting from the read index, and eventually updates the read index to point to the next page. IPI is used as a signaling mechanism to indicate changes in the read location index or the write location index.

[0139] The nature and interaction of the manager and agent components will now be described in more detail. The manager component running on node 0 and the agent components running on every other node are jointly responsible for the dynamic behavior of the otherwise static system (provided as the base state). These components are themselves part of the base state and are statically defined. The manager does not possess the ability to use memory regions as client physical memory—these capabilities do not exist at all in node 0. They do exist on all other nodes and are held by the agent components. The allocation of responsibilities between the manager and agents is as follows: The Manager component receives system management requests from the API component running on node 0. The Manager is the custodian of all shared resources in the system. Its primary functions are to allocate the resources (storage and passthrough devices) required to create new VMs and assign them to selected nodes, as well as to reclaim resources from terminated VMs. The Manager also relays system and VM status information to the API for monitoring.

[0140] The agent component receives commands from the manager and executes them on its local node. The agent is the custodian of all resources private to its node. When a VM is created, the agent transfers the ability of resources allocated by the manager to its local VMM task, populates its memory regions (code and data), and starts them. Through construction, shared resources are marked as memory frames to be mapped into the VMM address space: specifically, no kernel objects can be created from these resources. The agent also monitors the status of VMs running on the same node and reports status information to the manager.

[0141] The manager component is essentially a resource allocator and system manager. It does not use the ability to access shared resources in the system—in fact, it doesn't need to own the resources it manages. It only needs to know the addresses and sizes of memory regions that can be allocated to shared objects. Only agents have the ability to use these regions to create frames and map them into the address space of the appropriate component (usually the VMM).

[0142] The manager component enforces the allocation of all VM memory—both guest physical memory and memory used by the VMM itself—ensuring that DRAM subarrays are never shared between VMs as a mitigation of vulnerabilities in DRAM hardware such as Rowhammer. The specific layout of the DRAM modules is detected during configuration and used to partition available memory—the DRAM subarray size subsequently becomes the granularity of memory allocation.

[0143] The manager communicates directly with each agent using the communication mechanism described above, configured as a bidirectional link. Agents do not communicate with each other. The communication protocol includes the following messages: • CREATE: Sent by the manager to the agent to request the creation of a VM on its node. This message includes all the information required to create the VM, including the address and size of the physical memory pages to be used as guest physical memory, a list of physical devices to be passed to the VM, the address and size of memory regions for additional communication channels (e.g., between VMMs managing vCPUs of the same VM), vCPU numbers, etc. Upon receiving this message, the agent reconfigures the shell VMM task and reports to the manager. If it fails, any shared resources allocated to the VM are returned to the manager and marked as unallocated.

[0144] • DESTROY: Sent by the manager to the agent requesting the destruction of the VM on its node. Upon receiving this message, the agent revoks all additional capabilities transferred to the VMM and then resets the VMM task to its initial (“shell”) state. Note that untyped memory is never transferred to the thread supporting VM execution to avoid the potential situation where the thread can create more threads and escape its own destruction. Upon success, all shared resources allocated to the VM are returned to the manager and marked as unallocated.

[0145] • START: Sent by the manager to the agent to request the startup of the VM on its node. The VM is created in a stopped state, waiting for all VMMs on all participating nodes to be successfully created before starting. Note that only the VMM implementing the virtual boot processor core needs to be explicitly started. VMMs implementing other virtual processor cores will be started independently by the guest OS running in the VM.

[0146] • RESULT: Sent by the agent to the manager as a response to the CREATE, DESTROY, and START messages. This message includes a status code indicating whether the request was successfully executed; if unsuccessful, the message contains an error indication.

[0147] • STATUS: Sent by the agent to the manager to report the status of the node on which the agent is running. The message includes the amount of private resources available on the node and the status of the VMs running on the node.

[0148] If implemented correctly, this configuration of the manager and agent can provide VMs with security attributes of integrity and confidentiality. A misbehaving manager or agent could otherwise map the virtual memory of one VM into the address space of another VM, thus compromising their isolation.

[0149] Now refer to Figure 8 and Figure 9Considering different aspects of device management, this embodiment uses network devices supporting Single Root Input / Output Virtualization (SRIOV) to divide physical network devices into multiple virtual network devices, which are then directly assigned to VMs. Network data is transmitted between physical devices and VMs with minimal VMM intervention, resulting in low virtualization costs and latency. The complete implementation is broken down into the following different components: • The Physical Function (PF) device driver running in management node 0 is responsible for managing the global configuration of the network adapters. This driver only participates in control.

[0150] • Virtual function (VF) device drivers running within the VMM component manage the configuration of VF devices. The VMM exposes standard VirtIO network interfaces to guest VMs, abstracting away the details of the physical network adapters.

[0151] like Figure 8 As shown: PF device driver 800 runs in management node 80 (node ​​0), while VF device driver 810 runs in VMM component 81 of node N. The PF driver and VF driver communicate using a channel provided by physical network device 82. The VMM provides a standard VirtIO interface 83 to guest VM 84 (therefore, guest VM includes VirtIO network driver 840).

[0152] This division of responsibilities requires the establishment of a communication channel between the PF device driver and the VF device driver. The SRIOV standard does not define such a mechanism, which can be implemented by the hardware device or left to the host software—using a hardware mailbox system (such as that implemented in Intel 800 series network devices) is an efficient communication method that can reduce latency—these can use Message Signaling Interrupts (MSI) to signal network events to the VF, eliminating the need to forward signals from the PF driver to the VF driver.

[0153] In this embodiment, an external iSCSI server is used to provide persistent storage to the VMs. Decoupling storage from compute has advantages in a managed environment: VMs are not limited to the amount of storage present on their hosts; cloud operators do not need to fill all compute nodes with potentially underutilized large storage devices; and data redundancy and backup are easier to manage from a few storage servers rather than distributed across the entire compute cluster, and a few consolidated storage servers are more economical.

[0154] Figure 9This demonstrates how this is implemented. The interaction between the management node 90 (and its PF network driver 900) and node N VMM 91 (and its VF network driver 910) via physical network device 92 is the same as in the network case. The virtual storage driver 920 of VMM 91 interfaces with the external storage server via a private network link exposed as a VF. The virtual storage driver implements the iSCSI protocol. The external storage server is considered untrusted, so all data must be strongly encrypted. The virtual storage driver 920 can then implement transparent encryption and decryption, so that the external storage server only sees the encrypted data, while VM 94 sees the plaintext. The virtual storage driver 920 exposes a standard VirtIO block device 93 to the guest VM 94, which includes the appropriate block device driver 940.

[0155] Boot media (such as ISO files) are also exposed to guest VMs using VirtIO block devices supported by external iSCSI storage. However, the boot media is exposed in read-only mode.

[0156] Initial VM management and configuration, as well as system recovery, typically require out-of-band mechanisms to reach the guest OS. For example... Figure 3 and Figure 4 As shown, the VMM exposes the VirtIO console device to the client for this purpose, exposing the client kernel output and the console device that can be bound to the command-line session during startup.

[0157] The VMM replicates the console data stream to a console backend component running on node 0, which collects data streams from all console devices in the system. The data streams are accessed via a remote API. It is anticipated that cloud operators managing the cluster will use the hypervisor's remote API to expose the VM consoles to their respective users.

[0158] Remote management within this architecture will now be discussed. Preferably, the sole mechanism for software management and monitoring is an HTTPS-based API, enabling easy integration into a centrally managed virtualization cluster. A dedicated network interface (typically a network adapter on the server motherboard) should be used to host all system management operations.

[0159] The remote management component can be implemented as a VM running on node 0 and managed by a statically configured VMM. A dedicated network adapter is passed through to the VM, thus eliminating the need for native seL4 device drivers. This VM can, for example, run an embedded Linux-based guest OS built using lightweight components such as buildroot, musl libc, busybox, etc. It includes drivers for common network interfaces, a network IP stack, network configuration and management utilities (dhcp, ntp, etc.), a lightweight web server, and remote management applications.

[0160] The management application implements API endpoints, but it does not directly implement or authenticate requests. Instead, these are forwarded to and executed by a trusted manager component that also runs on node 0. The management VM communicates with the manager component using statically allocated shared memory regions and seL4 notifications. The VMM exposes the manager communication channel to the guest OS as a VirtIO socket device.

[0161] The remote management component comprises a complete Linux-based VM—therefore, it cannot be considered a trusted component. Instead, it is designed as a transparent interface between the untrusted management network and the trusted manager component. All requests sent to the hypervisor are cryptographically signed and verified by the manager component. Similarly, all responses from the manager component are cryptographically signed and returned to the external network. If the management component (including its VMM) is compromised, an attacker will be unable to impersonate a system administrator (or send forged requests to the manager, or forged responses to a legitimate system administrator)—at most, they can only achieve DoS-type interference.

[0162] This arrangement is like Figure 10 As shown. The remote management component runs in an (untrusted) Linux VM 1001 and forwards responses between the external system administrator and the trusted manager component 101. The manager verifies the message signature before accepting the request using its own private key via the authentication and signature module 1011. Encryption operations are performed by the hardware TPM module 1012.

[0163] Because public key certificates are used for authentication and thus authorize all management program operations, the security of the public key infrastructure (PKI) used is critical and requires a high degree of trust. PKIs are typically external PKIs controlled by cloud operators.

[0164] Each party interacting with the hypervisor, and each hypervisor itself, is identified by an x509 public key certificate. All certificates are signed by an intermediate CA (Certification Authority) and ultimately by the root CA—which is typically stored in an air-gap vault. Requests must be accompanied by a valid x509 certificate verifying the end user issuing the request, and an attribute certificate authorizing the request. Attribute certificates support fine-grained access control filters (e.g., system administration, access to read-only monitoring data, access to system logs, operations on specific VMs such as restarts or console access). Management of certificates and attribute certificates is left to the cloud operator.

[0165] The hypervisor's private key is stored in a locally attached TPM, which also performs all encryption operations. The root CA certificate is also stored in the TPM and used to authenticate incoming requests. Responses are signed by the hypervisor's key stored in the TPM and can be remotely authenticated. When provisioning the hypervisor, the TPM generates the private key, then extracts the public key and authenticates it with the cloud operator's PKI.

[0166] Using public key certificates eliminates the need to maintain databases of users, credentials, roles, etc., and allows users to manage, authenticate, and authorize externally through cloud operators' PKI, thereby maximizing flexibility.

[0167] This method is as follows Figure 11 As shown. The PKI 110 managed by the cloud operator uses the root CA 1101 and the derived purpose-based CA 1102 to sign certificates for all parties (users, administrators, hypervisors, etc.). The public key of the root CA 1101 is stored in the TPM of each hypervisor 111 and used for authentication requests.

[0168] Further functionality can be built on this architecture. For example, there is a critical trade-off between VM security (isolation) and physical resource utilization. VMs co-located on the same physical CPU will share access to CPU cycles and resources such as cache, leading to performance and security issues. However, some applications may not guarantee this hard resource separation and can aggregate onto a small subset of dedicated cores to improve physical resource utilization. These can be implemented as trust domains: a collection of VMs that trust each other sufficiently to share one or more CPUs.

[0169] Trust domains can implement additional constraints that affect the allocation of physical CPUs to virtual CPUs. Not every node has a single VMM task; a fixed number of VMM components can exist on each node, sharing access to the node's CPU. Priorities between VMMs can be defined using the Hybrid Critical Systems (MCS) extension of seL4, which is designed to support time-based partitioning.

[0170] Inter-VM networking mechanisms can also be provided. There are many use cases for VMs connected to private networks. Common examples include backend networks hosting web application databases, servers behind Virtual Private Networks (VPNs) or access control services, all the way to private clouds. Hypervisors of these types will not provide mechanisms such as point-to-point VirtIO devices to support direct inter-VM networking or virtual network switch devices. These would have performance issues, and private networks would likely extend beyond the host machine, requiring interconnection of VMs hosted on different physical servers. Instead, private inter-VM networking can be implemented as a layer on top of a regular uplink network. The VMM will create virtual switch ports using the VXLAN protocol and expose them to clients as VirtIO devices. Additional privacy can be achieved, for example, by encapsulating VXLAN traffic in IEEE 802.1AE (also known as MACsec) encryption. Network parameters, including encryption keys, can be provided via an API.

[0171] Different architectural approaches can also be employed within this general structure. Unlike other multi-core implementations, the VM server in this type of embodiment does not attempt to create and start all nodes at startup. Initially, only a single node is created and runs on the startup CPU: the system management node. All other CPUs remain in a stopped state. The system management node is granted exclusive access to a dedicated network interface on which a management API is exposed. It is also granted access to all resources existing in the system. These resource capabilities are assigned to a privileged multi-core management component, which is responsible for the creation and destruction of all other nodes in the system, including the allocation of physical resources with exclusive or shared access. Specific IPIs are used to start and stop application CPUs.

[0172] A complete system typically includes multiple service nodes, multiple user nodes, and communication links between these nodes. Service nodes are managed by the cloud operator and provide basic services to user nodes. Examples of service nodes include network nodes that reuse access to physical network devices and storage nodes that expose block devices to user nodes. User nodes include custom-built virtual machine monitor (VMM) tasks running guest OSes. It should be noted that there is a maximum number of nodes in a given system, limited by the number of CPU cores. Depending on the hardware and use case, it may be more preferable to bundle multiple services and VMs onto a single node.

[0173] Figure 12This architecture is illustrated. In this type of embodiment, node 0 runs the system management components and is the only node started at boot. Nodes 1 and 2 are service nodes, providing network and storage access to the other nodes, respectively. Nodes 3 and 4 run the end-user workload OS. Communication between nodes is implemented using a ring buffer in shared memory and IPI as a signaling mechanism, with the logic handled by cooperative tasks labeled as link interfaces. The vertical bar on the left represents the system's complete physical address space and the allocation of various memory regions to nodes and the ring buffer.

[0174] The user space of all nodes, including the system management node, can be built using the sel4 Microkit (formerly the seL4CP framework), described at https: / / docs.sel4.systems / projects / microkit / . The components of the system management node are responsible for assembling the sel4CP image at node creation time using locally stored ELF files and user-provided guest OS images and runtime configuration requests. Nodes can communicate with each other using a ring buffer stored in shared memory and IPIs for signaling. The ring buffer is collaboratively managed by trusted userspace tasks running on all nodes requiring such communication channels. Apart from IPI modeling, no involvement of the seL4 microkernel is required to implement the communication mechanism.

[0175] VM servers built using this architecture can also scale efficiently to the large number of cores present on modern cloud servers, thanks to their multi-core architecture. Here, the system can be considered a hybrid static / dynamic system. The software stack running on each node is statically defined and built using this framework, much like building a static seL4 embedded system—the userspace stack running on each node is statically defined, and its critical components should be small enough for formal verification. However, the nodes are dynamically managed; they are created and destroyed at runtime based on requests from system administrators and end users.

[0176] In cloud server architectures, user space is expected to be dynamic rather than static. The seL4 microkernel itself does not require a statically defined user space—seL4 is designed to be as policy-free as possible and impose few restrictions on user space. However, as a capability-based microkernel, implementing a fully dynamic user space is a challenging task due to the large number of capabilities to manage. A typical seL4 system is static, fully defined at compile time, and therefore relatively easy to develop, reason about, and (very importantly) ultimately validate—the method indicated in the previous examples does indeed use a fully defined static system, with agents used to provide dynamism.

[0177] Using a hybrid static / dynamic approach can provide alternative ways to achieve flexibility—as well as dynamic user spaces, such as... Figure 13 As shown. In this method: • The image is assembled according to the conventional method of building a static system, where the system state is described in an XML file, listing memory regions (MR) and protection domains (PD).

[0178] A special PD in the system, called the Node Manager, introduces dynamic behavior by being allocated additional MapReduce (MRs) pointing to node-private memory regions and memory regions shared among multiple nodes. Using the ability to access these memory regions, the Node Manager can create sub-PDs from MRs pointing to private node memory, and it can map MRs pointing to shared memory regions to implement cross-node connectivity mechanisms at runtime without affecting the static components of the user-space stack.

[0179] A simple communication channel is enabled in the memory area shared between management node 0 and the node manager PD of each other node, allowing all nodes to be centrally controlled from the management node.

[0180] Figure 13 An example of this implementation is shown. The PD at the bottom of the diagram is statically created and is initially the only running PD in that node. In this example, the Node Manager PD has created link interfaces to communicate with other nodes, as well as two VMM PDs running the customer OS leased to the user. This mechanism allows the node's content to ultimately be managed via an external cloud management console and using the server's API.

[0181] While different architectures are shown here, those skilled in the art will understand that the various features described with respect to the first embodiment can be used with the second embodiment, and vice versa. Some such features have been described above. Furthermore, it should be noted that for any embodiment, the following may apply: the user space can be developed to be dynamically or statically built at runtime, and the user space at runtime can be static or a hybrid of static and dynamic; and in different arrangements, VMs can be dynamically added to or removed from running nodes, or cannot be dynamically added to or removed from running nodes. The choice between a fully static and a static / dynamic user space stack can be seen as a design trade-off, with each choice being more suitable for certain applications. Another possibility is that some nodes are of one type and some are of another—for example, some nodes can be statically defined while others are dynamically created.

[0182] Another variation is to provide the various parts of the architecture in hardware rather than software. The manager component on node 0 (in fact, all nodes 0), the agent component on each node, and (where applicable) any service node are all suitable for such an implementation.

[0183] Although the seL4 microkernel is used in this embodiment, those skilled in the art will understand that the same design principles can be applied to multi-kernel architectures using different microkernels with properties similar to seL4.

[0184] Further modifications may be made to the above examples without departing from the scope of this disclosure.

Claims

1. A computer system with a multi-kernel architecture, the multi-kernel architecture comprising multiple operating system microkernels, the computer system comprising: Multiple processing cores; The memory includes multiple private memory regions and shared memory regions; as well as Multiple computing nodes, each of the multiple computing nodes including one or more of the processing cores, and also including one of the multiple operating system microkernels and an associated user space for non-kernel operations, wherein only the operating system microkernel runs at the highest privilege level of one or more of the processing cores of the computing node; Each of the plurality of computing nodes has an associated private memory region from the plurality of private memory regions for use by the operating system microkernel of each of the plurality of computing nodes, and wherein, for non-kernel operations, each of the plurality of computing nodes accesses the shared memory region.

2. The computer system as claimed in claim 1, wherein, The computer system is adapted to statically partition the system hardware into nodes at startup.

3. The computer system as described in claim 1 or 2, wherein, One of the plurality of computing nodes is a management node for managing user nodes, and one or more other computing nodes are user nodes for running user workloads.

4. The computer system as described in claim 3, wherein, Multiple compute nodes are management nodes.

5. The computer system as claimed in claim 3 or claim 4, wherein, The computer system is adapted to provide users with one or more virtual machines.

6. The computer system as described in claim 5, wherein, No more than one virtual machine is implemented on any user node.

7. The computer system as described in claim 5, wherein, Two or more mutually trusted virtual machines are implemented on one or more user nodes.

8. The computer system as claimed in any one of claims 5 to 7, wherein, One or more virtual machines are implemented using multiple user nodes.

9. The computer system as claimed in any one of claims 5 to 8, wherein, The computer system is suitable for operation as a cloud server.

10. The computer system as claimed in any one of claims 5 to 9, wherein, The management node includes a manager component, and each user node includes an agent, wherein the node management component is adapted to instruct the agent of the user node to create, destroy, or modify virtual machines in the user node.

11. The computer system as claimed in any one of claims 5 to 9, wherein, Communication between user nodes is not privileged, but communication between the manager component and the one or more agents is privileged.

12. The computer system of claim 11, wherein, Direct communication between nodes is limited to direct communication between the user spaces of these nodes.

13. The computer system as claimed in any one of claims 3 to 8, wherein, Each user node has a virtual machine manager component for managing virtual machines within the user node through interaction with the manager component.

14. The computer system as described in claim 13, which is dependent on claim 6, wherein, A virtual machine manager is provided for each virtual machine in the user node.

15. The computer system as claimed in claim 13 or 14, wherein, Each user node includes a console driver, and the management node includes a console relay for communication between the management node and the virtual machine manager.

16. The computer system as claimed in any one of claims 13 to 15, wherein, Each virtual machine manager component is statically defined at startup for subsequent configuration.

17. The computer system of any one of claims 3 to 16, further comprising one or more network devices, wherein, The management node includes a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; and wherein one or more of the user nodes include a virtual function device driver for a virtual function network device corresponding to one of the one or more network devices, wherein the virtual function device driver provides virtual network device functionality in the user node.

18. The computer system of claim 17, wherein, The physical function device driver is provided by the SR-IOV mechanism.

19. The computer system of claim 17 or 18, further comprising one or more network devices, wherein, One or more of the network devices are storage devices, and data is encrypted for communication between the virtual machine and the storage device.

20. The computer system as claimed in any one of claims 3 to 19, wherein, The computer system also includes a remote management interface, which includes a dedicated network link.

21. The computer system as claimed in any of the preceding claims, wherein, Each microkernel has proven implementation correctness.

22. The computer system as claimed in any of the preceding claims, wherein, Each microkernel is a third-generation microkernel.

23. The computer system of claim 22, wherein, Each microkernel is a seL4 microkernel.

24. A method of operating a computer system, the computer system including multiple processing cores and memory to implement a multi-core architecture, the method comprising: Define a management computing node with one or more processing cores, start the management computing node and statically partition the processing cores into multiple additional computing nodes, each computing node including one or more processing cores and having an operating system microkernel and an associated user space, wherein only the operating system microkernel runs at the highest privilege level of one or more of the processing cores of the computing node; And to start and provision the additional computing nodes as user nodes.

25. The method of claim 24, further comprising: The memory is established as a plurality of private memory regions and a shared memory region, wherein each of the plurality of computing nodes is provided with an associated private memory region from the plurality of private memory regions for use by the operating system microkernel of each of the plurality of computing nodes, and wherein some or all of the plurality of computing nodes access the shared memory region for non-kernel operations.

26. The method of claim 24 or 25, wherein, The user node is adapted to provide one or more virtual machines to the user.

27. The method of claim 26, wherein, No more than one virtual machine is implemented on any user node.

28. The method of claim 26, wherein, Two or more mutually trusted virtual machines are implemented on one or more user nodes.

29. The method of any one of claims 26 to 28, wherein, One or more virtual machines are implemented using multiple user nodes.

30. The method according to any one of claims 26 to 29, wherein, The computer system is suitable for operation as a cloud server.

31. The method according to any one of claims 26 to 30, wherein, The management node includes a manager component, and each user node includes an agent, wherein the method further includes: the node management component instructing the agent of the user node to create, destroy, or modify virtual machines in the user node.

32. The method according to any one of claims 26 to 31, wherein, Each user node is equipped with a virtual machine manager component for managing virtual machines within the user node through interaction with the manager component.

33. The method of claim 32, further comprising: Each virtual machine manager component is statically defined at startup.

34. The method of claim 33, further comprising: Configure the virtual machine manager component of the user node when creating a virtual machine in the user node.

35. The method of any one of claims 26 to 34, wherein the computer system further comprises one or more network devices, wherein, The management node includes a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; and wherein one or more of the user nodes include a virtual function device driver for interacting with a virtual function network device, the virtual function network device corresponding to one of the one or more network devices, and the method further includes: the virtual function device driver providing virtual network device functionality in the user node.

36. The method of claim 35, wherein the computer system further comprises one or more network devices, wherein one or more of the network devices are storage devices, and the method further comprises: Data is encrypted for communication between the virtual machine and the storage device.

37. The method of any one of claims 24 to 36, wherein, All communication between user nodes takes place between the user spaces of the user nodes.

38. The method according to any one of claims 24 to 37, wherein, Any privileged communication between nodes occurs between the manager node and another node.

39. A method for providing a dynamic user space in a computer system, wherein the computer system is adapted for virtualization, the method comprising: Multiple computing nodes are statically defined, each computing node including an operating system kernel, wherein one of the multiple computing nodes is a management node, and one or more other computing nodes of the multiple computing nodes are user nodes; The management node includes a manager component for managing user nodes, and each user node, or each user node, includes an agent; and The manager component instructs the agent of the user node to provide the creation, destruction, or modification of virtual machines in the user node, thereby providing a dynamic user space in the user node.

40. The method of claim 39, wherein, The operating system kernel of the compute node is a microkernel.

41. The method of claim 39 or 40, wherein, In the compute node, only the operating system kernel runs at the highest privilege level.

42. The method according to any one of claims 39 to 41, wherein, There are multiple management nodes.

43. The method according to any one of claims 39 to 42, wherein, The manager component is adapted to receive management requests from a remote management interface that includes a dedicated network link.

44. The method according to any one of claims 39 to 43, wherein, The manager component determines the allocation of all memory for the virtual machine.

45. The method of any one of claims 39 to 44, further comprising: A virtual machine is created in a user node by instructions from the manager component, the instructions including an identifier of the memory to be used by the virtual machine and a physical device to be passed through to the virtual machine.

46. ​​The method of claim 45, wherein, The virtual machine was created in a stopped state.

47. The method of claim 46, further comprising: The manager component instructs the agent to start the virtual machine.

48. The method according to any one of claims 45 to 47, wherein, The user node includes a virtual machine manager component, and the method further includes configuring the virtual machine manager component for use with the virtual machine to be created.

49. The method of claim 48, further comprising: The virtual machine manager component is statically created as a shell component during system startup for configuration when virtual machines are created.

50. The method of claim 48 or 49, wherein, Each user node includes a console driver, and the management node includes a console relay for communication between the management node and the virtual machine manager.

51. The method according to any one of claims 39 to 50, wherein, Communication between user nodes is not privileged, but communication between the manager component and the one or more agents is privileged.

52. The method according to any one of claims 39 to 51, wherein, Direct communication between nodes is limited to direct communication between the user spaces of these nodes.

53. The method of any one of claims 39 to 52, wherein the computer system further comprises one or more network devices, wherein, The management node includes a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; the method further includes providing a virtual function device driver at one or more user nodes for a virtual function network device corresponding to one of the one or more network devices, wherein the virtual function device driver provides virtual network device functionality at the user nodes.

54. The method of claim 53, wherein, The physical function device driver is provided by the SR-IOV mechanism.

55. The method of claim 53 or 54, wherein the computer system further comprises one or more network devices, wherein one or more of the network devices are storage devices, and wherein, The method further includes encrypting data for communication between the virtual machine and the storage device.

56. A computer system suitable for virtualization using dynamic user space, the computer system comprising: Multiple computing nodes, each of which includes an operating system kernel, wherein one of the multiple computing nodes is a management node, and one or more other computing nodes of the multiple computing nodes are user nodes; The management node includes a manager component for managing user nodes, and wherein the user node, or each user node, includes an agent; and The computer system is adapted to instruct the agent of the user node through the manager component to provide the creation, destruction or modification of virtual machines in the user node, and to provide a dynamic user space in the user node.

57. The computer system of claim 56, wherein, Each compute node includes one or more processing cores.

58. The computer system as claimed in claim 56 or 57, wherein, The operating system kernel is a microkernel.

59. The computer system as claimed in any one of claims 56 to 58, wherein, In the user node, only the operating system kernel runs at the highest privilege level.

60. The computer system as claimed in any one of claims 56 to 59, wherein, Communication between user nodes is not privileged, but communication between the manager component and the one or more agents is privileged.

61. The computer system as claimed in any one of claims 56 to 60, wherein, Direct communication between nodes is limited to direct communication between the user spaces of these nodes.

62. A computer system having a multi-kernel architecture, the multi-kernel architecture including multiple operating system kernels, wherein the computer system is adapted to provide and manage virtual machines, the computer system comprising: Multiple processing cores; Multiple computing nodes, wherein each computing node is assigned one or more processing cores, and wherein each computing node includes an operating system kernel, and wherein one of the multiple computing nodes is a management node for managing user nodes, and one or more other computing nodes of the multiple computing nodes are user nodes; The management node includes a manager component, and each user node includes a virtual machine manager component for managing virtual machines within the user node through interaction with the manager component.

63. The computer system of claim 62, wherein, Each virtual machine manager component is statically defined at startup for subsequent configuration.

64. The computer system of claim 63, wherein, The virtual machine manager component is statically defined as a shell to be populated when a virtual machine is created in the user node.

65. The computer system as claimed in any one of claims 62 to 64, wherein, Provide a virtual machine manager for each virtual machine in the user node.

66. The computer system as claimed in any one of claims 62 to 65, wherein, The computing system includes multiple management nodes.

67. The computer system as claimed in any one of claims 62 to 66, wherein, Each user node includes a console driver, and the management node includes a console relay for communication between the management node and the virtual machine manager.

68. The computer system as claimed in any one of claims 62 to 67, wherein, The management node includes a manager component, and each user node includes an agent, wherein the node management component is adapted to instruct the agent of the user node to create, destroy, or modify virtual machines in the user node.

69. The computer system of claim 68, which is dependent on claim 64, wherein, The agent of the user node is adapted to configure the virtual machine manager component for the user node when the virtual machine is created.

70. The computer system according to any one of claims 62 to 69, wherein, The operating system kernel of the compute node is a microkernel.

71. The computer system of claim 70, wherein, Only the microkernel runs at the highest privilege level of the CPU core.

72. The computer system of claim 71, wherein, The virtual machine manager component does not run at the highest privilege level.

73. A computer system suitable for virtualization using network devices, the computer system comprising: Multiple processing cores; Multiple computing nodes, wherein each computing node is assigned one or more processing cores, and wherein each computing node includes an operating system kernel, and wherein one of the multiple computing nodes is a management node, and one or more other computing nodes of the multiple computing nodes are user nodes; and One or more network devices; The management node includes a physical function device driver for each of the one or more network devices for physical interaction with the associated network device; and one or more of the user nodes include a virtual function device driver for a virtual function network device corresponding to one of the one or more network devices, wherein the virtual function device driver provides virtual network device functionality in the user node.

74. The computer system of claim 73, wherein, The physical function device driver is provided by the SR-IOV mechanism.

75. The computer system of claim 73 or 74, further comprising one or more network devices, wherein, One or more of the network devices are storage devices, and data is encrypted for communication between the virtual machine and the storage device.

76. The computer system as claimed in any one of claims 73 to 75, wherein, The operating system kernel of the compute node is a microkernel.

77. The computer system of claim 76, wherein, Only the microkernel described runs at the highest privilege level of the CPU core.

78. The computer system according to any one of claims 73 to 77, wherein, The computing system includes multiple management nodes.

Images