EXTENSION OF SUPERVISORY SERVICES TO TRUSTED CLOUD OPERATOR DOMAS

Abstract

A procedure that includes the following: at least one hardware processor (116) of a node (110) executing initial instructions connected to an application operating environment (120) and connected to an untrusted cloud tenant domain (111), wherein the application operating environment (120) uses a service of a smart input / output peripheral (I / O) device (180), the service is connected to a trusted cloud operator domain (113), and a host runs a monitoring service (150, 152, 154, 156) for the application operating environment (120); and Extending the monitoring service (150, 152, 154, 156) of node (110) into the trusted cloud operator domain (113), wherein: Extending the monitoring service (150, 152, 154, 156) includes the at least one hardware processor (116) executing second instructions to monitor the intelligent I / O peripheral device (180); and Monitoring the intelligent I / O peripheral (180) includes the at least one hardware processor (116) configuring the intelligent I / O peripheral (180), determining the state of a readiness indicator provided by the intelligent I / O peripheral (180), and determining, based on the state, whether the intelligent I / O peripheral (180) is ready to be configured.

Description

BACKGROUND

[0001] A cloud architecture may comprise a computer system domain with multiple physical server nodes (e.g., blade servers or rack-mounted servers), or "domain nodes." The domain nodes define a physical infrastructure domain of the computer system. The physical infrastructure domain may be managed by a central management system that orchestrates software-defined logical infrastructure and services (e.g., software-defined compute services (SDC), software-defined storage services (SDS), and software-defined networking services (SDN)) hosted on the domain nodes. The software-defined logical infrastructure relies on an abstraction infrastructure layer (e.g., a hypervisor for virtual machine (VM) abstraction or an operating system for container abstraction) to enable centralized management control of the services.

[0002] US 2021 / 0141658 A1 discloses a method and a computer system for trusted devices using trust domain extensions. A device trust domain is implemented in a trusted address space, and one or more devices are bound to the device trust domain, allowing one or more virtual machines or trusted domains to securely access one or more functions provided by the bound devices.

[0003] It is an object of the present invention to close an architectural gap in previous server designs caused by the opaque architecture of intelligent input / output (I / O) peripherals, whereby it cannot be ensured that host services that can be outsourced to the intelligent I / O peripheral can be securely configured. This object is achieved by a method according to claim 1, by a system according to claim 7, and by a storage medium according to claim 17. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic diagram of a cloud computing system according to an example implementation. Fig. 2 shows an exemplary power-on sequence for an intelligent input / output (I / O) device of a domain node of the cloud computing system of Fig. 1 according to an example implementation. The Fig. 3A, Fig. 3B, Fig. 3C and Fig. 3D show example architectures related to domain node host monitoring services according to example implementations. Fig. Figure 4 is a flowchart illustrating a bus enumeration process used by a host monitoring service according to an example implementation. Fig. Figure 5 is a flowchart illustrating a process used by a host monitoring service responding to a runtime container instance change event according to an example implementation. Fig. Figure 6 is a flowchart illustrating a process used by an intelligent I / O peripheral to manage ready state indications according to an example implementation. Fig. Figure 7 is a flowchart showing a process for extending a monitoring host service into a trusted cloud operator domain according to an example implementation. Fig. 8 is a schematic diagram of a system including a host to perform a monitoring service for an intelligent I / O peripheral according to an example implementation. Fig. 9 is an illustration of a non-transitory storage medium storing machine-readable instructions that, when executed by a machine, cause the machine to maintain a ready state indicator of an intelligent I / O peripheral, according to an example implementation. DETAILED DESCRIPTION

[0004] A computing platform, such as a server (e.g., a blade server or a rack server), may have an architecture (e.g., a traditional PC-based architecture or a "legacy architecture") characterized by a "host-centric" control model. In such a control model, server controls are embedded in the server's host. As used herein, a "host" refers to the computing platform software and hardware that provides one or more application operating environments for the computing platform. In other words, server controls are embedded in the server's central processing units (CPUs).

[0005] The server may be a compute node of a cloud-based computing system, or a "cloud computing system." The software layer of the compute node's host may provide the logical infrastructure abstraction and control interfaces through a hypervisor software stack (for virtual machines (VMs)) and / or an operating system software stack (for containers). The result of this host-centric control model is a separation of administrative control of the compute node's physical infrastructure from the compute node's logical infrastructure.

[0006] A centralized management system of the cloud computing system may be built around a hierarchical management structure. At the lowest level of the hierarchical management structure, the compute node may be divided into two distinct control surfaces: a first control surface for the physical infrastructure of the compute node and a second control surface for the logical infrastructure of the compute node. Each of these infrastructures may be associated with its own central administrator, so there may be two distinct associated administrators: an administrator for the physical infrastructure and a virtualization administrator for the logical infrastructure. Accordingly, there may be two distinct centralized management software stacks, and there may be a cloud infrastructure manager at the top of the hierarchical management structure to unify the software stacks.The cloud infrastructure manager can replace the roles of the traditional infrastructure administrator with the following distinct cloud roles: the operator (for control of the physical and logical infrastructure), the tenant (for abstracted instance control), and the infrastructure support engineer (for support).

[0007] Cloud-native architectures may look similar to traditional cloud architectures, but they differ in one key aspect. Cloud-native architectures merge physical and logical infrastructure by using intelligent input / output (I / O) subsystems, referred to herein as "intelligent I / O peripherals," to offload services from the host (i.e., offload the traditional processing performed by the host) and by isolating node management control within the intelligent I / O peripherals. The merging of physical and logical infrastructure changes the general-purpose nature of traditionally architectured servers by using an independent software stack that can be managed in segments (e.g., segments targeted at SDC, SDS, and SDN layers). Accordingly, such a server is referred to herein as a segmented server (or "segmented compute node").This segmented control model presents the legacy server with the challenge of requiring all independent managers to be placed in a "ready state" before the backend I / O services provided by the intelligent I / O peripherals are delivered. This new orchestration and sequencing results in a more vertically integrated (i.e., tightly coupled or unified) architecture. While the cloud-native architecture uses a hierarchical structure for control management, the result is a unified experience centered around three control roles: the operator, the tenant, and the support engineer.

[0008] The term "intelligent I / O peripheral" refers to a device or component of the computing platform that provides one or more functions to the host that were controlled by the host in older architectures. Generally, an intelligent I / O peripheral is a hardware processor that has been assigned (e.g., programmed) a specific personality. The intelligent I / O peripheral can provide one or more backend I / O services (or "host-offloaded services") consistent with its personality. The backend I / O services can be either non-transparent or transparent. An example of a non-transparent host service is a hypervisor virtual switch offload service that uses PCIe direct I / O (e.g., CPU Input-Output Memory Management Unit (IOMMU) mapping of physical and / or virtual PCIe device functions) without host control.A transparent host backend I / O service requires no changes to the host software. Examples of transparent backend I / O services include network-related services such as encryption services, overlay network access services, and firewall-based network protection services. In general, the intelligent I / O peripheral can provide any number of transparent and / or non-transparent backend network services to the host. Examples of network-related backend services include overlay network services, virtual switching services, virtual routing services, and network function virtualization services. Storage-related backend services can include backend storage I / O services for the host, such as storage acceleration services (e.g., NVMe-based services), direct-attached storage services, or SAS (Serial Attached SCSI) storage services.

[0009] An intelligent I / O peripheral can be coordinated with another network component to provide one or more backend I / O services. For example, the intelligent I / O peripheral can be a "smart NIC" connected to an Ethernet top-of-the-rack (ToR) switch. This combination can provide host isolation by using the smart NIC in its "locked down" state for physical and virtual Peripheral Component Interconnect Express (PCIe) functions. Here, the "locked down" state refers to restricting host access to the device's full reconfiguration capability based on the cloud operator's objectives. For example, the lock down state can prevent the host from reconfiguring certain infrastructure (e.g., turning off memory redundancy, turning off GPUs, etc.).), which could, for example, impact the cloud operator's contractual obligations to the tenant. With this arrangement, the ToR switch can be used to provide network services such as a network protection service (e.g., a firewall service) or an instance overlay service (e.g., a VxLAN (Virtual Extensible Local Area Network) service).

[0010] The intelligent I / O peripheral can take one of many different physical forms and offer a variety of backend I / O services. A given intelligent I / O peripheral can provide network connectivity (e.g., an Ethernet port), multiple CPU cores, hardware acceleration modules, and a rich programming environment (i.e., an environment that enhances the integration of independent software components). In addition, the intelligent I / O peripheral can provide endpoint connectivity (e.g., one or more PCIe ports) to an in-band communication channel (e.g., a PCIe link or PCIe bus) accessible by the host.As a specific example, the components of a PCIe card-based smart I / O peripheral may be mounted on a printed circuit board substrate having a PCIe card edge connector that has a form factor that allows the smart I / O peripheral to be installed in a PCIe card slot of the server. In accordance with further implementations, a smart I / O peripheral may be designed to be installed in a card connector other than a PCIe card connector, and in accordance with still further example implementations, components of the smart I / O peripheral may be mounted (e.g., surface-mounted) on a server motherboard.

[0011] A cloud-native, segmented compute node with intelligent I / O peripherals results in a dual management control system within the compute node, separating control of the intelligent I / O device from control of the host CPU by adding a hardware-abstracted interface to the host. The hardware-abstracted interface can be presented to the host as a standard local I / O peripheral (i.e., a non-intelligent I / O peripheral).

[0012] The host can access an intelligent I / O peripheral (e.g., a PCIe-based intelligent I / O peripheral with direct PCIe I / O communication) to map physical functions and virtual functions (e.g., PCIe physical and virtual functions) into host-abstracted application operating environments. The backend I / O services provided by the intelligent I / O peripheral can be fully managed by the intelligent I / O peripheral independently of the host. Furthermore, the lifecycles of these services can be controlled by the intelligent I / O peripheral independently of the host.

[0013] Therefore, a segmented compute node with a cloud-native infrastructure architecture and intelligent I / O peripherals abstracted from the hardware differs from a non-segmented compute node with a legacy architecture based on host-centric control. This difference creates an architectural gap in current server designs, both in the pre-boot and runtime environments. This architectural gap poses a challenge for control orchestration because the host is no longer the root of control.

[0014] One approach to addressing this problem and bridging the architectural gap is to modify a legacy server hardware architecture so that control of the server can be enshrined in a server service processor. For example, a slot connector (e.g., a PCIe connector) of the legacy server hardware architecture can be modified to support a sideband communication channel with a Baseboard Management Controller (BMC). Generally, the BMC is a service processor designed to manage the computing platform. This approach transfers control to the BMC, extending the BMC's traditional role to include managing intelligent I / O peripherals by passing management traffic over the sideband communication channel.Another approach to bridging the architectural gap may be to modify the legacy server's hardware architecture to include an additional service processor, a platform controller, and a root controller within the platform controller, so that the platform controller monitors both the BMC and the intelligent I / O peripheral.

[0015] The approaches described above for bridging the architectural gap in a segmented server require changes to the architectures of next-generation servers. However, changing the hardware architecture of legacy servers to accommodate intelligent I / O peripherals in next-generation servers does not provide backward compatibility for existing servers.

[0016] According to the example implementations described here, a trust domain (here referred to as the "cloud tenant domain") of a server's host is intermingled with a hardware-isolated trust domain (here referred to as the "cloud operator domain") of an intelligent I / O peripheral of the server. This intermingling ensures that the host's services that can be offloaded to the intelligent I / O peripheral are secured and controlled both before and after the operational environment is deployed. In other words, the host's software stack is intermingled with the offloaded services, in accordance with the example implementations. This intermingling of the host's offloaded services with the host software stack can affect the abstracted application operating environments (e.g.,Bare-metal OS environments, hypervisor VM environments, and OS container applications supported by the intelligent I / O periphery. Consistent with sample implementations, a server includes a host monitoring service (or "monitoring service") for each application operating environment.

[0017] For a bare-metal application operating environment (i.e., an environment where the software has full access to the host CPU and other software-accessible resources of the physical platform), the server provides a host monitoring service that connects the backend I / O services provided by the intelligent I / O peripheral to the host domain in a single security trust domain. Furthermore, according to example implementations, the bare-metal operating system can provide hypervisor and container application operating environments, and the server can provide corresponding host monitoring services for these environments. The host monitoring services provide a way to compensate for the architectural gap created by the opacity of the intelligent I / O peripheral's architecture.

[0018] Using host monitoring services to manage smart I / O peripherals can cause problems because smart I / O peripherals behave differently than traditional peripherals. For example, a particular host monitoring service may perform PCIe bus enumeration, and the smart I / O peripheral may be a PCIe card. According to the PCIe standard, a PCIe bus device should respond to a valid configuration request within one second after the PCIe bus device is reset or powered on. If the PCIe bus device does not respond to a valid configuration request within one second, the PCIe bus device is traditionally considered to have failed.

[0019] However, an intelligent I / O peripheral can be relatively more complex than a traditional PCIe bus device that does not provide intelligent I / O services. Due to this complexity, a fully functional PCIe-based smart I / O peripheral may not be able to respond within the one-second time limit after the smart I / O peripheral is powered on or reset. Unlike traditional PCIe bus devices, an intelligent I / O peripheral can have a relatively complex architecture, for example, one that includes a complex multi-core processing system and hardware accelerator engines (e.g., cryptography engines and packet inspection engines).The intelligent I / O peripheral, which responds to a boot event, can also be controlled over the network by a domain manager to determine a physical infrastructure inventory and establish logical domain connections, such as SDN, SDC, and SDS connections. Due to this complexity, the intelligent I / O peripheral can take a relatively long time to become responsive after the intelligent I / O peripheral is powered on or reset (compared to a non-intelligent PCIe I / O peripheral).

[0020] An intelligent I / O peripheral may also experience a significant, indefinite delay when configuration changes are made to the intelligent I / O peripheral, such as changes that modify logical connections or when the intelligent I / O peripheral otherwise makes changes to its backend I / O services. For example, the number of processing cores assigned to a VM instance may be increased, and because of this change, an overlay network connection may be added for a backend I / O service used by the VM instance. For example, adding the overlay network connection may mean that the intelligent I / O peripheral configures or reconfigures a virtual function provided by the intelligent I / O peripheral.Accordingly, the intelligent I / O peripheral may experience a relatively long delay before the virtual function is ready for use.

[0021] Accordingly, due to its relative complexity, an intelligent I / O peripheral may not respond as expected from traditional I / O peripherals.

[0022] According to example implementations, an intelligent I / O peripheral provides a readiness indicator that a host monitoring service can access to determine whether the intelligent I / O peripheral is ready or not. In this context, according to example implementations, the "readiness" of the intelligent I / O peripheral generally means whether the intelligent I / O peripheral is ready to proceed with the host. For example, the readiness indicator can indicate whether a configuration area of ​​the intelligent I / O peripheral is ready to be configured by a host monitoring service. Another example is that the readiness indicator indicates whether a particular function of the intelligent I / O peripheral is available or not.

[0023] As a more specific example, in response to a server power-on or reset, a host monitoring service may detect the presence of peripherals (e.g., PCIe bus peripherals) as part of bus enumeration and, for each detected peripheral, establish a peripheral configuration space with the addresses of the allocated memory and / or I / O space. For example, after detecting the presence of the intelligent I / O peripheral, the host monitoring service may access a ready indicator of the intelligent I / O peripheral to determine whether or not the intelligent I / O peripheral is in a state that allows the host monitoring service to proceed with establishing the intelligent I / O peripheral configuration space.The Host Monitoring Service can therefore hold off on setting up the configuration until the intelligent I / O peripheral sets the ready indicator to a state that allows the Host Monitoring Service to proceed with the setup.

[0024] As another example, an intelligent I / O peripheral may provide single-root input-output virtualization (SR-IOV), which provides sets of virtual functions for corresponding physical functions of the intelligent I / O peripheral. For example, in response to a VM instance being activated, a hypervisor-based host monitoring service may lock the VM instance's use of an I / O service to be provided by the intelligent I / O peripheral. As further described herein, the locking of the I / O service may remain in effect until the intelligent I / O peripheral sets one or more ready-state indicators associated with virtual and / or physical functions associated with the I / O service to indicate that the function(s) are ready to be used.

[0025] Therefore, according to example implementations, the readiness indicators enable the extension of host monitoring services to the cloud operator domain to manage intelligent I / O peripherals without requiring changes to legacy server architectures and without rigid timing constraints that do not account for the complex nature of intelligent I / O peripherals. According to example implementations, the extension of host monitoring services into the cloud operator domain may involve the vertical integration of host software (e.g., host software running to execute the host monitoring services, smart I / O peripheral hardware, and smart I / O peripheral software). In this vertical integration, the intelligent I / O peripheral hardware may, for example, be designed to manage a register space that stores bits representing corresponding readiness indicators.A software service stack of the intelligent I / O peripheral can be configured to manage the readiness indicators. Extending host monitoring services into the cloud operator domain can involve modifying the hardware of the intelligent I / O peripheral (e.g., modifying a register space to store bits representing corresponding readiness indicators). Furthermore, extending host monitoring services to the cloud operator domain can involve modifying the software of the intelligent I / O peripheral (e.g., modifying a software service stack of the intelligent I / O peripheral) to manage the readiness indicators.

[0026] With reference to Fig. 1, as a more specific example, a cloud-based computing system 100 associated with a particular cloud operator may provide multi-tenant cloud services to multiple customers or tenants. The cloud services may be any number of different cloud services, such as Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and so on. Furthermore, depending on the implementation, the cloud services may be associated with one of several different cloud infrastructures, such as a public cloud available to all potential users over a public network, a private cloud with limited access provided over a private network (e.g., cloud services provided by an on-premises data center), or a hybrid cloud, which is a combination of public and private clouds.

[0027] Tenants may access the domain nodes 110 (e.g., compute nodes) of the cloud computing system 100 via cloud clients (e.g., laptops, desktop computers, smartphones, tablet computers, portable computers, etc.). As in Fig. 1, the domain nodes 110 may be interconnected by a physical network fabric 184. In general, the physical network fabric 184 may be connected to one or more types of communication networks, such as (as examples) Fibre Channel networks, Gen-Z fabrics, dedicated management networks, local area networks (LANs), wide area networks (WANs), global networks (e.g., the Internet), wireless networks, or any combination thereof. As shown in Fig. 1, the domain nodes 110 may be interconnected by logical connections 194, such as SDN, SDC, and SDS connections.

[0028] As also in Fig. 1, a domain node manager 190 (or "node manager 190") may be connected to the physical network fabric 184 and the logical connections 194. Depending on the particular implementation, the node manager 190 may be located on the domain node 110-1, on another domain node 110, on a dedicated management node, distributed across multiple nodes of the cloud computing system 100, etc. Generally, the node manager 190 includes physical hardware resources and logical software resources for managing the domain nodes 110. In accordance with some implementations, the node manager 190 may be connected to a database that stores data related to the domain nodes 110, such as data representing domain node configurations, telemetry logs, audit logs, etc.Node manager 190 may include an orchestration engine that manages the operation of leaf managers of cloud computing system 100 to control the lifecycles and configure the leaf resources (e.g., the intelligent input / output peripherals 180 described herein) of system 100. In accordance with some implementations, node manager 190 may communicate intent-based commands (e.g., Redfish commands) with one or more vendor root managers (not shown). The vendor root manager includes physical hardware resources and logical software resources to translate the intent commands into vendor-specific commands for the leaf resources associated with a particular vendor, and these vendor-specific commands may be processed by leaf resource managers (not shown) for the leaf resources.Depending on the particular implementation, the vendor root managers and leaf resource managers may be located on one or more administrative nodes, distributed on one or more domain nodes 110-1, and so on.

[0029] Domain node 110-1 may be a computing platform according to example implementations. In this context, a "computing platform" refers to a unit comprising a chassis and hardware mounted on the chassis, where the hardware is capable of executing machine-executable instructions (or "software"). A blade server is an example of a computing platform according to one example implementation. However, according to further implementations, the computing platform may be a platform other than a blade server, such as a rack server, a client, a desktop, a smartphone, a storage array, a laptop computer, a tablet computer, and so on.

[0030] In example implementations where the computing platform is a blade server, the server may have a frame or chassis; one or more motherboards may be attached to the chassis; and each motherboard may contain one or more multi-core semiconductor packages (or "sockets" or "chips") of the central processing unit (CPU). Consistent with example implementations, the blade server may have a form factor, mechanical lock(s), and corresponding electrical connectors to allow the server blade to be installed and removed from a corresponding server blade opening or slot in a rack-mounted blade enclosure.

[0031] In accordance with example implementations, domain node 110-1 may have a cloud-segmented architecture in which intelligent I / O peripherals 180 of domain node 110-1 may control various logical connection segments independently of the host of domain node 110-1. As in Fig. 1, domain node 110-1 may provide one or more application operating environments 120 located within a cloud tenant domain 111. In general, tenant application instances (e.g., VM instances, container instances, non-virtualized application instances, etc.) may execute in application operating environments 120. In general, application operating environments 120 may represent both virtualized and non-virtualized environments.

[0032] For example, an application operating environment 120 may be a bare-metal operating system (OS) environment (an "OS bare-metal application operating environment") that contains application instances that have access to the non-abstracted physical resources of domain node 110-1. As another example, an application operating environment 120 may be an OS container bare-metal application operating environment in which application instances may execute within container instances. As another example, an application operating environment 120 may be an OS bare-metal / VM environment in which application instances may execute within VM instances or outside of VM instances. As another example, an application operating environment 120 may be an OS container bare metal / V / VM environment in which application instances may run inside container instances, inside VMs, or outside of a VM or container instance.

[0033] In this context, a "virtual machine" or "VM" (also called "virtual guest machine," "virtual machine instance," or "virtual guest machine instance") refers to a virtual environment that acts as a machine-level abstraction or virtual computer system and has its own physical resources (e.g., CPU(s), system memory, network interface(s), and storage). In addition, the VM may have its own abstraction of an operating system, and generally, the VM is a virtual abstraction of the hardware and software resources of domain node 110-1. The lifecycle (e.g., provisioning and termination) of the VM may be managed by a Virtual Machine Monitor (VMM) or hypervisor 167 of domain node 110-1.

[0034] A "container" (also called an "instantiated container," a "container instance," or a "software container"), as used herein, generally refers to a virtual runtime environment for one or more applications and / or application modules, and this virtual runtime environment is structured to interface with an operating system kernel. For example, a container for a particular application may contain the executable code for the application and its dependencies, such as system tools, libraries, configuration files, executable files, and binaries for the application. Consistent with example implementations, the container contains an interface for incorporating the operating system kernel, but not the operating system kernel. For example, a particular computing platform may contain multiple containers that share an operating system kernel through corresponding operating system kernel mount interfaces.Docker containers and rkt containers are examples of software containers.

[0035] According to example implementations, each application operating environment 120 may have an operating system or hypervisor interface 121 (referred to as an "operating system hypervisor interface 121"), and collectively, the operating system hypervisor interfaces 121 may form a tenant workload isolation barrier 197 between the cloud tenant domain 111 and a cloud operator domain 113 of the domain node 110-1. According to example implementations, the cloud tenant domain 111 is considered an untrusted domain of the cloud computing system 100 because the domain 111 is associated with cloud tenant software. The cloud operator domain 113 may be considered a trusted domain with respect to the cloud tenant domain 111.

[0036] In accordance with example implementations, the cloud operator domain 113 comprises a mixed physical and logical infrastructure, including physical hardware and trusted monitoring software components. The components associated with the monitoring services described herein are separated from the tenant workloads by the workload isolation barrier 197 and a workload isolation barrier 198 formed by the host interfaces of the intelligent I / O peripherals 180. The physical hardware components associated with the monitoring services may include, for example, one or more CPUs 116 and storage components forming a system memory 118.According to example implementations, system memory 118 and other memories discussed herein are non-transitory storage media that may be formed from semiconductor memory devices, memristor-based memory devices, magnetic memory devices, phase-change memory devices, a combination of devices from one or more of these memory technologies, etc. System memory may consist of both volatile and non-volatile memory devices.

[0037] The trusted monitoring software components of the cloud operator domain 113 may include, for example, a hypervisor 167, a basic input / output operating system (BIOS) 165, a Unified Extensible Firmware Interface (UEFI), and an operating system 163. As also described in Fig. 1, the intelligent I / O peripherals 180 are located within the cloud operator domain 113. The cloud operator domain 113 may, according to example implementations, include one or more host monitoring services provided by trusted monitoring software components. For example, according to example implementations, the cloud operator domain 113 may include a pre-boot monitoring service 156 and one or more host monitoring services operating during the post-boot or runtime of the domain node 110-1.

[0038] In particular, in accordance with some implementations, the host runtime monitoring services may include an OS container runtime monitoring service 150, a hypervisor VM runtime monitoring service 152, and an OS bare metal runtime monitoring service 154. In general, the host monitoring services, as described herein, may serve a variety of purposes for the domain node 110-1 that bridge the architectural gap between the host-centric control model and the cloud-native control model.As examples, in accordance with example implementations, the host monitoring services may configure the intelligent I / O peripherals 180; control when locks on I / O services provided by the intelligent I / O peripherals 180 are released based on readiness indicators provided by the intelligent I / O peripherals 180; provision resources of the intelligent I / O peripherals 180; provide evidence of the operational states of the intelligent I / O peripherals 180; reconcile audit logs for the intelligent I / O peripherals 180; and so on.

[0039] According to an example implementation, the intelligent I / O peripheral 180 provides one or more readiness indicators (RSI). An RSI represents a corresponding readiness state of the intelligent I / O peripheral 180 and can be retrieved by a host monitoring service to determine the readiness state of the intelligent I / O peripheral 180. The "readiness" of the intelligent I / O peripheral 180 can be, for example, the readiness of the intelligent I / O peripheral 180 to be configured by a host monitoring service; the readiness of the intelligent I / O peripheral 180 to continue after an I / O service provided by the peripheral or a logical connection created by the peripheral 180 has been modified or created; the readiness of a physical or virtual function of the intelligent I / O peripheral 180; etc.

[0040] The RSIs may be RSIs for physical functions and / or virtual functions of the intelligent I / O peripheral 180. In particular, according to example implementations, the intelligent I / O peripheral 180 provides one or more physical function RSIs 182, where each physical function RSI 182 represents a ready state of a corresponding physical function (e.g., a physical PCIe function) of the intelligent I / O peripheral 180. In accordance with example implementations, the intelligent I / O peripheral 180 may provide one or more virtual function RSIs 183, where each virtual function RSI 183 represents a ready state of a corresponding virtual function (e.g., a virtual PCIe function) of the intelligent I / O peripheral 180.

[0041] As described further below, a host monitoring service may perform one or more actions based on RSIs provided by the intelligent I / O peripherals 180. For example, a host monitoring service may delay the launch of a VM instance or a container instance until an RSI corresponding to a virtual function or a physical function for that instance indicates that the function is available or ready. As another example, in response to an RSI condition, the host monitoring service may lock an instance from using a particular I / O service provided by an intelligent I / O peripheral while the intelligent I / O peripheral reconfigures the I / O service, and release the lock (thus allowing the instance to use the I / O service) when the RSI condition changes to indicate that the I / O service is ready.As another example, a host monitoring service may wait to set up a configuration range of an intelligent I / O peripheral 180 until an RSI indicates that the intelligent I / O peripheral 180 is ready to proceed with configuration.

[0042] In accordance with some implementations, a host monitoring service may read the RSIs directly from the intelligent I / O peripherals 180. However, in accordance with further implementations, a service processor of the domain node 110-1, such as a Baseboard Management Controller (BMC) 123, may monitor the RSI for the intelligent I / O peripherals 180 and update a register space 128 of the BMC 123 with values ​​representing the RSI states. For example, the register space 128 may contain one or more registers, with the registered bit fields corresponding to different RSIs indicating the respective RSI states. For example, the host monitoring service may read the BMC register space 128 over a sideband communication channel to determine a particular RSI state.

[0043] In accordance with some implementations, all error messages for domain node 110-1 are forwarded to BMC 123. Furthermore, in accordance with example implementations, BMC 123 may receive "hot-plug notifications" from the host (e.g., operating system 163 or hypervisor 167) in response to corresponding instance changes. A "hot-plug notification," as used herein, refers to a notification that a particular backend I / O service of intelligent I / O peripheral 180 is to be placed on hold. This interruption of service occurs while intelligent I / O peripheral 180 performs reconfiguration to account for the instance change. BMC 123 may, in accordance with example implementations, store data in its register space 128 representing the holding of a particular service provided by intelligent I / O peripheral 180.As used herein, an "instance change" (or "instance change event") refers to a change associated with an instance (e.g., a VM instance or a container instance) that corresponds to a change in one or more configurations of an intelligent I / O peripheral 180. For example, an instance change may be associated with a configuration change for a virtual function or a physical function of an intelligent I / O peripheral 180, a start of an instance, a termination of an instance, an allocation of processing resources (e.g., cores) to an instance, a release of resources for an instance, etc.

[0044] According to example implementations, in response to an instance change, the host monitoring service may suspend the instance's use of one or more affected I / O services until the intelligent I / O peripheral makes the configuration changes to account for the instance change. As further described herein, in accordance with example implementations, the host monitoring service restores normal operation of the instance (e.g., unblocks the instance's use of the I / O services) when one or more corresponding RSIs indicate that the corresponding intelligent I / O peripheral 180 configuration change(s) are present to resume execution of the instance.

[0045] In general, a "BMC" is a specialized service processor that monitors the physical state of a server, node, or other hardware device using sensors and communicates with a management system over a management network. The BMC can communicate with applications running at the operating system level through an input / output controller (IOCTL) interface driver, a REST application program interface (API), or another system software proxy that facilitates communication between the BMC and the applications. The BMC can access the hardware devices of the device at the hardware level, including system memory, local storage, and so on. The BMC can be capable of directly modifying the hardware devices. The BMC can operate independently of the hardware device's operating system instances. The BMC can be located on a motherboard or main circuit board of the hardware device.The fact that the BMC is mounted on the motherboard or otherwise connected or attached to the motherboard does not prevent the BMC from being considered "separate" from the processors monitored / managed by the BMC. As used herein, a BMC provides management capabilities for subsystems of the hardware unit and is separate from a processing resource running an operating system of the computing device.

[0046] Regardless of its particular form or implementation, the BMC 123 may include one or more embedded processors 124 that execute machine-executable instructions 125 (stored in a memory 126 of the BMC 123) to perform actions of the BMC 123 as described herein. In accordance with example implementations, these actions may include communicating with the intelligent I / O peripherals 180 to read the RSIs 182 and 183. In accordance with example implementations, the BMC 123 may provide one or more application programming interfaces (APIs) for reading the RSIs 182 and 183. For example, a host monitoring service may send an API request to the BMC 123 for a particular RSI value, and the processor 124 of the BMC 123 may, in response to the API request, read the RSI value and provide the read RSI value in a corresponding API response.In accordance with further implementations, the BMC 123 may periodically read the RSI values ​​and store the RSI values ​​in the register space 128 of the BMC 123. For example, a host monitoring service may read a specific register bit in the register space 128 to determine the state of a specific RSI. One or more processors 124 of the BMC 123 may, in accordance with example implementations, execute machine-executable instructions 125 for the purpose of receiving and managing hot-plug notifications from the host.

[0047] The BMC 123 may perform the tasks described above as part of its management layer. The BMC 123 may provide a variety of management services other than those described above, such as monitoring sensors (e.g., temperature sensors, fan speed sensors), monitoring the operating system status, monitoring the power status, controlling the powering on and off of the computer system, logging computer system events, enabling remote management of the computer system, performing recovery operations after an operating system or computer system failure, and so on. In accordance with some implementations, the BMC 123 may also have a security layer in which the BMC 123 performs various security-related functions for the domain node 110-1, such as:validating firmware before the firmware may be loaded and executed by processor 124; storing, generating, and providing cryptographic keys, and so on.

[0048] In accordance with example implementations, the intelligent I / O peripheral 180 may include one or more hardware processing cores 187 that execute instructions 189 stored in a memory 181 of the peripheral 180 to perform the functions described herein. In this way, in accordance with some implementations, the instructions 189 may be coupled to a software stack of the intelligent I / O peripheral 180. In general, one or more processors 187 may execute the software stack to manage the states of the RSIs 182 and 183 and to provide one or more backend I / O services for the intelligent I / O peripheral 180. In accordance with example implementations, one or more processors 187 may execute the instructions 189 to execute an example process 600 ( Fig. 6) to manage the RSI states. The intelligent I / O peripheral 180 may further include one or more hardware accelerators 175 (e.g., P4 hardware engines) to perform a variety of functions such as encryption, decryption, packet inspection (e.g., packet inspection for a distributed firewall service), etc. It should be noted that the intelligent I / O peripheral 180 may have a variety of different architectures, including components implemented in Fig. 1 are not shown. Among other components, the intelligent I / O peripheral 180 may include one or more network interface controllers (NICs) 186 that include corresponding ports (e.g., Ethernet ports) for communicating with the physical network fabric 184. In accordance with example implementations, the intelligent I / O peripherals form a network isolation barrier 199 of the domain node 110-1.

[0049] As in Fig. 1, in accordance with example implementations, the intelligent I / O peripheral 180 may include a configuration space 179, such as a PCIe configuration space, that is established or configured by a host monitoring service, as further described herein.

[0050] In accordance with some implementations, the intelligent I / O peripheral 180 includes a register space 177 containing bit fields representing corresponding RSIs (e.g., physical function RSIs 182 and virtual function RSIs 183). In accordance with example implementations, the register space 177 may be accessible by the host monitoring services (e.g., the register space 177 may be accessible through an in-band communication bus, such as a PCIe bus). In accordance with further implementations, the register space 177 may be accessible to the BMC 123 via a sideband communication bus (e.g., the register space 177 may be accessible via a Serial Peripheral Interface (SPI) bus or an extended SPI (eSPI) bus).

[0051] In accordance with some implementations, register space 177 may store data indicating whether intelligent I / O peripheral 180 is offline. For example, in some implementations, one or more processors 187 of intelligent I / O peripheral 180 may execute instructions 189 to manage the offline indication. In particular, in some implementations, a device (e.g., a CPU 116 or BMC 123) reading the state of an RSI 182 or 183 may acknowledge the read of the state by writing a corresponding acknowledge bit (e.g., a bit of register space 177). This allows intelligent I / O peripheral 180 to determine whether peripheral 180 is installed in a platform that uses the RSI. For example, after the intelligent I / O peripheral 180 has completed its power-up sequence, the intelligent I / O peripheral 180 may read an acknowledge bit corresponding to an RSI (e.g.,an RSI corresponding to primary function zero, as further explained herein) to determine whether a state of the RSI state has been read. For example, if the intelligent I / O peripheral 180 determines that the state has not been asserted and the configuration space 179 of the intelligent I / O peripheral 180 has not been set up, the intelligent I / O peripheral 180 may set an offline bit to indicate that the intelligent I / O peripheral 180 has taken itself offline.

[0052] With reference to Fig. 3A in conjunction with Fig. 1, the pre-boot monitoring service 156 operates during the pre-boot or pre-operating system environment of the domain node 110-1 (i.e., the environment of the domain node 110-1). A CPU 116 may execute BIOS or UEFI instructions to provide the pre-boot monitoring service 156, and the provisioning of the pre-boot monitoring service 156 may involve the CPU 116 communicating with the intelligent I / O peripheral 180 and the BMC 123. The pre-boot monitoring service 156 may establish visibility of the resources of the domain node 110-1 to the tenant, in accordance with example implementations. Resources may be hidden from a tenant for various reasons, such as: B. because the tenant does not want to pay for the resources to control the quality of service (since releasing the resources may affect the performance of domain node 110-1), etc.In accordance with some implementations, the pre-boot monitor 156 may appropriately control the visibility of certain physical functions and / or virtual functions of the intelligent I / O peripheral 180 so that the function(s) are not exposed to the node manager 190. The hidden resources may be any number of different resources, such as devices (e.g., graphics processing units (GPUs), CPUs, NVMe drives, etc.), memory above a certain capacity, CPU core frequency above a certain threshold, and so on. In general, the pre-boot monitor service 156 takes an inventory of the logical and physical infrastructure resources, controls the visibility of the resources of the domain node 110-1, and sets up the application operating environments 120. The pre-boot monitor service 156, in accordance with example implementations, checks for the node manager 190 (e.g.,the pre-boot monitor service 156 searches for the domain node 110-1 via an encrypted overlay network) and binds the domain node 110-1 to the node manager 190 for control.

[0053] As part of performing the pre-boot monitoring service 156, the CPU 116 may, for example, set up a PCIe configuration space of the intelligent I / O peripheral 116 and determine an operating state of the intelligent I / O peripheral 116. For example, the CPU 116 may determine whether the intelligent I / O peripheral 180 is active or offline. As part of the pre-boot monitoring service 156, the CPU 116 may perform other actions, such as setting up the application operating environments 120, reconciling audit logs, verifying evidence of an operating state, and so on.

[0054] As part of the pre-boot monitoring service 156, the CPU 116 may perform a bus enumeration (e.g., a PCIe bus enumeration), which includes detecting the presence of smart I / O peripherals 180 and establishing a bus configuration space (e.g., a PCIe configuration space) for each detected smart I / O peripheral. As a more specific example, in accordance with some implementations, the smart I / O peripheral 180 may be a PCIe smart I / O peripheral 180 that has a device 0, function 0 (herein called "primary function zero"). The smart I / O peripheral 180 controls a bit value (i.e., a physical function RSI 182) of a control configuration register associated with primary function zero to indicate whether the entire smart I / O peripheral 180 is ready to be configured.As further described herein, such a bit value may be useful for PCIe bus enumeration to allow a bus enumeration service to determine whether to proceed with configuring the intelligent I / O peripheral 180 after detecting the presence of the intelligent I / O peripheral 180. In accordance with further implementations, the intelligent I / O peripheral 180 may provide a physical RSI 182 that is distinct from an RSI associated with primary function zero to represent or indicate whether the intelligent I / O peripheral 180 is ready to be configured.For example, in some implementations, in response to the RSI of physical function zero indicating that the intelligent I / O peripheral 180 is ready for configuration, the host monitoring service 156 may write to the configuration space of the intelligent I / O peripheral 180 to set up the memory space and / or I / O space for the intelligent I / O peripheral 180.

[0055] As in Fig. 2 in conjunction with Fig. 1, due to its relative complexity, the intelligent I / O peripheral 180 may not be ready for configuration within a certain predetermined time (e.g., the one specified PCIe timeout). Fig. 2 shows a power-on sequence 200 for an intelligent I / O peripheral 180 according to an example implementation. As shown in Fig. 2, the power-on sequence occurs over a period of time 204 and includes interaction between the node manager 190 (e.g., a node manager of the cloud management system) and the intelligent I / O peripheral. After powering on or resetting the intelligent I / O peripheral 180, the peripheral 180 goes through a first phase 204 in which the intelligent I / O peripheral configures its physical infrastructure. This may include, for example, setting up audit logs, setting up telemetry logs, determining a security state, setting up a working baseline, etc. Next, in an SDN configuration phase 208, the intelligent I / O peripheral 180 sets up its SDN connections to one or more overlay networks. This may include setting up resilience logs, setting up performance logs, and evaluating a security status.The intelligent I / O peripheral 180 may then set up the SDS connections in an SDS configuration phase 212. Phase 212 may include determining memory capacities, establishing isolation and access, and assessing security status. In an SDC phase 216, the intelligent I / O peripheral 180 may determine a number of virtual core instances, determine virtual memory instance parameters, determine the number and types of network instances, and assess a security status. In accordance with example implementations, at the end of the SDC phase 216, the intelligent I / O peripheral 180 is ready to be configured by the pre-boot host monitoring service 156. In a subsequent service configuration phase 220, shown in FIG. Fig. As illustrated in Figure 2, the intelligent I / O peripheral 180 may set up one or more backend I / O services to be provided by the intelligent I / O peripheral 180. This setup may include instance provisioning and ordering, setting up instance audit and telemetry logs, setting up instance clusters, setting up failover clusters, setting up instance data replication, assessing instance certifications and compliance, etc.

[0056] In Fig. 4 illustrates a bus enumeration process 400 that may be used by the host monitoring service 156 in response to a power-on or reset of the computing platform 110-1. Referring to Fig. 4 in conjunction with Fig. 1 and Fig. 3A, process 400 may be initiated upon power-on or reset of domain node 110-1 for the purpose of discovering and configuring bus devices. Specifically, in accordance with example implementations, host monitor service 156 discovers intelligent I / O peripheral 180 (block 402).

[0057] For example, in some implementations, the intelligent I / O peripheral 180 may be a PCIe bus card. In general, during PCIe bus enumeration, the host monitoring service 156 may attempt to read a vendor identification (ID) and a device ID for various combinations of bus device, bus number, and physical function to discover corresponding PCIe bus devices. Here, a "PCIe bus device" refers to a specific PCIe physical function. Since all PCIe cards implement primary function zero (i.e., physical function 0 for device 0), the host monitoring service 156 may, in accordance with example implementations, attempt to read a bus device ID and a vendor ID for primary function zero for all bus numbers and bus device numbers to discover PCIe bus devices.

[0058] In accordance with example implementations, in response to detecting intelligent I / O peripheral 180, host monitoring service 156 determines the state of physical RSI 182 corresponding to primary function zero (block 404). Based on the state of RSI 182, host monitoring service 156 may then determine (decision block 408) whether intelligent I / O peripheral 180 is ready to be configured.

[0059] More specifically, in accordance with example implementations, the host monitoring service 156 may use the configuration space 179 ( Fig. 1) Program or configure the smart I / O peripheral 180 based on configuration criteria provided by the smart I / O peripheral 180 via data in configuration space 179. In implementations where the smart I / O peripheral 180 is a PCIe card, the smart I / O peripheral 180 may communicate, for example, via base address registers (BARs) of configuration space 280, appropriate memory criteria for the various corresponding physical functions of the smart I / O peripheral 180. In this way, the smart I / O peripheral 180 may, for example, request the allocation of memory or I / O space, as well as the specific amount of memory and / or I / O space, for a particular physical function.However, when the intelligent I / O peripheral 180 initially boots, the intelligent I / O peripheral 180 may not be ready to present the requested memory criteria for a period of time after power-on or reset, and the intelligent I / O peripheral 180 may not be ready for its configuration space 280 to be programmed by the host monitoring service 156. By monitoring the appropriate physical function RSI 182, the host monitoring service 156 can determine when to proceed with configuring the configuration space 280.

[0060] Therefore, after determining (decision block 408) that the intelligent I / O peripheral 180 is ready to proceed with configuration, the host monitoring service 156 communicates with the intelligent I / O peripheral 180 over the in-band communication channel 211, as per block 412, to determine the configuration criteria requested by the intelligent I / O peripheral 180. As per block 416, the host monitoring service 156 sets up the configuration area 280 of the intelligent I / O peripheral 180 based on the requested configuration criteria.

[0061] As in Fig. 3B, according to example implementations, the host monitoring service 150 may be connected to a bare-metal operating system with direct I / O application operating environment 120. In this example, an O / S container runtime host monitoring service 150 may respond to an indication from the BMC 123 that a particular configuration change event has occurred at a container instance 324. In response to the container change event, the host monitoring service 150 may perform a process 500 that Fig. 5 is shown.

[0062] With reference to Fig. 5 in conjunction with Fig. 1 and Fig. 3B, process 500 may be performed in response to a runtime container instance change event, such as the creation of a container instance, the termination of a container instance, the allocation of additional cores to a container instance, etc. These changes may correspond to configuration changes for one or more physical or virtual functions of the intelligent I / O peripheral 180. For example, a particular virtual function may be assigned to a new container instance, a new SDN connection may be established, an SDN connection may be removed, a virtual function may be reassigned to another container instance, etc.

[0063] According to process 500, the host monitoring service 150 places the container instance into a dormant operation mode (block 501) to account for the change event (ie, the container instance is decommissioned).

[0064] In this context, quiescing the container instance means quiescing or shutting down the container instance's operations affected by the change event. In other words, quiescing may put the container instance's use of one or more I / O services affected by the change event on hold. For example, the change event may correspond to a reconfiguration of a specific virtual function of the intelligent I / O peripheral, and quiescing the container instance may involve pausing the sending of workload transactions to a virtual function and draining or removing any pending transactions awaiting a response from the virtual function.According to block 502, the host monitoring service 150 communicates with the intelligent I / O peripheral 180 to initiate a configuration change in one or more services provided by the intelligent I / O peripheral 180. The intelligent I / O peripheral 180 may require an indefinite amount of time to make these changes. Accordingly, in accordance with example implementations, the host monitoring service 150 determines the state of the corresponding RSI 182 and / or 183 (block 504) and, based on the state, determines whether the changed function(s) are ready (decision block 508). If they are ready, the host monitoring service 150 returns the container instance to normal operating mode, according to block 512, in which the container instance resumes using the function(s) affected by the configuration change.

[0065] Similarly, a particular hypervisor VM runtime host monitoring service 152 may operate in conjunction with a hypervisor with direct I / O application operating environment 120 (see Fig. 3C). In this way, in response to a configuration change event at a particular VM instance 344, the host monitoring service 152 may perform a process to suspend operations of the VM instance 344 affected by the configuration change event until the corresponding RSI(s) indicates or represents that the corresponding backend I / O service(s) is / are ready; and when this occurs, the host monitoring service 152 returns the VM instance 344 to a normal mode. In accordance with further implementations, the hypervisor with direct I / O application operating environment 120 may Fig. 3C can be replaced by a kernel-based virtualization environment, such as a kernel-based virtual machine (KVM), which is a module of the Linux kernel.

[0066] Fig. 3D illustrates an OS bare-metal runtime host monitoring service 154 operating in conjunction with a bare-metal O / S instance-based operating environment 120. For example, the host monitoring service 154 may perform BIOS-related and / or UEFI-related configuration services using the RSI provided by the smart I / O peripherals 180. For example, in accordance with some implementations, the BMC 123 may receive a hot-plug notification in response to a particular card-based smart I / O peripheral 180 being inserted into or removed from the domain node 110-1 while the domain node 110-1 is powered on and in the runtime environment. In this example, the host monitoring service 154 may rely on an RSI of the smart I / O peripheral 180 to establish a configuration space of the peripheral 180, similar to process 400 ( Fig. 4). In accordance with example implementations, the host monitoring service 154 may perform other BIOS-related or UEFI-related monitoring services for the intelligent I / O peripherals 180.

[0067] Back to Fig. 1: In accordance with example implementations, one or more compute cores 187 of the intelligent I / O peripheral 180 may execute the instructions 189 for the purpose of managing the RSI 182 and 183 according to a process 600 described in Fig. 6. With reference to Fig. 6 in conjunction with Fig. 1, process 600 may be performed in response to a configuration or reconfiguration of logical connections and / or backend I / O services provided by intelligent I / O peripheral 180. According to process 600, processing core(s) 187 writes data to register space 177 (block 602) to change the state(s) of one or more RSIs 182 and / or 183 affected by the configuration to indicate that the corresponding physical and / or virtual functions are not ready. Processing core(s) 187 may then perform the configuration according to block 604.If the processing core(s) 187 determines (decision block 606) that the function(s) is (are) ready, then the processing core(s) 187 writes (writes) data to the register space 177 (block 612) to change the RSI state(s) affected by the configuration to indicate that the corresponding function(s) is (are) ready.

[0068] Referring to Fig. 7, in accordance with example implementations, a process 700 includes at least one hardware processor of a host executing first instructions (block 704) associated with an application operating environment and connected to an untrusted cloud tenant domain. The application operating environment uses an intelligent input / output (I / O) peripheral service; and the intelligent I / O peripheral service is connected to a trusted cloud operator domain. The process 700 includes extending (block 708) a monitoring service of the host into the trusted cloud operator domain. Extending the monitoring service includes the hardware processor(s) executing second instructions to monitor the intelligent I / O peripheral.Monitoring the intelligent I / O peripheral includes the hardware processor(s) configuring the intelligent I / O peripheral and determining a state of a ready indicator provided by the intelligent I / O peripheral. Monitoring the intelligent I / O peripheral further includes determining whether the intelligent I / O peripheral is ready to be configured based on the state.

[0069] As in Fig. 8, according to example implementations, a system 800 includes a port 804 and a node 812. The port 804 receives an intelligent input / output (I / O) peripheral. The intelligent I / O peripheral is to provide an I / O service associated with a trusted cloud operator domain. The node 812 includes at least a central processing unit (CPU) 816 and memory 820. The CPU(s) 816 execute instructions 822 associated with an application operating environment and an untrusted cloud tenant domain. The CPU(s) 816 execute instructions 824 associated with the trusted cloud operator domain to perform a monitoring service for the intelligent I / O peripheral in response to a change in an instance associated with the application operating environment. The change corresponds to a configuration change of the intelligent I / O peripheral.As part of the monitoring service, the CPU(s) execute instructions 824 to determine a state of a ready state indicator provided by the intelligent I / O peripheral and to regulate availability of the instance based on the state.

[0070] Referring to Fig.9, according to example implementations, a non-transitory storage medium 910 stores machine-readable first instructions 914 that, when executed by a machine, cause the machine to provide an input / output (I / O) service associated with a trusted cloud operator domain and with an intelligent I / O peripheral. The instructions 914, when executed by the machine, cause the machine to maintain a readiness indicator of the intelligent I / O peripheral that is accessible to a monitoring service of a host and represents a readiness state of the intelligent I / O peripheral. The monitoring service is associated with the trusted cloud operator domain. The host provides an instance associated with an application operating environment and associated with the untrusted cloud tenant domain. The instance uses the I / O service.Instructions 914, when executed by the machine, cause the machine, in response to a configuration event, to cause the ready state indicator to indicate that a function associated with the I / O service and provided by the intelligent I / O peripheral is unavailable. Furthermore, instructions 914, when executed by the machine, cause the machine to change a configuration associated with the function and, in response to the configuration change being completed, cause the ready state indicator to indicate that the function is available.

[0071] According to example implementations, the intelligent I / O peripheral comprises a bus device. Configuring the intelligent I / O peripheral involves setting up an intelligent I / O peripheral configuration space as part of a bus enumeration process. A particular advantage, according to some implementations, is that for a segmented domain node, an intelligent I / O peripheral can bridge an architectural gap between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware changes.

[0072] According to example implementations, the intelligent I / O device comprises a bus device associated with a primary physical function. Monitoring the intelligent I / O peripheral further includes the monitoring service detecting the presence of the bus device. Determining the state of the ready indicator includes determining a state of a ready indicator associated with the primary physical function. A particular advantage, according to some implementations, is that for a segmented domain node, an intelligent I / O peripheral can bridge an architectural gap between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware modification.

[0073] According to example implementations, the execution of the second instructions involves the execution of BIOS (Basic Input / Output System) or UEFI (Unified Extensible Firmware Interface) commands by the hardware processor(s). A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O peripheral, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0074] According to example implementations, the intelligent I / O peripheral undergoes a power-on sequence in response to power-on or reset, and the ready state indicator corresponds to a delay associated with the power-on sequence. A particular advantage, according to some implementations, is that for a segmented domain node, an intelligent I / O peripheral can bridge an architectural gap between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware changes.

[0075] According to example implementations, the intelligent I / O peripheral configures logical connections and a service associated with the logical connections during the power-on sequence. A particular advantage, according to some implementations, is that for a segmented domain node, an intelligent I / O peripheral can bridge an architectural gap between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware changes.

[0076] According to example implementations, the readiness indicator indicates whether a configuration change is complete. In response to a change in the instance, the instance stops using the I / O service. In response to the state indicating that the configuration change is complete, the lock is released. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O peripheral, an architectural gap can be bridged between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware changes.

[0077] According to example implementations, the change in the instance is a start of the instance, a termination of the instance, or a change in a resource allocation for the instance. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O edge, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any changes to legacy server hardware.

[0078] According to example implementations, the configuration change is a change that affects at least one virtual function of the intelligent I / O peripheral connected to the instance or one physical function of the intelligent I / O peripheral connected to the instance. A particular advantage, according to some implementations, is that for a segmented domain node, an intelligent I / O peripheral can bridge an architectural gap between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware changes.

[0079] According to example implementations, a baseboard management controller reads the state of the ready indicator and provides a representation of the state to the monitoring service. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O periphery, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring modification of legacy server hardware.

[0080] According to example implementations, a baseboard management controller stores a representation of the state in a baseboard management controller register space. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O peripheral, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0081] According to example implementations, a baseboard management controller receives a notification of the configuration change and notifies the monitoring service in response to receiving the notification. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O periphery, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0082] According to example implementations, the instance is a VM instance. An operating system locks the VM instance's use of the I / O service. In response to the state indicating that the configuration change is complete, the lock is released. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O edge, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0083] According to the example implementations, the instance is a container instance. A hypervisor locks the container instance's use of the I / O service. In response to the state indicating that the configuration change is complete, the lock is released. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O edge, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0084] According to example implementations, the intelligent I / O peripheral provides a single root input-output virtualization (SR-IOV), and changing the configuration corresponds to a virtual function of the SR-IOV. A particular advantage, according to some implementations, is that for a segmented domain node, an intelligent I / O peripheral can bridge an architectural gap between a host-centric control model and a cloud-native architecture without requiring any legacy server hardware changes.

[0085] According to example implementations, a monitoring service error is detected during the configuration of the intelligent I / O peripheral, and in response to the detected error, data is stored in a register space of the intelligent I / O peripheral indicating that the intelligent I / O peripheral is offline. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O peripheral, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0086] According to example implementations, the configuration event is an event associated with powering on or resetting the intelligent I / O peripheral. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O peripheral, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0087] According to example implementations, the configuration event is associated with an instance of the host's application operating environment. A particular advantage, according to some implementations, is that for a segmented domain node with an intelligent I / O periphery, an architectural gap between a host-centric control model and a cloud-native architecture can be bridged without requiring any legacy server hardware changes.

[0088] Although the present disclosure has been described with respect to a limited number of implementations, those skilled in the art, having the benefit of this disclosure, will appreciate numerous modifications and variations thereof. The appended claims are intended to cover all such modifications and variations.

Claims

[1] A process comprising: at least one hardware processor (116) of a node (110) executing first instructions associated with an application operating environment (120) and connected to an untrusted cloud tenant domain (111), wherein the application operating environment (120) uses an intelligent input / output peripheral (I / O) (180) service, the service is connected to a trusted cloud operator domain (113), and a host executes a monitoring service (150, 152, 154, 156) for the application operating environment (120); and Extending the monitoring service (150, 152, 154, 156) of the node (110) into the trusted cloud operator domain (113), wherein: extending the monitoring service (150, 152, 154, 156) comprises the at least one hardware processor (116) executing second instructions to monitor the intelligent I / O peripheral device (180); and monitoring the intelligent I / O peripheral (180) comprises the at least one hardware processor (116) configuring the intelligent I / O peripheral (180), determining a state of a readiness indicator provided by the intelligent I / O peripheral (180), and determining whether the intelligent I / O peripheral (180) is ready to be configured based on the state. [2] The method of claim 1, wherein: the intelligent I / O peripheral device (180) comprises a bus device; and configuring the intelligent I / O peripheral (180) comprises setting up a configuration space of the intelligent I / O peripheral (180) as part of a bus detection process. [3] The method of claim 1, wherein: the intelligent I / O peripheral (180) comprises a bus device connected to a primary physical function; monitoring the intelligent I / O peripheral device (180) further comprises the monitoring service (150, 152, 154, 156) detecting the presence of the bus device; and determining the state of the readiness indicator comprises determining a state of a readiness indicator associated with the primary physical function. [4] The method of claim 1, wherein executing the second instructions comprises the at least one hardware processor (116) executing at least one of the BIOS (Basic Input / Output System) or UEFI (Unified Extensible Firmware Interface) instructions. [5] The method of claim 1, wherein the intelligent I / O peripheral (180) undergoes a power-on sequence in response to the power-on or reset, and the ready state indicator corresponds to a delay associated with the power-on sequence. [6] The method of claim 5, wherein the intelligent I / O peripheral (180) configures logical connections and a service associated with the logical connections in the power-up sequence. [7] A system (100) consisting of: a port for receiving an intelligent input / output (I / O) peripheral (180), the intelligent I / O peripheral (180) providing an I / O service associated with a trusted cloud operator domain (113); a node (110) comprising at least one central processing unit (CPU) (116) and a memory (118), wherein the at least one CPU (116) executes first instructions associated with an application operating environment (120) and associated with an untrusted cloud tenant domain (111), and the at least one CPU (116) executes second instructions associated with the trusted cloud operator domain (113), for: Performing a monitoring service (150, 152, 154, 156) for the intelligent I / O peripheral (180) in response to a change in an instance associated with the application operating environment (120), wherein the change corresponds to a configuration change of the intelligent I / O peripheral (180); and Determining, as part of the monitoring service (150, 152, 154, 156), the state of a readiness indicator provided by the intelligent I / O peripheral (180) and regulating the availability of the instance based on the state. [8] The system (100) of claim 7, wherein the ready state indicator indicates whether the configuration change is complete, and the second instructions, when executed by the at least one CPU (116), further cause the at least one CPU (116) to: in response to the change in the instance, stop a function for the instance's use of the I / O service; and in response to the condition indicating that the configuration change is complete, release the hold. [9] The system (100) of claim 7, wherein the change to the instance comprises starting the instance, terminating the instance, or changing a resource allocation for the instance. [10] The system (100) of claim 7, wherein the configuration change comprises a change affecting at least one virtual function of the intelligent I / O peripheral (180) connected to the instance or a physical function of the intelligent I / O peripheral (180) connected to the instance. [11] The system (100) of claim 7, further comprising: a base board management controller (123) to read the state of the ready indicator and provide a representation of the state to the monitoring service (150, 152, 154, 156). [12] The system (100) of claim 11, wherein the base board management controller (123) stores a representation of the state in the register space (128) of the base board management controller (123), and the second instructions, when executed by the at least one CPU (116), cause the at least one CPU (116) to access the register space (128) to read the representation. [13] The system (100) of claim 7, further comprising: a base board management controller (123) to receive a notification of the configuration change and to notify the monitoring service (150, 152, 154, 156) in response to receiving the notification. [14] System (100) according to claim 7, wherein: the instance comprises an instance of a virtual machine; and the second instructions, when executed by the at least one CPU (116), cause an operating system (163) to suspend the use of the I / O service by the virtual machine instance and, in response to the condition indicating that the configuration change is complete, unsuspend. [15] System (100) according to claim 7, wherein: the instance includes a container instance; and the second instructions, when executed by the at least one CPU (116), cause a hypervisor (167) to suspend the use of the I / O service by the container instance and, in response to the condition indicating that the configuration change is complete, to unsuspend it. [16] The system (100) of claim 7, wherein the intelligent I / O peripheral (180) provides a single root input-output virtualization (SR-IOV) and the configuration change corresponds to a virtual function of the SR-IOV. [17] A non-transitory storage medium for storing machine-readable instructions which, when executed by a machine, cause the machine to: Providing an input / output (I / O) service associated with a trusted cloud operator domain (113) and connected to an intelligent I / O peripheral (180); Managing a readiness indicator of the intelligent I / O peripheral (180), wherein the readiness indicator is accessible to a monitoring service (150, 152, 154, 156) of a host and represents a readiness state of the intelligent I / O peripheral (180), wherein the monitoring service (150, 152, 154, 156) is connected to the trusted cloud operator domain (113), the host provides an instance connected to a cloud tenant application operating environment (120) and connected to an untrusted cloud tenant domain (111), and the instance uses the I / O service; and In response to a configuration event, causing the readiness state indicator to represent that a function associated with the I / O service and provided by the intelligent I / O peripheral (180) is unavailable, changing a configuration associated with the function, and in response to the completed configuration change, causing the readiness state indicator to represent that the function is available. [18] The storage medium of claim 17, wherein the instructions, when executed by the machine, further cause the machine to detect an error of the monitoring service (150, 152, 154, 156) in the configuration of the intelligent I / O peripheral (180) and, in response to the detected error, to store data in a register space of the intelligent I / O peripheral (180) indicating that the intelligent I / O peripheral (180) is offline. [19] The storage medium of claim 17, wherein the configuration event comprises an event associated with powering on or resetting the intelligent I / O peripheral (180). [20] The storage medium of claim 17, wherein the configuration event is associated with an instance of an application operating environment (120) of the host.

Images