System for multiple PCIe hosts to share an SR-IOV device through a standard host driver

Abstract

An apparatus includes a plurality of upstream PCIe ports, each upstream PCIe port connected to a host, and a downstream PCIe port connected to a single root I / O virtualization (SR-IOV) device shared by the hosts. The apparatus provides access to virtual functions (VFs) of the SR-IOV device associated with physical functions (PFs). Control circuitry enumerates the SR-IOV device in an internal PCIe domain, emulates a first VF as a first separate PCIe device facing a first host connected through a first upstream PCIe port, emulates a second VF as a second separate PCIe device facing a second host connected through a second upstream PCIe port, and implements inter-domain bridging of PCIe transactions between the hosts and the VFs of the SR-IOV device.

Description

priority

[0001] This disclosure claims priority to Indian Application No. 202411016879, filed on March 8, 2024, the contents of which are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure relates to electronic devices, such as computers, that share device resources, and more specifically, to systems for multiple peripheral component interconnect rapid (PCIe) hosts to share a single root input / output virtualization (SR-IOV) device via standard host drivers. Background Technology

[0003] For multiple PCIe hosts sharing SR-IOV devices, some solutions can use sophisticated virtual intermediary software running on a given host, or sophisticated fabric-mode switches that support and manage the interconnection of multiple PCIe switches and devices to multiple hosts.

[0004] For example, one solution is to use a Microchip Switchtec PAX Advanced Fabric PCIe switch. PAX Fabric switches may be overkill and costly in some applications, and they involve more computing resources, including RAM, translation tables, and proprietary routing for PCIe transactions.

[0005] In another example, there might be dedicated software running on each host. This software is sometimes referred to as a virtual intermediary (VI). The VI software ensures that the processors of the various hosts work together, and that one host does not interfere with the operation of other hosts. The VI software for a given operating system (OS) is unique to the given OS on which the VI software can run. Therefore, for each OS that might be used on the various hosts sharing an SR-IOV device, a different version of the VI software might be required. Because the behavior is based on cooperation, problems could arise if a processor on the first host fails and writes into the space of the processor on the second host.

[0006] The examples disclosed herein can solve one or more of these problems. Attached Figure Description

[0007] Figure 1 This is an illustration of a device host for simultaneously sharing an SR-IOV device with a host, according to an example of this disclosure.

[0008] Figure 2 This is an illustration of the operation of configuring and managing physical functions and providing access to virtual functions associated with physical functions in an SR-IOV device, according to an example of this disclosure.

[0009] Figure 3 This is an illustration of an apparatus according to an example of this disclosure that emulates a virtual function as a separate PCIe device for the host while simultaneously sharing an SR-IOV device with the host.

[0010] Figure 4 This is an illustration of an apparatus according to an example of the present disclosure that provides a host with access to an additional PCIe device.

[0011] Figure 5 This is an illustration of the operation of a control circuit bridging a request from a given SR-IOV device, according to an example of this disclosure.

[0012] Figure 6 This is a diagram illustrating the operation of a control circuit that processes communication from a given host according to an example of this disclosure.

[0013] Figure 7 This is another illustration of the operation of a control circuit that processes communications from a given host according to an example of this disclosure.

[0014] Figure 8 This is a diagram illustrating the operation of a control circuit according to an example of this disclosure, processing communications from a corresponding virtual function.

[0015] Figure 9 This is another illustration of the operation of a control circuit that processes communications from a corresponding virtual function according to an example of this disclosure.

[0016] Figure 10 This is a more detailed illustration of a bridge based on an example of this disclosure.

[0017] Figure 11 This is a diagram illustrating an example method based on an example of this disclosure.

[0018] Figure 12 This is an illustration of an example article based on the present disclosure and an example method performed by such article.

[0019] Figure 13 This is a diagram illustrating an example method according to the present disclosure, which can be... Figure 12 A more detailed illustration of the method.

[0020] Figure 14 This is a diagram illustrating the operation of a control circuit according to an example of this disclosure processing a configuration read request for a logical virtual function from a given host.

[0021] Figure 15 This is a diagram illustrating the operation of a control circuit according to an example of this disclosure processing a configuration write request for a logical virtual function from a given host. Summary of the Invention

[0022] According to an aspect of this disclosure, an apparatus is provided. The apparatus includes: a plurality of upstream PCIe ports, a given upstream PCIe port being used to connect to a given host among a plurality of hosts. The apparatus includes: a first downstream PCIe port being used to connect to a single root I / O virtualization (SR-IOV) device shared by the plurality of hosts, the apparatus being used to provide access to a plurality of virtual functions (VFs) associated with the physical function (PF) of the SR-IOV device. The apparatus includes control circuitry configured to enumerate SR-IOV devices in PCIe partitions within the apparatus. The control circuitry is configured to emulate a first VF among the plurality of VFs as a first individual PCIe device for a first host among the plurality of hosts connected to the apparatus via a first upstream PCIe port among the upstream PCIe ports. The control circuitry is configured to emulate a second VF among the plurality of VFs as a second individual PCIe device for a second host among the plurality of hosts connected to the apparatus via a second upstream PCIe port among the upstream PCIe ports. The control circuitry is configured to enable inter-domain bridging of PCIe transactions between multiple hosts and multiple VFs of the SR-IOV device.

[0023] According to other aspects of this disclosure, the apparatus may include one or more of the following features: Control circuitry may be configured to configure and manage the power field (PF) and provide access to the virtual function field (VF) associated with the PF of the SR-IOV device. Control circuitry may be configured to emulate a first VF and a second VF as host-oriented first and second separate PCIe devices, respectively, to simultaneously share the SR-IOV device with the host. Control circuitry may emulate the first VF and the second VF as host-oriented first and second separate PCIe devices, respectively, to simultaneously share the SR-IOV device with the host while instructing a given host among the hosts that the given host has exclusive control over the given VF. Control circuitry may be configured to provide non-transparent access to the SR-IOV device to multiple hosts via the VF.

[0024] A first host can be configured to access a first upstream PCIe port in a first partition of the device. A second host can be configured to access a second upstream PCIe port in a second partition of the device. Control circuitry can be configured to access SR-IOV devices from a third partition of the device, which is an internal PCIe partition where the first, second, and third partitions of the device are separate partitions. A first host can be configured to access a second downstream PCIe port in the first partition of the device to access downstream devices. Control circuitry can be configured to implement an inter-domain PCIe bridge to route PCIe signals between a given host and a corresponding given VF via the third partition. Control circuitry can be configured to bridge or emulate configuration access requests for a corresponding given VF from a given host via the third partition.

[0025] The control circuitry can be configured to determine whether a request originating from an SR-IOV device originates from a PF or from one of a plurality of VFs. Based on the determination that the request originates from an SR-IOV device from one of a plurality of VFs, the control circuitry can bridge the request to the corresponding host. Otherwise, based on the determination that the request originates from an SR-IOV device from a PF, the control circuitry can process the request within a third partition. The control circuitry can be configured to perform one or more of the following: determine a first requester identifier and a first completer identifier of a first transaction layer packet (TLP) to bridge a configuration access request from a given host among the plurality of hosts to a corresponding VF in the third partition; determine a memory address, a second requester identifier, and a second completer identifier of a second TLP to bridge a first memory access, a first message, and a first completion from a given host among the plurality of hosts to a corresponding VF in the third partition; determine a third requester identifier of a third TLP to bridge a second memory access and a second message from a corresponding VF among the plurality of hosts to a given host in the third partition; and determine a fourth requester identifier and a third completer identifier of a fourth TLP to bridge a second completion from a corresponding VF among the plurality of hosts to a given host in the third partition.

[0026] The control circuitry can process configuration read requests for a corresponding LVF from a given host in the following ways: based on the configuration register address in the configuration read request, determine whether the requested configuration data is based on input from one or more of the local storage device, the PF, or the corresponding VF; perform one or more of the following: based on the determination that the requested configuration data requires input from the local storage device, retrieve stored configuration simulation data from the local storage device for processing in the data processor; based on the determination that the requested configuration data requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor; and based on the determination that the requested configuration data requires input from the corresponding VF, retrieve VF configuration data from the corresponding VF for processing in the data processor; process one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor, and bridge them through the corresponding LVF to return the processed configuration data to the given host for multi-host emulation of the SR-IOV device; and update the NTB rules.

[0027] The control circuitry can process configuration write requests for a given LVF from a given host in the following ways: Based on the configuration register address in the configuration write request, and based on input from one or more of the local storage device, the PF, or the corresponding VF, determine whether configuration data should be written; perform one or more of the following: based on the determination that the configuration data to be written requires input from the local storage device, retrieve stored configuration simulation data from the local storage device for processing in the data processor; based on the determination that the configuration data to be written requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor; and based on the determination that the configuration data to be written requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor; and based on the determination that the configuration data to be written... The process requires input from the corresponding VF, retrieves VF configuration data from the corresponding VF for processing in the data processor, and processes one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor to produce processed configuration data. If the configuration register address addresses a valid register supported by the corresponding LVF, the processed configuration data is written to one or more of the local storage device, PF, and corresponding VF. The processed configuration data is then used for multi-host simulation of the SR-IOV device to a given host via the corresponding LVF. The completion status is returned to the given host via the corresponding LVF. The NTB rules are then updated.

[0028] According to another aspect of this disclosure, an article of manufacture is provided. The article of manufacture includes instructions that, when loaded and executed by a processor, cause the processor to enumerate a single root I / O virtualization (SR-IOV) device in a PCIe partition within the device. The device includes: a plurality of upstream PCIe ports, a given upstream PCIe port for connection to a given host among a plurality of hosts; and a first downstream PCIe port for connection to an SR-IOV device shared by the plurality of hosts. The device provides access to a plurality of virtual functions (VFs) associated with the physical function (PF) of the SR-IOV device. The instructions cause the processor to emulate a first VF among the plurality of VFs as a first separate PCIe device for a first host among the plurality of hosts connected to the device via a first upstream PCIe port among the upstream PCIe ports. The instructions also cause the processor to emulate a second VF among the plurality of VFs as a second separate PCIe device for a second host among the plurality of hosts connected to the device via a second upstream PCIe port among the upstream PCIe ports. These instructions enable the processor to program the inter-domain PCIe bridging rule table to provide non-transparent access between the SR-IOV device and multiple hosts via VFs. These instructions enable the processor to bridge configuration access requests from a given host to the corresponding given VF via a third partition. These instructions enable the processor to emulate configuration data for configuration access requests from a given host. These instructions enable the processor to manage inter-domain bridging of PCIe transactions between multiple hosts and the VFs of the SR-IOV device.

[0029] According to other aspects of this disclosure, the article of manufacture may include one or more of the following features: These instructions enable a processor to configure and manage a PF and provide access to a VF associated with the PF of an SR-IOV device. These instructions enable a processor to emulate a first VF and a second VF as host-oriented first separate PCIe devices and second separate PCIe devices, respectively, to simultaneously share the SR-IOV device with a host. These instructions enable a processor to emulate a first VF and a second VF as host-oriented first separate PCIe devices and second separate PCIe devices, respectively, to simultaneously share the SR-IOV device with a host while instructing a given host in the host hierarchy that the given host has exclusive control over the given VF. These instructions enable a processor to program an inter-domain PCIe bridging rule table to provide non-transparent access to the SR-IOV device to multiple hosts via the VF.

[0030] A first host can be configured to access a first upstream PCIe port in a first partition of the device. A second host can be configured to access a second upstream PCIe port in a second partition of the device. These instructions enable the processor to access the SR-IOV device from a third partition of the device, where the first, second, and third partitions of the device are separate partitions. These instructions enable the processor to program and manage inter-domain PCIe bridges to route PCIe signals between a given host and a corresponding given VF through the third partition.

[0031] These instructions may cause the processor to perform one or more of the following: (A) determine whether a request originating from an SR-IOV device originates from a PF or from one of a plurality of VFs; based on the determination that the request originating from an SR-IOV device originates from one of a plurality of VFs, bridge the request to the appropriate host; and otherwise, based on the determination that the request originating from an SR-IOV device originates from a PF, process the request within a third partition; or (B) determine the first requester identifier and the first completer identifier of a first transaction layer packet (TLP) to bridge a configuration access request from a given host among the plurality of hosts to the appropriate VF in the third partition.

[0032] These instructions enable the processor to process configuration read requests for a given LVF from a given host in the following ways: determining, based on the configuration register address in the configuration read request, whether the requested configuration data is based on input from one or more of the local storage device, the PF, or the corresponding VF; performing one or more of the following: retrieving stored configuration simulation data from the local storage device for processing in the data processor based on the determination that the requested configuration data requires input from the local storage device; retrieving PF configuration data from the PF for processing in the data processor based on the determination that the requested configuration data requires input from the PF; and retrieving VF configuration data from the corresponding VF for processing in the data processor based on the determination that the requested configuration data requires input from the corresponding VF; processing one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor, bridging through the corresponding LVF, and returning the processed configuration data to the given host for multi-host emulation of the SR-IOV device; and updating the NTB rules.

[0033] These instructions enable the processor to process configuration write requests for a given LVF from a given host in the following ways: Based on the configuration register address in the configuration write request, determine whether the configuration data to be written is based on input from one or more of the local storage device, the PF, or the corresponding VF; perform one or more of the following: based on the determination that the configuration data to be written requires input from the local storage device, retrieve stored configuration simulation data from the local storage device for processing in the data processor; based on the determination that the configuration data to be written requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor; and based on... Upon determining that the configuration data to be written requires input from the corresponding VF, retrieve the VF configuration data from the corresponding VF for processing in the data processor; process one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor; if the configuration register address addresses a valid register supported by the corresponding LVF, write the processed configuration data to one or more of the local storage device, PF, and corresponding VF; bridge the processed configuration data via the corresponding LVF for multi-host simulation of the SR-IOV device to a given host; return the completion status to the given host via the corresponding LVF; and update the NTB rules.

[0034] According to another aspect of this disclosure, a method is provided. The method includes, at a PCIe switch, enumerating a single root I / O virtualization (SR-IOV) device in a PCIe partition within the device, the device including: a plurality of upstream PCIe ports, a given upstream PCIe port for connection to a given host among a plurality of hosts; and a first downstream PCIe port for connection to the SR-IOV device, the SR-IOV device being shared by the plurality of hosts, the device providing access to a plurality of virtual functions (VFs) associated with the SR-IOV device's physical function (PF). The method includes emulating a first VF among the plurality of VFs as a first individual PCIe device facing a first host among the plurality of hosts, the first host being connected to the device via a first upstream PCIe port among the upstream PCIe ports. The method includes emulating a second VF among the plurality of VFs as a second individual PCIe device facing a second host among the plurality of hosts, the second host being connected to the device via a second upstream PCIe port among the upstream PCIe ports. The method includes implementing inter-domain bridging of PCIe transactions between the plurality of hosts and the individual PCIe device. Detailed Implementation

[0035] This disclosure relates to networking of electronic devices, and more specifically to systems for multiple PCIe hosts to share SR-IOV devices via standard host drivers. The complexity of multiple hosts sharing input / output (I / O) devices can be abstracted in, for example, hardware (HW) and firmware (FW) implementations. Virtual intermediary software components may not be required compared to other solutions.

[0036] The examples disclosed herein allow multiple PCIe host processors to share virtualized I / O devices without using dedicated host software and without impacting data throughput. This can offer lower costs than fabric-based switches compared to other solutions. Furthermore, it may eliminate the need to run complex virtualization mediation software on the host system.

[0037] Figure 1 This is an illustration of an apparatus 100 according to an example of this disclosure. Apparatus 100 can be implemented in any suitable manner, such as via a PCIe switch. Apparatus 100 may include control circuitry 102. Furthermore, apparatus 100 may include any suitable number and type of PCIe ports 110, 112, 130. Apparatus 100 can be configured to connect to any suitable number and type of PCIe hosts 114, 116. Apparatus 100 can be configured to share access to any suitable number and type of SR-IOV devices 106 with any suitable number and type of hosts 114, 116. Although... Figure 1 A single instance of SR-IOV device 106 is shown, along with two hosts 114 and 116, but any suitable number and type of SR-IOV device 106 and hosts 114 and 116 can be used.

[0038] PCIe ports 110, 112, and 130 can be configured in any suitable manner. PCIe ports 110 and 112 can each be configured to connect to a given host among hosts 114 and 116. PCIe ports 110 and 112 can be upstream PCIe ports. PCIe port 130 can be configured to connect to SR-IOV device 106. PCIe port 130 can be a downstream PCIe port.

[0039] Device 100 can be configured to facilitate the sharing of SR-IOV device 106 by hosts 114 and 116. SR-IOV device 106 may include physical functions (PF) 128 and any suitable number and type of virtual functions (VF), such as VF1 122, VF2 124, and VF3 126. Virtual functions may be functions that share one or more physical resources of PF 128. However, each virtual function may be individually assigned to a single host such as host 114 or 116, where each host understands or perceives itself as the sole operator of the corresponding PF 128 or device 106. By assigning different virtual functions among virtual functions 122, 124, and 126 to different hosts 114 and 116, device 106 can be effectively shared by hosts 114 and 116, even though sharing of SR-IOV device 106 across multiple PCIe hosts is not permitted under the PCIe specification.

[0040] Device 100 can be configured to allow SR-IOV device 106 to be shared by hosts 114, 116. Specifically, this sharing can be enabled by control circuitry 102. Control circuitry 102 can be implemented in any suitable manner, such as analog circuitry, digital circuitry, instructions executed by a processor, field-programmable gate arrays, application-specific integrated circuits, programmable logic, embedded processors, firmware, or any suitable combination thereof. Control circuitry 102 may include or be communicatively coupled to an article of art. The article of art can be implemented as a non-transitory memory, such as read-only memory, random access memory, or any other suitable memory. The article of art may include instructions. When loaded and executed by a processor, these instructions can cause the processor to perform the operations of control circuitry 102 as described in this disclosure.

[0041] Control circuitry 102 can be configured to provide inter-domain bridging for PCIe transactions between hosts 114, 116 and virtual functions 122, 124, 126. For illustrative purposes, in Figure 1 An inter-domain bridge 121 is illustrated, but such a bridge 121 can be implemented by different parts of the control circuit 102, as described in the following figures.

[0042] Control circuitry 102 can be configured to enumerate SR-IOV devices 106 within a PCIe partition (referred to as internal partition 104) within device 100. As described in the following figures, each of the hosts 114, 116 can be assigned additional partitions. A given host may only be aware of the partition it has been assigned, and activity and entities in other partitions may be invisible or opaque to such hosts. Figure 1 In this example, partition 104 may be inaccessible or invisible to either host 114 or 116. Port 130 can connect control circuitry 102 to device 106 and partition 104.

[0043] Control circuitry 102 can be configured to emulate a first virtual function, such as VF1 122, as a logical virtual function (LVF) LVF1 118 in a partition allocated to host 114. Host 114 can perceive LVF1 118 as a first PCIe device attached to host 114 via port 110. Similarly, control circuitry 102 can be configured to emulate a second virtual function, such as VF2 124, as LVF2 120 in another partition allocated to host 116. Host 116 can perceive LVF2 120 as a second PCIe device attached to host 116 via port 112.

[0044] An example implementation of host 114 could be an x86- or ARM-based CPU with a PCIe host. An example of SR-IOV device 106 could include an SR-IOV NVMe solid-state drive (SSD) or an SR-IOPV network interface controller (NIC).

[0045] In device 100, each host 114, 116 can be assigned its own partition within device 100. If a given host wants to access a virtualized I / O device (such as device 106), the given host must actually cross partition boundaries to partitions hosted by different hosts (including SR-IOV device 106). Control circuitry 102 can regulate host processor access to other partitions accessed via virtualization functions. Control circuitry 102 can apply a set of rules to cross such partition boundaries. Some examples of rules include the translation of requester IDs in transaction layer packets (TLPs), the translation of completer IDs in TLPs, or the translation of memory addresses in TLPs. Any suitable number and type of rules can be used.

[0046] For each SR-IOV device, a Virtual Function (VF) can be assigned to each host as needed. For example, such an allocation may include: SR-IOV device 0 may have Virtual Function 0 and Virtual Function 1 assigned to host 0; SR-IOV device 0 may have Virtual Function 2 and Virtual Function 3 assigned to host 1; SR-IOV device 0 may have Virtual Functions 4, 5, 6, and 7 assigned to host 2; SR-IOV device 1 may have Virtual Functions 0, 1, 2, and 3 assigned to host 1; and SR-IOV device 1 may have Virtual Function 4 and Virtual Function 5 assigned to host 0.

[0047] In one example, all data transactions can be handled in hardware by control circuitry 102 without impacting throughput. In another example, within the firmware, all configuration transactions can be redirected by hardware within control circuitry 102 to hypervisor firmware running on an embedded processor. This hypervisor firmware can operate either inside or outside control circuitry 102. The hypervisor firmware can maintain the illusion that each host owns a PCIe device bridged to the corresponding VF. For example, if a host issues a "power-down" command to an object it considers its device, the hypervisor firmware can respond with an acknowledgment, but if other processors are still active, the hypervisor firmware does nothing.

[0048] Although Figure 1 A single SR-IOV device 106 and a single PF 128 therein are shown, but any suitable number of SR-IOV devices can be connected to device 100 via corresponding downstream ports, and any suitable number of PFs can reside on a given SR-IOV device. SR-IOV devices can be otherwise configured to connect to a single PCIe host and share various functionalities of different virtual and physical capabilities with different operating systems running in the virtualization environment on the same host. Examples of this disclosure can allow multiple PCIe hosts to share virtualized I / O devices without using dedicated host software in hosts 114, 116.

[0049] The control circuitry 102 may utilize non-transparent bridging (NTB) (an inter-domain bridging technology for supporting multiple partitions), an embedded CPU running controller firmware including embedded hypervisor firmware, and a multi-host to single-root virtual function bridge 121. The control circuitry 102 may include these components or may be communicatively coupled to these components.

[0050] Figure 2 This is an illustration of the operation of an apparatus 100 for configuring and managing a PF and providing access to a VF associated with the PF in an SR-IOV device 106, according to an example of this disclosure.

[0051] Control circuitry 102 can issue management or configuration commands to power field 128 (PF) of SR-IOV device 106. These commands are discussed further below. Such commands can configure the use of different power fields (VFs) 122, 124, and 126 for use by various hosts 114 and 116. Then, when hosts 114 and 116 request access to SR-IOV device 106, control circuitry 102 can facilitate access by hosts 114 and 116 to the detected VFs 122, 124, and 126.

[0052] Figure 3This is an illustration of the operation of an apparatus 100, according to an example of this disclosure, for emulating a VF as a separate PCIe device for hosts 114, 116 to simultaneously share the SR-IOV device 106 with hosts 114, 116.

[0053] Host 114 can have exclusive surface access to VF1 122 via LVF1 118, which can represent a PCIe device virtualized or emulated by control circuitry 102. LVF1 118 can be mapped to VF1 122 and SR-IOV device 106. Simultaneously, host 116 can have exclusive surface access to VF2 124 via LVF2 120, which can represent a PCIe device virtualized or emulated by control circuitry 102. LVF2 120 can be mapped to VF2 124 within SR-IOV device 106.

[0054] Host 114 may not be able to see host 116's use or access to SR-IOV device 106. Host 116 may also be unable to see host 114's use or access to SR-IOV device 106. This can be achieved by control circuitry 102 emulating access to SR-IOV device 106 by hosts 114 and 116, such that each of hosts 114 and 116 is aware of the PCIe device's attachment to its own downstream port, but has no visibility to other partitions. Control circuitry 102 effectively instructs each host 114 and 116 that it believes the corresponding host has exclusive control over a given VF in SR-IOV device 106. Access to SR-IOV device 106 by a given host 114 or 116 is not transparent to other hosts.

[0055] Figure 4 This is an illustration of an apparatus 100 for providing access to additional PCIe devices for hosts 114, 116, according to an example of this disclosure.

[0056] The host 114 can access the upstream port 160 in the first partition 132 of the device 100, which can be connected to the upstream port (USP) PCI to PCI (P2P) bridge 138, which can be connected to a bus, which can be connected to the downstream (DSP) P2P bridge 139 and DSP P2P 172.

[0057] Host 116 can access upstream PCIe port 112 in the second partition 134 of device 100. PCIe port 110 can be implemented using USP P2P 140, which can be connected to a bus, which in turn can be connected to DSP P2P 142.

[0058] Control circuitry 102 can access SR-IOV devices from a third partition of device 100, which can be an internal PCIe partition 104. Partitions 104, 132, and 134 can be separate partitions. Access to resources in other partitions can be made via bridge 121 controlled by control circuitry 102.

[0059] Host 114 can access another downstream port 144, and another downstream device, such as PCIe device 148, can be connected to this downstream port. Host 114 can access PCIe device 148 through USP P2P bridge 138 to DSP P2P bridge 139 to downstream port 144.

[0060] Similarly, host 116 can access another downstream port 146, and another downstream device, such as PCIe device 150, can be connected to this downstream port. Host 116 can access PCIe device 150 via USP P2P bridge 140 to DSP P2P bridge 142 to downstream port 146.

[0061] Control circuitry 102 may include or be communicatively coupled to non-transparent bridge circuitry (NTB) 404 and embedded central processing unit (CPU) 402. Embedded CPU 402 may be configured to run a supervisor, firmware, or any other suitable instructions. NTB circuitry 404 may be implemented in any suitable manner, such as analog circuitry, digital circuitry, instructions executed by a processor, field-programmable gate arrays, application-specific integrated circuits, programmable logic, embedded processors, firmware, or any suitable combination thereof. In different specific implementations, operation of embedded CPU 402 may be performed by NTB circuitry 404, and vice versa. Embedded CPU 402 and NTB circuitry 404 may implement bridge 121, and in which VFs are emulated as LVF1 118 and LVF2 120.

[0062] Host 114 can be connected to LVF1 118, with the connection passing through upstream port 160, USP P2P bridge 138, the bus, and DSP P2P 172. Access to LVF1 118 can be routed via or through NTB circuit 404 to PCIe port 130 to reach SR-IOV device 106. This can be performed via upstream port 164 and downstream port 168. Furthermore, other data can be routed via DSP P2P 166, passing through downstream port 170 to other components. Similarly, host 116 can be connected to LVF2 120, with the connection passing through upstream port 162, USP P2P bridge 140, the bus, and DSP P2P 174. Access to LVF2 120 can be routed via or through NTB circuit 404 to PCIe port 130 to reach SR-IOV device 106. Therefore, PCIe signals can be routed by bridge 121 between a given host and the corresponding given VF via the third internal partition 104.

[0063] Figure 5 This is an illustration of the operation of control circuit 102 bridging a request from a given SR-IOV device 106, according to an example of this disclosure.

[0064] Control circuitry 102 can receive requests from SR-IOV device 106. Control circuitry 102 can determine whether the request originates from a PF or from a VF associated with the PF. If it originates from a VF, the request can be bridged to the corresponding host 114, 116 associated with the VF. If it originates from the PF, the request can be processed by control circuitry 102 in the third internal partition 104.

[0065] Figure 6 This is an illustration of the operation of a control circuit 102 processing communication from a given host, according to an example of this disclosure.

[0066] Control circuit 102 can receive communications from given hosts 114 and 116. Requests may include transaction layer packets (TLP) 602 and configuration access requests.

[0067] Control circuitry 102 can determine the requester identifier and completer identifier in the TLP and use these identifiers to bridge the TLP, configuration access request, from a given host to the corresponding VF. This bridging can be performed via a third internal partition 104. The requester identifier can be a combination of the requesting host's bus number, device number, and function number. The completer identifier can be a combination of the completer host's bus number, device number, and function number.

[0068] Figure 7 This is another illustration of the operation of control circuit 102 processing communication from a given host according to an example of this disclosure.

[0069] Control circuit 102 can receive communications from given hosts 114, 116. Requests may include TLP 702, memory access requests, messages, or completion requests.

[0070] Control circuitry 102 can determine the memory address in the TLP (such as a specific location in the address space of a PCIe device), the requester identifier, and the completer identifier, and use them to bridge the TLP, memory access, message, or completion request from a given host to the appropriate VF. This bridging can be performed via a third internal partition 104.

[0071] Figure 8 This is yet another illustration of the operation of control circuit 102, according to an example of this disclosure, in processing communications from corresponding VFs 122, 124, 126.

[0072] Control circuit 102 can receive communication from given VFs 122, 124, and 126. Requests may include TLP 802, memory access requests, or messages.

[0073] Control circuitry 102 can determine the requester identifier in the TLP and use that identifier to bridge the TLP, memory access request, or message from a given VF to the appropriate host. This bridging can be performed via third internal partition 104.

[0074] Figure 9 This is yet another illustration of the operation of the control circuit 102 according to the example of this disclosure, which processes communications from the respective VFs 122, 124, 126.

[0075] Control circuit 102 can receive communication from given VFs 122, 124, and 126. Requests may include TLP 902 and request completion.

[0076] Control circuitry 102 can determine the requester identifier and the completer identifier in the TLP, and use these identifiers to bridge the TLP and the completion request from a given VF to the appropriate host. This bridging can be performed via a third internal partition 104.

[0077] Figure 10 This is a more detailed illustration of bridge 121 according to the example of this disclosure. It illustrates requests for configured access bridging from an LVF (and therefore associated hosts 114, 116) to a corresponding VF, requests for unconfigured access bridging from an LVF to a corresponding VF, and requests for VF-to-host bridging.

[0078] Requests from host 114 via LVF1 118 can be routed based on whether the request is for configuration access 1004, non-configuration access 1002, memory access, message, or completion. Non-configuration access 1002 can be routed by NTB circuit 404 to SR-IOV shared port 130 to reach the SR-IOV device. Configuration access 1004 can be handled by hypervisor firmware (HV FW) 1008 running on embedded CPU 402. Configuration access 1004 can involve creating a configuration access TLP and submitting it to SR-IOV shared port 130 to reach the SR-IOV device, or bridging the configuration access TLP to SR-IOV shared port 130 via NTB circuit 404 to reach the SR-IOV device. Such requests can be handled via CPU interface 1006 leading to embedded CPU 402. Thus, configuration of device 100 can be performed appropriately. Similarly, host 116 can access configured access 1004 or unconfigured access 1002 via LVF2 120.

[0079] The hypervisor firmware 1008 may include an ePF driver, an eSR-PCIM driver, and a vRC bus driver.

[0080] Control circuitry 102 can perform the step of first configuring a specific port of device 100 for SR-IOV shared functionality. SR-IOV shared port 130 can be connected to internal partition 104, which is not directly connected to or visible to either host 114 or 116. Next, control circuitry 102 can enumerate the attached SR-IOV device 106 using the virtual root complex bus driver (vRC-bus driver) of HVFW 1008. Control circuitry 102 can perform configuration and management of the SR-IOV-enabled device 106. This can be performed via the embedded single-root PCI manager (eSR-PCIM) of HV FW 1008. Control circuitry 102 can also manage PFs such as PF 128 via the embedded PF (ePF) driver of HV FW 1008. Control circuitry 102 can then expose the VF as an LVF to any host, such as hosts 114 and 116, that wants to share the given VF. Control circuitry 102 can then handle configuration access to the LVF from hosts 114 and 116 by a combination of forwarding requests to the associated PF, forwarding requests to the appropriate VF, and merging PF and VF data and emulation via, for example, the operation of HV FW 1008. Control circuitry 102 can set and manage NTB rules to allow NTB circuitry 404 to directly bridge requests such as DMA, memory space access, and interrupt handling between the host and VF (in SR-IOV-enabled functionality), while reducing intervention from the CPU of device 100.

[0081] like Figure 4 As shown, given hosts 114 and 116 can communicate with given PCIe devices 148 and 150 or with SR-IOV device 106. Communication between given hosts 114 and 116 and PCIe devices 148 and 150 attached to the same partitions 132 and 134 as the respective hosts 114 and 116 can be referred to as pass-through operation mode. Communication between given hosts 114 and 116 and shared SR-IOV device 106 in different partitions, such as partition 104, which is not connected to any external PCIe host and is managed by control circuitry 102, can be referred to as SR-IOV shared mode.

[0082] The pass-through operation mode can be implemented according to the PCIe specification.

[0083] In SR-IOV shared mode, the enumeration of downstream devices such as SR-IOV device 106, resource allocation for memory space, interrupts, configuration, and power management can be managed and executed by control circuitry 102, specifically by HV FW 1008. The enumeration and configuration of PCIe devices in SR-IOV shared mode can be performed via internal partitioning according to standard PCIe functions. A PCIe endpoint with at least one SR-IOV-enabled function (such as SR-IOV device 106) can be supported as part of standard PCIe functionality at the corresponding downstream port (such as port 130). When control circuitry 102 detects an unsupported PCIe device (non-endpoint, such as a switch or endpoint without any SR-IOV-enabled functionality) at the DSP, port operation can switch from SR-IOV shared mode to pass-through mode until these devices are disconnected, at which point the port switches back to SR-IOV shared mode.

[0084] In SR-IOV shared mode, bridge 121 can perform the connection between LVF 118 and 120 between ports 138 and 140 and port 130 under the control of control circuit 102.

[0085] The vRC bus driver in HV FW 1008 can be a virtual root complex bus driver and can be responsible for enumerating PCIe devices attached to SR-IOV shared port 130, allocating resources for memory space, and handling interrupts, configuration, and power management.

[0086] eSR-PCIM can be an embedded single-core PCI resource manager, and can be responsible for configuring and managing the SR-IOV expansion capabilities and the resources required by the SR-IOV device.

[0087] ePF drivers can be embedded physical function drivers and can be responsible for PF management, VF enumeration, and the management of all functions shared between PF and VF.

[0088] LVFs 118 and 120 can be presented as PCIe devices for hosts 114 and 116, and can be emulated by control circuitry 102. LVFs 118 and 120 can reflect the characteristics of underlying VFs 122, 124, and 126. Most transactions destined for LVFs 118 and 120 are forwarded and bridged to SR-IOV device 106, targeting specific VFs 122, 124, and 126.

[0089] LVF-to-VF bridging can be performed by control circuitry 102. Specifically, configuration requests (such as configuration read requests and configuration write requests) from given hosts 114, 116 to given LVFs 118, 120 can be bridged to SR-IOV device 106 via HV FW 1008 and NTB circuitry 404. However, non-configuration access from given hosts 114, 116 to given LVFs 118, 120 can be bridged to SR-IOV device 106 via NTB circuitry 404, and can involve minimal involvement of HV FW 1008.

[0090] Communication between given VFs 122, 124, 126 and associated LVFs 118, 120 (such as memory read requests and memory write requests from SR-IOV device 106) can be bridged to LVFs 118, 120 via NTB circuit 404 and can involve minimal involvement of HV FW 1008.

[0091] In contrast, in standard PCIe devices, the associated PCIe bus driver, SR-PCIM, and PF driver are typically implemented at each host for all ports exposed to a given host.

[0092] In bridge 121, port 130 for SR-IOV 106 may not be directly exposed to any host 114, 116 outside device 100, and the vRC bus driver, eSR-PCIM, and ePF driver components are implemented in HV FW 1008 within device 100. The complexity of these components and their operation is abstracted within device 100, and only VFs 122, 124, 126 of SR-IOV device 106 are visible to hosts 114, 116 outside device 100 (and are then emulated only as LVFs).

[0093] HV FW 1008 can read the capabilities and memory space requirements of SR-IOV device 106, including the capabilities of VFs, and emulate the associated LVFs for each VF to be shared. Given LVFs 118, 120 can be attached to given hosts 114, 116 at boot time or runtime using PCIe hot-plug capabilities.

[0094] The ePF driver can write the "System Page Size" of a given PF 128 as 4KB because SR-IOV devices support 4KB pages in the "Supported Page Sizes". The ePF driver can calculate n base address registers (BARs) based on the BAR0...BARn register values ​​read from the PF 128 and the number of VFs (n), given as LVF BAR0...LVF BARn, and report them to the corresponding host. LVFs 118 and 120 may not report any I / O space requirements as part of the BAR, as SR-IOV-enabled devices may not require I / O space. Fields common to multiple or all VFs (such as error enable bits, Max_Payload_size, Max_read_request_size, etc.) can be virtualized by the HV FW 1008. The remainder of the standard enumeration procedure according to the SR-IOV specification can be followed to enable VFs from the PF.

[0095] LVF-to-VF configuration access bridging can also be handled by control circuitry 102. Type 0 configuration read and write requests from given hosts 114, 116 to the associated LVF can be routed to HV FW 1008 for further processing and disposal. HV FW 1008 can optionally read the entire 4 kilobytes of configuration data from the associated PF 128 and all associated VFs at boot time and store this configuration data for processing configuration requests, or otherwise read the configuration data as needed when a given host accesses the relevant registers of the LVF.

[0096] Depending on the address of the LVF register being read or written, control circuitry 102 may: forward a translated access request to the VF if all bits of the register are implemented in the VF; forward a translated access request to the PF if all bits of the register are implemented in the PF; forward a translated access request for the bits implemented in the VF and merge the remaining data from the PF when the bits of the register are shared in both the PF and VF; or ignore configuration writes for features that are not applicable to the VF or the SR-IOV device.

[0097] Whenever a given host writes configuration information (implemented as registers or bits of registers) shared by all VFs, control circuitry 102 can store the written data in a VF context-specific manner, so that the data can be processed and returned when the same host reads such configuration information in a subsequent request. This contrasts, for example, with actually writing configuration information to a VF when the configuration information is not shared with other VFs.

[0098] LVFs 118 and 120 can be implemented using command registers to store configuration information. When the memory space enable bit in the command register of a given LVF is set or cleared by a given host 114 or 116, the NTB rules used for memory access bridging can be updated.

[0099] When the error enable bit in the command register of a given LVF is set or cleared by the host, the NTB rule can be updated to allow or block error messages from a specific associated VF to a given corresponding host.

[0100] Link management and power management commands to a given LVF are primarily handled by HV FW 1008 because there is a single link upstream of SR-IOV device 106, and therefore individual VFs 122, 124, and 126 cannot be granted control of the link. The same applies to power management capabilities, administration, and processing.

[0101] For features supported by PF or VF but whose associated LVF will not provide or cannot provide access to, such capabilities are masked by changing the next capability pointer that is to be updated as part of the LVF's configuration data processing.

[0102] Configuration access to the TLP is routed via ID-based routing using the bus number, device number, and function number (BDF) fields of the TLP-based routing. When a given LVF configuration request needs to be forwarded to SR-IOV device 106, HV FW 1008 can translate the LVF completer ID (assigned in the domain of the given host) to the SR-IOV device 106's BDF (assigned by the vRC bus driver). Similarly, the requester ID will be translated across domains. Likewise, the completer ID and requester ID are updated before returning completion data in response to a configuration read command.

[0103] LVF to VF non-configuration access can also be bridged by control circuit 102.

[0104] For memory access TLP requests, such TLPs can be routed based on address routing. To route a memory access TLP addressing LVFs 118 and 120 from given hosts 114 and 116 to internal partition 104 to reach VFs 122, 124, and 126 in SR-IOV device 106, the NTB circuitry 404 can be programmed as follows: NTB rules can be established to translate memory addresses within the BAR range of an LVF in an incoming TLP to the BAR range of the associated VF based on the entry ports 110 and 112 of the memory access TLP. Since multiple hosts 114 and 116 may have already assigned the same BAR to their visible LVFs 118 and 120, the entry port is included in the translation along with the address.

[0105] The requester ID can be converted from the host region to the internal region.

[0106] The NTB rules for memory access bridging are enabled or disabled based on the "Memory Space Enable" bit in the command register of a given LVF. These rules are typically implemented in hardware to avoid overhead in TLP processing.

[0107] Completion TLPs can be routed based on ID routing. These can include data-free completion, data-containing completion, data-free lock-on memory read completion, and data-containing lock-on memory read completion. To route a completion TLP addressed from given hosts 114, 116 to LVFs 118, 120 to internal partition 104 to reach given VFs 122, 124, 126 in SR-IOV device 106, the NTB circuit 404 can be programmed as follows: Based on the entry ports 110, 112 of the completion TLP, the completion party ID can be translated from the domains of host partitions 132, 134 to the domain of internal partition 104.

[0108] The TLP message can be routed by the NTB circuit 404, as configured by the HV FW 1008. However, when multiple VFs 122, 124, and 126 are active, the power management TLP message, including the slot power limit message, can be processed by the HV FW 1008 for LVFs 118 and 120 without further bridging, because multiple VFs 122, 124, and 126 can share PF 128, and therefore power management by individual VFs 122, 124, and 126 may not be possible.

[0109] The vendor-defined message TLP can be transferred from host domain partitions 132 and 134 to internal partition 104.

[0110] Access from VF 122, 124, 126 to LVF 118, 120 and their upstream components can be handled by control circuit 102.

[0111] Memory TLP bridging can be performed by control circuitry 102. During bus master DMA transfers and MSI / MSI-X interrupts, memory requests typically originate from SR-IOV device 106. Although the memory TLP is designed to be routed based on address, control circuitry 102 can use the requester ID in the TLP to identify the VF 122, 124, 126 from which the request originates. To route the memory access TLP from a given VF 122, 124, 126 to internal partition 104 to reach the upstream of the corresponding LVF 118, 120, NTB circuitry 404 can be programmed as follows. If the requester ID matches the BDF of PF 128, the memory read or write request is handled by the ePF driver in HV FW 1008 and is not further forwarded or bridged. Otherwise, memory read or write requests are bridged to LVFs 118 and 120 corresponding to hosts 114 and 115 bridged to VFs 122, 124, and 126, and the requester ID field is updated to the requester ID of LVF 118 or 120. Note that addresses may not be translated because DMA transfers are typically set up by programming the device DMA controller registers, which are accessible by the host (and host driver) in the BAR space. The context of these DMA registers and the data entering them may not be known to device 100. Therefore, although SR-IOV device 106 operates in the vRC domain, DMA transfers initiated by SR-IOV device 106 (such as memory read requests) reference the memory addresses of the actual host partitions 132 and 134 to which it is connected. Therefore, addresses may not be translated.

[0112] Control circuitry 102 can bridge a completion TLP originating from SR-IOV device 106 based on the completion party ID field in the completion header. To route a completion TLP from a given VF to internal partition 104 to reach upstream LVFs 118 and 120, NTB circuitry 404 can be programmed as follows: If the completion party ID matches the BDF of PF 128, the completion can be processed by the ePF driver in HVFW 1008. Otherwise, the completion TLP can be bridged to LVFs 118 and 120 corresponding to hosts 114 and 116 bridged with VFs 122, 124, and 126, where the completion party ID field is updated to the completion party ID of LVFs 118 and 120, and the requester party ID field is updated to the requester party ID of a domain in host internal partition 104 (as stored by LVFs 118 and 120).

[0113] Message TLP bridging can be performed by NTB circuit 404, as configured by HV FW 1008.

[0114] Figure 11 This is an illustration of an exemplary method 1100 according to an example of this disclosure. Method 1100 can begin at any suitable point. Method 1100 may include... Figure 11 The steps of method 1100 may be more or fewer than those shown. The steps of method 1100 can be executed in any suitable order, and the steps of method 1100 may optionally be omitted, repeated, executed in parallel, or executed recursively. Method 1100 may be... Figures 1 to 10 This can be implemented by any suitable part, such as device 100, and specifically by control circuit 102, HV FW 1008 and NTB circuit 404.

[0115] At 1105, SR-IOV devices can be enumerated in the PCIe partitions within the device. The device may include: multiple upstream PCIe ports, a given upstream PCIe port for connecting to a given host among multiple hosts; and a first downstream PCIe port for connecting to the SR-IOV device, wherein the SR-IOV device will be shared by multiple hosts, and the device will provide access to multiple VFs of the SR-IOV device based on the PF of the SR-IOV device.

[0116] At 1110, the first VF among a plurality of VFs can be emulated as the first individual PCIe device facing the first host among a plurality of hosts, wherein the first host is connected to the device via the first upstream PCIe port among the upstream PCIe ports.

[0117] At 1115, the second VF among multiple VFs can be emulated as a second separate PCIe device facing a second host among multiple hosts, the second host being connected to the device via a second upstream PCIe port among upstream PCIe ports.

[0118] At 1120, inter-domain bridging of PCIe transactions between multiple hosts and individual VFs can be implemented.

[0119] Figure 12 This is an illustration of an example article 1200 according to the present disclosure and an example method 1201 performed by such article 1200. Article 1200 may include a non-transitory machine-readable medium 1204, which may include instructions 1202 that, when loaded and executed by a processor 1203, may perform any suitable operation of the present disclosure. Such operations may include method 1100, Figures 1 to 10 The operation or method 1201 of a suitable part (such as device 100, and specifically control circuitry 102, HV FW 1008, and NTB circuitry 404). Processor 1203 may be implemented by any suitable circuitry or processor such as embedded processor 402. Medium 1204 may be implemented in any suitable manner, such as via memory.

[0120] Method 1201 may be a more detailed illustration of method 1100. Method 1201 can begin at any suitable point. Method 1201 may include more than Figure 12 The steps shown may be more or fewer. The steps of method 1200 can be executed in any suitable order, and the steps of method 1201 can optionally be omitted, repeated, executed in parallel, or executed recursively. Method 1201 can be... Figures 1 to 10 and Figure 12 It can be implemented by any suitable part (such as device 100, and specifically control circuit 102, HV FW 1008 and NTB circuit 404) or by processor 1203 executing instruction 1202.

[0121] At 1205, SR-IOV devices can be enumerated in the PCIe partitions within the device. The device may include: multiple upstream PCIe ports, a given upstream PCIe port for connecting to a given host among multiple hosts; and a first downstream PCIe port for connecting to the SR-IOV device, wherein the SR-IOV device will be shared by multiple hosts, and the device will provide access to multiple VFs of the SR-IOV device based on the advertised capabilities within the PF of the SR-IOV device.

[0122] At 1210, the first VF among a plurality of VFs can be emulated as the first individual PCIe device facing the first host among a plurality of hosts, wherein the first host is connected to the device via the first upstream PCIe port among the upstream PCIe ports.

[0123] At 1215, the second VF among multiple VFs can be emulated as a second separate PCIe device facing either the first host or the second host among multiple hosts, with the second host connected to the device via a second upstream PCIe port in the upstream PCIe ports.

[0124] At 1220, the inter-domain PCIe bridging rule table can be programmed to provide non-transparent access between SR-IOV devices and multiple hosts via VF.

[0125] At 1225, any configuration access request from a given host for a given VF can be bridged via the third partition.

[0126] At 1230, configuration data from a configuration access request from a given host can be simulated.

[0127] At position 1235, inter-domain bridging of PCIe transactions between multiple hosts and VFs of SR-IOV devices can be managed as needed.

[0128] Figure 13 The illustration is of method 1300 according to the example of this disclosure, which may be a more detailed illustration of method 1201.

[0129] At 1305, SR-IOV devices can be enumerated in PCIe partitions within the device. The device may include: multiple upstream PCIe ports, a given upstream PCIe port for connecting to a given host among multiple hosts; and a first downstream PCIe port for connecting to the SR-IOV device, wherein the SR-IOV device will be shared by multiple hosts, and the device will provide access to multiple VFs of the SR-IOV device based on the advertised capabilities within the PF of the SR-IOV device.

[0130] Inter-domain PCIe bridges can be programmed and managed to route PCIe signals between a given host and a corresponding given VF via internal partitions of the device (referred to as third partitions).

[0131] At 1310, the first VF among a plurality of VFs can be emulated as the first individual PCIe device facing the first host among a plurality of hosts, wherein the first host is connected to the device via the first upstream PCIe port among the upstream PCIe ports.

[0132] At 1315, the second VF among multiple VFs can be emulated as a second separate PCIe device facing a second host among multiple hosts, the second host being connected to the device via a second upstream PCIe port in the upstream PCIe ports. The VF can be emulated as a host-facing device to simultaneously share the SR-IOV device with the host.

[0133] At 1320, each host in the host is instructed that the corresponding host has exclusive control over the corresponding VF in the underlying device.

[0134] At point 1325, the inter-domain PCIe bridging rule table can be programmed to provide non-transparent access between the SR-IOV device and multiple hosts via VF. This can include non-transparent access to the SR-IOV device by multiple hosts via VF.

[0135] At 1330, PF can be configured and managed, and access to VF can be provided through internal partitions.

[0136] At 1335, after determining the configuration register address, any configuration access request from a given host for a given VF can be bridged via the third partition.

[0137] At 1340, after determining the configuration register address, configuration data from a configuration access request from a given host can be simulated.

[0138] At 1345, inter-domain bridging of PCIe transactions between multiple hosts and the VF of the SR-IOV device can be managed as needed. This can include accessing the SR-IOV device from a third partition of the device, where the first, second, and third partitions of the device are separate partitions, provided that a first host wants to access a first upstream PCIe port in the first partition of the device and a second host wants to access a second upstream PCIe port in the second partition of the device.

[0139] At 1350, such transactions may include configuration transactions processed within the third partition. Other transactions may be offloaded to hardware. Processing such transactions may include: determining whether a request originating from an SR-IOV device originates from a PF or from one of a plurality of VFs; bridging the request to the appropriate host based on the determination that the request originating from the SR-IOV device originates from one of the plurality of VFs; and otherwise, processing the request within the third partition based on the determination that the request originating from the SR-IOV device originates from a PF. Processing such transactions may include determining a first requester identifier and a first completer identifier for a first transaction layer group to bridge configuration access requests from a given host among a plurality of hosts to the appropriate VF in the third partition.

[0140] Figure 14 This is an illustration of the operation of control circuitry 102, according to an example of this disclosure, processing configuration read requests from given hosts 114, 116 to LVFs 118, 120. A configuration read request may include a transaction for reading configuration registers for functions within a device.

[0141] One of the hosts 114 and 116 can issue a configuration request that is a read from the associated one of the LVFs 118 and 120.

[0142] Receiving LVF 118 and 120 can provide a configuration read request to the control circuit 102.

[0143] Control circuitry 102 can be configured to determine where a configuration read request will need to be input from. For example, control circuitry 102 can determine whether the requested configuration data needs to be input from one or more of the following: local storage device 1402; PF 128; or a corresponding one of VF 122, 124, 126. This determination can be based on the configuration register address of the configuration read request.

[0144] If the configuration register address indicates that a configuration read request will require input from the local storage device, the control circuit 102 can retrieve configuration simulation data stored in the local storage device 1402 for processing in the data processor 1404.

[0145] If the configuration register address indicates that a configuration read request will require input from PF 128, the control circuit 102 can retrieve PF configuration data from PF 128 for processing in the data processor 104.

[0146] If the configuration register address indicates that a configuration read request will require input from one of VF 122, 124, 126, then the control circuit 102 can retrieve VF configuration data from the one identified by VF 122, 124, 126.

[0147] The input can be processed in data processor 1404. The processed configuration data can be returned to control circuitry 102. The processed configuration data can be used for multi-master emulation of SR-IOV device 106, enabling a given host 114, 116 to access the device via the corresponding LVF 118, 120. NTB rules in NTB circuitry 404 can be updated. Data processor 104 can apply arithmetic and logical operations to data retrieved from one or more of local storage devices 1402, PF 128, and VF 122, 124, 126 to complete the processed configuration data.

[0148] Figure 15 This is an illustration of the operation of control circuitry 102, according to an example of this disclosure, processing configuration write requests from given hosts 114, 116 to LVFs 118, 120. A configuration write request may include a transaction for writing configuration registers for functions within a device.

[0149] One of the hosts 114 and 116 can issue a configuration request that is a write to the associated one of the LVFs 118 and 120.

[0150] Receiving LVF 118 and 120 can provide configuration write requests to control circuit 102.

[0151] Control circuitry 102 can be configured to determine where a configuration write request will need to be input from and where the updated configuration data will be written. For example, control circuitry 102 can determine whether the requested configuration data write requires input from one or more of the following: local storage device 1402; PF 128; or a corresponding one of VFs 122, 124, 126. This determination can be based on the configuration register address of the configuration read request.

[0152] If the configuration register address indicates that a configuration write request will require input from the local storage device, the control circuit 102 can retrieve configuration simulation data stored in the local storage device 1402 for processing in the data processor 1404.

[0153] If the configuration register address indicates that a configuration write request will require input from PF 128, the control circuit 102 can retrieve PF configuration data from PF 128 for processing in the data processor 104.

[0154] If the configuration register address indicates that a configuration write request will require input from one of VF 122, 124, 126, then the control circuit 102 can retrieve VF configuration data from the one identified by VF 122, 124, 126.

[0155] The input can be processed in data processor 1404. Data processor 1404 can apply arithmetic and logical operations to data retrieved from one or more of local storage devices 1402, PF 128, and VFs 122, 124, 128 to complete the processed configuration data. If the configuration register address addresses a valid register supported by the corresponding LVF, the processed configuration data can be written to one or more of local storage devices 1402, PF 128, and VFs 122, 124, 126. The processed configuration data can be used for multi-master emulation of SR-IOV device 106, allowing a given host 114, 116 to have its access bridged via the corresponding LVFs 118, 120. NTB rules in NTB circuit 404 can be updated. Control circuit 102 can return a completion status to the given host 114, 116 via the corresponding LVFs 118, 120.

[0156] For the purposes of this disclosure, computer-readable media may include any tool or set of tools that can retain data and / or instructions for a period of time. Computer-readable media may include, but is not limited to: storage media such as direct access storage devices (e.g., hard disk drives or floppy disks), sequential access storage devices (e.g., magnetic tape drives), optical discs, CD-ROMs, DVDs, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and / or flash memory; and non-transitory communication media and / or any combination thereof.

[0157] Examples of this disclosure may include an apparatus. The apparatus may include one or more upstream PCIe ports. A given upstream PCIe port can be connected to a given host among one or more hosts. The apparatus may include a first downstream PCIe port for connection to a single root I / O virtualization (SR-IOV) device. The SR-IOV device may be shared by one or more hosts. The apparatus may provide access to one or more virtual functions (VFs) associated with the physical function (PF) of the SR-IOV device. The apparatus may include control circuitry. The control circuitry may be configured to enumerate SR-IOV devices in PCIe partitions within the apparatus. The control circuitry may be configured to emulate a first VF among the VFs as a first separate PCIe device for a first host among the hosts. The first host can connect to the apparatus via a first upstream PCIe port among the upstream PCIe ports. The control circuitry may be configured to emulate a second VF among the VFs as a second separate PCIe device for a second host among the hosts. The second host can connect to the apparatus via a second upstream PCIe port among the upstream PCIe ports. The control circuitry can be configured to implement inter-domain bridging of PCIe transactions between the host and the VF of the SR-IOV device. The control circuitry can be implemented in any suitable manner, such as using analog circuitry, digital circuitry, field-programmable gate arrays, application-specific integrated circuits (ASICs), programmable logic devices, microcontrollers, system-on-a-chip (SoCs), instructions for processor execution, or any suitable combination thereof.

[0158] Combining any of the examples above, the control circuitry can be configured to configure and manage the PF and provide access to the VF associated with the PF of the SR-IOV device.

[0159] Combining any of the examples above, the control circuitry can be configured to emulate the first VF and the second VF as a first separate PCIe device and a second separate PCIe device facing the host, respectively, so as to simultaneously share the SR-IOV device with the host.

[0160] Combining any of the examples above, the control circuit can emulate the first VF and the second VF as a first separate PCIe device and a second separate PCIe device facing the host, respectively, to simultaneously share the SR-IOV device with the host and simultaneously indicate to a given host in the host that the given host has exclusive control over the given VF.

[0161] Combining any of the examples above, the control circuitry can be configured to provide the host with non-transparent access to the SR-IOV device via VF.

[0162] Combining any of the examples above, a first host can be configured to access a first upstream PCIe port in a first partition of the device. A second host can be configured to access a second upstream PCIe port in a second partition of the device. Control circuitry can be configured to access SR-IOV devices from a third partition of the device. The third partition can be an internal PCIe partition. The first, second, and third partitions of the device can be separate partitions.

[0163] Combining any of the examples above, the first host can be configured to access the second downstream PCIe port in the first partition of the access device to access downstream devices.

[0164] Combining any of the examples above, the control circuitry can be configured to implement an inter-domain PCIe bridge to route PCIe signals between a given host and a corresponding given VF via a third partition.

[0165] In conjunction with any of the examples above, the control circuitry can be configured to bridge or emulate configuration access requests for a given VF from a given host via a third partition.

[0166] Combining any of the examples above, the control circuitry can be configured to determine whether a request originating from the SR-IOV device originates from a PF or from a VF within a VF. Based on the determination that the request originates from the SR-IOV device from a VF, the control circuitry can bridge the request to the appropriate host. Otherwise, based on the determination that the request originates from the SR-IOV device from the PF, the control circuitry can process the request within the third partition.

[0167] In conjunction with any of the examples above, the control circuitry can be configured to perform one or more of the following: determine a first requester identifier and a first completer identifier of a first transaction layer packet (TLP) to bridge a configuration access request from a given host in a host to a corresponding VF in a third partition; determine a memory address, a second requester identifier, and a second completer identifier of a second TLP to bridge a first memory access, a first message, and a first completion from a given host in a host to a corresponding VF in a third partition; determine a third requester identifier of a third TLP to bridge a second memory access and a second message from a corresponding VF in a third partition to a given host in a host; and determine a fourth requester identifier and a third completer identifier of a fourth TLP to bridge a second completion from a corresponding VF in a third partition to a given host in a host.

[0168] Combining any of the examples above, the control circuitry can process a configuration read request from a given host for a corresponding LVF in the following manner: based on the configuration register address in the configuration read request, determine whether the requested configuration data is based on input from one or more of the local storage device, or the PF, or the corresponding VF; perform one or more of the following: based on determining that the requested configuration data requires input from the local storage device, retrieve stored configuration simulation data from the local storage device for processing in the data processor; based on determining that the requested configuration data requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor; and based on determining that the requested configuration data requires input from the corresponding VF, retrieve VF configuration data from the corresponding VF for processing in the data processor; process one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor, and bridge through the corresponding LVF to return the processed configuration data to the given host for multi-host emulation of the SR-IOV device; and update the NTB rules.

[0169] Combining any of the examples above, the control circuitry can process a configuration write request from a given host for a corresponding LVF in the following manner: based on the configuration register address in the configuration write request, and based on input from one or more of the local storage device, or the PF, or the corresponding VF, determine whether configuration data should be written; perform one or more of the following: based on the determination that the configuration data to be written requires input from the local storage device, retrieve stored configuration simulation data from the local storage device for processing in the data processor; based on the determination that the configuration data to be written requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor; and ... The configuration data to be written requires input from the corresponding VF; VF configuration data is retrieved from the corresponding VF for processing in the data processor; and one or more of the configuration simulation data, PF configuration data, and VF configuration data are processed in the data processor to produce processed configuration data; if the configuration register address addresses a valid register supported by the corresponding LVF, the processed configuration data is written to one or more of the local storage device, PF, and corresponding VF; and the processed configuration data is used for multi-host simulation of the SR-IOV device to a given host via the corresponding LVF; the completion status is returned to the given host via the corresponding LVF; and the NTB rules are updated.

[0170] Examples of this disclosure may include an artifact. The artifact may include instructions. When loaded and executed by a processor, these instructions may cause the processor to: enumerate a single root I / O virtualization (SR-IOV) device in a PCIe partition within the device. The device may include: one or more upstream PCIe ports, a given upstream PCIe port for connection to a given host among one or more hosts; and a first downstream PCIe port for connection to the SR-IOV device. The SR-IOV device may be shared by hosts. The device may provide access to one or more virtual functions (VFs) associated with the physical function (PF) of the SR-IOV device. These instructions enable the processor to: emulate a first VF in a VF as a first separate PCIe device for a first host in a host, the first host being connected to the device via a first upstream PCIe port in the upstream PCIe ports; emulate a second VF in a VF as a second separate PCIe device for a second host in a host, the second host being connected to the device via a second upstream PCIe port in the upstream PCIe ports; program an inter-domain PCIe bridging rule table to provide non-transparent access between the SR-IOV device and the host via the VF; bridge configuration access requests from a given host to the corresponding given VF via a third partition; emulate configuration data for configuration access requests from a given host; and manage inter-domain bridging of PCIe transactions between the host and the VF of the SR-IOV device. These instructions can be implemented in any suitable manner, such as using analog circuitry, digital circuitry, field-programmable gate arrays, application-specific integrated circuits, programmable logic devices, microcontrollers, system-on-a-chip, instructions for processor execution, or any suitable combination thereof.

[0171] Combining any of the examples above, these instructions enable the processor to configure and manage the PF, and provide access to the VF associated with the PF of the SR-IOV device.

[0172] Combining any of the examples above, these instructions can cause the processor to emulate the first VF and the second VF as a first separate PCIe device and a second separate PCIe device facing the host, respectively, so as to simultaneously share the SR-IOV device with the host.

[0173] Combining any of the examples above, these instructions can cause the processor to emulate the first VF and the second VF as a first separate PCIe device and a second separate PCIe device facing the host, respectively, so as to simultaneously share the SR-IOV device with the host and indicate to a given host in the host that the given host has exclusive control over the given VF.

[0174] Combining any of the examples above, these instructions enable the processor to program the inter-domain PCIe bridging rule table to provide the host with non-transparent access to SR-IOV devices via VF.

[0175] Combining any of the examples above, a first host can be configured to access a first upstream PCIe port in the first partition of the device. A second host can be configured to access a second upstream PCIe port in the second partition of the device. These instructions can cause the processor to access the SR-IOV device from a third partition of the device. The first, second, and third partitions of the device can be separate partitions.

[0176] Combining any of the examples above, these instructions enable the processor to program and manage inter-domain PCIe bridges to route PCIe signals between a given host and the corresponding given VF via a third partition.

[0177] In conjunction with any of the examples above, these instructions can cause the processor to perform one or more of the following: (A) determine whether a request originating from an SR-IOV device originates from a PF or from a VF within a VF; based on the determination that the request originating from an SR-IOV device originates from a VF within a VF, bridge the request to the appropriate host; and otherwise, based on the determination that the request originating from an SR-IOV device originates from a PF, process the request within a third partition; or (B) determine the first requester identifier and the first completer identifier of a first transaction layer packet (TLP) to bridge a configuration access request from a given host in a host to the appropriate VF in a third partition;

[0178] Combining any of the examples above, these instructions enable the processor to process a configuration read request from a given host for a corresponding LVF in the following ways: determining, based on the configuration register address in the configuration read request, whether the requested configuration data is based on input from one or more of the local storage device, the PF, or the corresponding VF; performing one or more of the following: retrieving stored configuration simulation data from the local storage device for processing in the data processor based on the determination that the requested configuration data requires input from the local storage device; retrieving PF configuration data from the PF for processing in the data processor based on the determination that the requested configuration data requires input from the PF; and retrieving VF configuration data from the corresponding VF for processing in the data processor based on the determination that the requested configuration data requires input from the corresponding VF; processing one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor, bridging through the corresponding LVF, and returning the processed configuration data to the given host for multi-host emulation of the SR-IOV device; and updating the NTB rules.

[0179] Combining any of the examples above, these instructions enable the processor to process a configuration write request from a given host for a corresponding LVF in the following ways: Based on the configuration register address in the configuration write request, determine whether the configuration data to be written is based on input from one or more of the local storage device, the PF, or the corresponding VF; perform one or more of the following: based on the determination that the configuration data to be written requires input from the local storage device, retrieve stored configuration simulation data from the local storage device for processing in the data processor; based on the determination that the configuration data to be written requires input from the PF, retrieve PF configuration data from the PF for processing in the data processor. The process includes: processing configuration data; retrieving VF configuration data from the corresponding VF for processing in the data processor based on the determination that the configuration data to be written requires input from the corresponding VF; processing one or more of the configuration simulation data, PF configuration data, and VF configuration data in the data processor; writing the processed configuration data to one or more of the local storage device, PF, and corresponding VF if the configuration register address addresses a valid register supported by the corresponding LVF; bridging the processed configuration data for multi-host simulation of the SR-IOV device to a given host via the corresponding LVF; returning a completion status to the given host via the corresponding LVF; and updating the NTB rules.

[0180] Examples of this disclosure may include a method. This method may include the operation of any of the examples described above. This method may be performed by any suitable element such as a PCIe switch, control circuitry, processor, or device. The method may include, at a PCIe switch: enumerating a single root I / O virtualization (SR-IOV) device in a PCIe partition within the device. The device may include: one or more upstream PCIe ports, a given upstream PCIe port for connection to a given host among one or more hosts; and a first downstream PCIe port for connection to an SR-IOV device. The SR-IOV device may be shared by hosts. The device may provide access to one or more virtual functions (VFs) associated with the SR-IOV device's physical function (PF). The method may include emulating a first VF in a VF as a first separate PCIe device for a first host among hosts. The first host may connect to the device via a first upstream PCIe port among the upstream PCIe ports. The method may include emulating a second VF in a VF as a second separate PCIe device for a second host among hosts. A second host can connect to the device via a second upstream PCIe port in the upstream PCIe ports. This method may include inter-domain bridging of PCIe transactions between the host and the individual PCIe device.

[0181] Although this disclosure has been described in detail, it should be understood that various changes, substitutions and alterations may be made thereto without departing from the spirit and scope of this disclosure as defined by the appended claims.

Claims

1. An apparatus, the apparatus comprising: Multiple upstream PCIe ports, a given upstream PCIe port is used to connect to a given host among multiple hosts; A first downstream PCIe port is used to connect to a single root I / O virtualization (SR-IOV) device shared by the plurality of hosts, and the device is used to provide access to a plurality of virtual functions (VFs) associated with the physical function (PF) of the SR-IOV device. and Control circuit, the control circuit being configured to: The SR-IOV devices are enumerated in the PCIe partition within the device. The first VF among the plurality of VFs is emulated as a first individual PCIe device facing a first host among the plurality of hosts, and the first host is connected to the device through a first upstream PCIe port among the upstream PCIe ports; The second VF among the plurality of VFs is emulated as a second separate PCIe device facing a second host among the plurality of hosts, the second host being connected to the device through a second upstream PCIe port among the upstream PCIe ports; as well as Implement inter-domain bridging of PCIe transactions between the multiple hosts and the multiple VFs of the SR-IOV device.

2. The apparatus of claim 1, wherein the control circuitry is configured to: configure and manage the PF, and provide access to the VF associated with the PF of the SR-IOV device.

3. The apparatus according to any one of claims 1 to 2, wherein the control circuitry is configured to emulate the first VF and the second VF as the first separate PCIe device and the second separate PCIe device facing the host, respectively, so as to simultaneously share the SR-IOV device with the host.

4. The apparatus according to any one of claims 1 to 3, wherein the control circuitry is configured to emulate the first VF and the second VF as the first separate PCIe device and the second separate PCIe device facing the host, respectively, so as to simultaneously share the SR-IOV device with the host and simultaneously instruct a given host in the host that the given host has exclusive control over the given VF.

5. The apparatus according to any one of claims 1 to 4, wherein the control circuitry is configured to provide non-transparent access to the SR-IOV device to the plurality of hosts via the VF.

6. The apparatus according to any one of claims 1 to 5, wherein: The first host is configured to access the first upstream PCIe port in the first partition of the device; The second host is configured to access the second upstream PCIe port in the second partition of the device; and The control circuitry is configured to access the SR-IOV device from a third partition of the device, the third partition being a PCIe internal partition, wherein the first partition, the second partition, and the third partition of the device are separate partitions.

7. The apparatus of claim 6, wherein the first host is configured to access a second downstream PCIe port in the first partition of the apparatus to access a downstream device.

8. The apparatus of any one of claims 6 to 7, wherein the control circuitry is configured to implement an inter-domain PCIe bridge to route PCIe signals between a given host and a corresponding given VF via the third partition.

9. The apparatus according to any one of claims 6 to 8, wherein the control circuitry is configured to bridge or emulate configuration access requests for a given VF from a given host via the third partition.

10. The apparatus according to any one of claims 6 to 9, wherein the control circuit is configured to: Determine whether the request originating from the SR-IOV device comes from the PF or from one of the multiple VFs; Based on the determination that the request originating from the SR-IOV device comes from one of the plurality of VFs, the request is bridged to the corresponding host; as well as Otherwise, based on the determination that the request originating from the SR-IOV device comes from the PF, the request is processed within the third partition.

11. The apparatus according to any one of claims 6 to 10, wherein the control circuit is configured to perform one or more of the following: Determine the first requester identifier and the first completer identifier of the first transaction layer packet (TLP) to bridge the configuration access request from the given host among the plurality of hosts to the corresponding VF in the third partition; The memory address of the second TLP, the second requester identifier, and the second completer identifier are determined to bridge the first memory access, the first message, and the first completion from the given host among the plurality of hosts to the corresponding VF in the third partition; A third requester identifier is determined for the third TLP to bridge the second memory access and the second message from the corresponding VF to the given host among the plurality of hosts in the third partition; as well as Determine the fourth requester identifier and the third completer identifier of the fourth TLP to bridge the second completion from the corresponding VF to the given host among the plurality of hosts in the third partition.

12. The apparatus according to any one of claims 6 to 11, wherein the control circuitry is configured to process a configuration read request for a corresponding LVF from a given host in such a manner as follows: Based on the configuration register address in the configuration read request, determine whether the requested configuration data is based on input from one or more of the local storage device, the PF, or the corresponding VF. Perform one or more of the following: Based on the determination that the requested configuration data requires input from the local storage device, the stored configuration simulation data is retrieved from the local storage device for processing in the data processor. Based on the determination that the configuration data requested requires input from the PF, PF configuration data is retrieved from the PF for processing in the data processor; as well as Based on the determination that the configuration data requested requires input from the corresponding VF, VF configuration data is retrieved from the corresponding VF for processing in the data processor; as well as The data processor processes one or more of the following: configuration simulation data, PF configuration data, and VF configuration data. It then bridges these data via the corresponding LVF and returns the processed configuration data to the given host for use in multi-host simulation of the SR-IOV device. Update NTB rules.

13. The apparatus according to any one of claims 6 to 12, wherein the control circuitry is configured to process a configuration write request for a corresponding LVF from a given host in such a manner as follows: Based on the configuration register address in the configuration write request, determine whether to write configuration data based on input from one or more of the local storage device, or PF, or the corresponding VF. Perform one or more of the following: Based on the determination that the configuration data to be written requires input from the local storage device, the stored configuration simulation data is retrieved from the local storage device for processing in the data processor. Based on the determination that the configuration data to be written requires input from the PF, retrieve the PF configuration data from the PF for processing in the data processor; as well as Based on the determination that the configuration data to be written requires input from the corresponding VF, VF configuration data is retrieved from the corresponding VF for processing in the data processor; as well as The data processor processes one or more of the configuration simulation data, PF configuration data, and VF configuration data to generate processed configuration data. If the configuration register address addresses a valid register supported by the corresponding LVF, the processed configuration data is written to one or more of the local storage device, the PF, and the corresponding VF; and The SR-IOV device is used for multi-host emulation of the given host by bridging through the corresponding LVF and the processed configuration data. The completion status is returned to the given host via the appropriate LVF; and Update NTB rules.

14. An article of manufacture comprising instructions that, when loaded and executed by a processor, cause the processor to perform operation of a control circuit according to any one of claims 1 to 13.

15. A method, the method comprising: The operation of the control circuit according to any one of claims 1 to 14 is performed at the PCIe switch.

Images