Virtual Hot-Plug System and Method for PCIe Devices

Abstract

A method for managing a pool of physical functions in a PCIe integrated endpoint includes receiving a configuration command indicating a topology for a PCIe-connected integrated endpoint (IE) and implementing the topology on the IE. The method further includes receiving a hot-plug command and adding or removing a virtual endpoint (vEP) to or from a virtual downstream port (vDSP) on the IE based at least in part on the hot-plug command.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the addition and removal of PCIe devices, and more particularly to virtual plug systems and methods for hot adding or hot removing PCIe devices.

Background Art

[0002] The Peripheral Component Interconnect Express (PCIe) architecture is designed to natively support both hot addition and hot removal ("hot plug") of cables, add-in cards, and modules. PCIe hot plug support provides a "toolbox" of mechanisms that enable different user / operator models to be supported using a self-collision-free infrastructure.

[0003] In modern data center applications, adaptability is particularly important on the input / output (IO) side. For example, when using a SmartNIC or SmartSSD connected to a data center via a link, the conventional method of connecting via a PCIe link is through a PCIe switch that uses a hot plug mechanism. In this method, the hot plug mechanism is incorporated into the downstream port of the PCIe switch to perform either hot addition or hot removal of a device having one or more physical functions. Note that the SmartNIC example above is a network interface card that offloads processing tasks normally performed by the system CPU. A smartNIC can potentially execute any combination of encryption / decryption, firewall, Transmission Control Protocol / Internet Protocol (TCP / IP), and Hypertext Transfer Protocol (HTTP) processing using its own on-board processor.

[0004] Traditionally, according to PCIe topology, the actual PCIe switch is physically placed between the data center host and the PCIe device, thus adding cost to both silicon and board design. Furthermore, the traditional approach also requires on-site actions, such as attention buttons, to physically add or remove devices connected to the host. In modern connectivity environments, this is not simply feasible, as a given data center (where the host is located) may be far from the manager site where the company's operations center is located. Therefore, a mechanism for remote configuration of PCIe-linked devices is required.

[0005] In addition, the increased capability of FPGAs to host more physical functions for more accelerators makes it desirable to have dynamic assignments for adding or removing single physical functions or groups of physical functions without having to pre-assign fixed relationships with hot-plugging mechanisms such as PCIe devices connected to PCIe switch downstream ports.

[0006] What is needed is a system and related methods to overcome the limitations of conventional PCIe hot-plug mechanisms. [Overview of the project]

[0007] Methods for virtually hot-adding or hot-removing PCIe physical functions ("PF") are described herein. PCIe integrated endpoint systems are also described herein. In one embodiment, a method for managing a pool of physical functions in a PCIe integrated endpoint includes receiving a configuration instruction specifying a topology on an integrated endpoint (IE) of a PCIe connection, implementing the topology on the IE, and receiving a hot-plug instruction. The method further includes, at least in part, adding or removing at least one virtual endpoint (vEP) to or from a virtual downstream port (vDSP) on the IE, based at least in part on a hot-plug instruction.

[0008] In another example, a PCIe integrated endpoint system includes an IE (Internet Explorer) with PCIe connectivity having a pool of physical functions, and a virtual hot-plug controller ("VHPC"). The VHPC is configured to receive configuration instructions specifying a topology on the IE and to implement the topology on the IE. The VHPC is further configured to receive hot-plug instructions and, in response to the hot-plug instructions, to add or remove a virtual endpoint (vEP) to or from a virtual downstream port (vDSP) on the IE, and to initiate at least one PCIe interrupt to inform the host about the topology.

[0009] The disclosed technology may also be illustrated by one or more of the following non-limiting embodiments. Other embodiments and further embodiments of the disclosed technology may be devised without departing from the teachings of this disclosure.

[0010] Example 1. A method for managing a pool of physical functions ("PF") in a PCIe integrated endpoint, comprising: receiving a configuration instruction specifying a topology for an integrated endpoint (IE) of a PCIe connection; implementing the topology on the IE; receiving a hot-plug instruction; and, at least in part, adding a virtual endpoint (vEP) to or removing a virtual downstream port (vDSP) on the IE from the vDSP.

[0011] Example 2. The method according to Example 1, further comprising adding one or more PFs to a vEP, removing one or more PFs from a vEP, or any combination of adding and removing PFs from a vEP, based on a hot-plug command.

[0012] Example 3. Configuration commands and hot-plug commands are received from a remote management agent, as described in Example 1.

[0013] Example 4. The method according to Example 1, further comprising initiating at least one PCIe interrupt to notify a connected host of a status change, and receiving a configuration request from the host in response to the PCIe interrupt.

[0014] Example 5. The method according to Example 4, further comprising initiating at least one PCIe interrupt by sending at least one of the following to a host: a device driver to bind and allocate resources to hot-add the endpoint and its assigned PF, or a device driver to unbind and release resources to hot-remove the endpoint and its assigned PF.

[0015] Example 6. The method according to Example 1, wherein the PCIe IE is provided to a PCIe integrated endpoint system, and the host is coupled to the PCIe integrated endpoint system via a PCIe link.

[0016] Example 7. The method according to Example 6, wherein both the host and PCIe integrated endpoint system are located within a data center.

[0017] Example 8. The PCIe integrated endpoint system is implemented on a field programmable gate array (FPGA) according to the method of Example 6.

[0018] Example 9. The topology is as described in Example 1, wherein the topology specifies a number of K virtual downstream ports (vDSPs) to be assigned to the IE, and each vDSP has an associated vEP.

[0019] Example 10. The method according to Example 9, wherein each vEP is coupled to a virtual downstream port.

[0020] Example 11. The method according to Example 9, wherein the topology implementation further includes grouping the PFs into each of the vEPs.

[0021] Example 12. The method according to Example 9, further comprising implementing a topology that divides the IE into K vDSPs.

[0022] Example 13. Integrated endpoints ("IE") with PCIe connectivity that have a pool of PFs, A PCIe integrated endpoint system comprising a virtual hot-plug controller ("VHPC"), VHPC receives configuration instructions from the remote manager that specify the topology for IE. Implement topology on IE, Received a hot-plug command from the remote manager. A PCIe-integrated endpoint system configured to add or remove at least one vEP from an IE's vDSP based on a hot-plug instruction.

[0023] Example 14. The system according to Example 13, further comprising a PCIe upstream port coupled to the VHPC and the host and configured to communicate at least one PCIe interrupt to the host.

[0024] Example 15. The system according to Example 13, wherein the VHPC receives configuration instructions from a remote management agent.

[0025] Example 16. The system according to Example 13, comprising an input interface configured to receive configuration instructions and hot plug instructions, and a PCIe topology mapper configured to generate a topology for the IE based at least in part on the configuration instructions.

[0026] Example 17. The system according to Example 13, wherein the topology includes a set of vDSPs and corresponding vEPs, each vEP having one or more PFs.

[0027] Example 18. The system according to Example 17, wherein the topology further includes a set of virtual downstream ports respectively corresponding to the set of vEPs and coupled to the set of vEPs.

[0028] Example 19. The system according to Example 13, further comprising a PCIe topology mapper configured to map configuration instructions to the topology, and a PCIe interrupt generator configured to generate at least one PCIe interrupt to notify the host of the PCIe connection with respect to the implemented topology or any vEP connectivity change within the topology as a result of a hot plug instruction.

[0029] Example 20. The system according to Example 19, further configured to initiate at least one PCIe interrupt to notify the host connected with respect to a revised set of connections from the vEP to the vDSP, and to receive a configuration request from the host in response to the PCIe interrupt. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Various features are described below with reference to the drawings. Note that the drawings may or may not be drawn to scale, and that elements of similar structure or function are represented by the same reference numerals throughout the drawings. Note that the drawings are intended solely to facilitate the description of the features. They are not intended to be an exhaustive description of the invention described in the "Claims" or to limit the scope of the invention described in the "Claims". In addition, the illustrated embodiments do not necessarily have all the embodiments or advantages shown. Embodiments or advantages described in relation to a particular embodiment are not necessarily limited to that embodiment and may be implemented in any other embodiment, even if not illustrated or explicitly described as such. [Figure 1] An exemplary system provided in a field-programmable gate array ("FPGA") according to one embodiment is shown. [Figure 2] An alternative exemplary FPGA implementation is shown illustrating a "firmware intervention" method according to one embodiment. [Figure 3] This is a flowchart illustrating a method for virtually hot-adding or hot-removing PCIe physical functions according to one embodiment. [Figure 4] This specification shows an exemplary FPGA architecture in which the exemplary systems described herein may be employed.

[0031] For ease of understanding, the same reference numerals are used to indicate identical elements common to the drawings, where possible. Elements of one embodiment are intended to be usefully incorporated into other embodiments. [Modes for carrying out the invention]

[0032] In one or more embodiments, PCIe hot-plug limitations are resolved by using a virtual hot-plug controller to manage a pool of physical functions at PCIe-integrated endpoints. In one or more embodiments, the virtual hot-plug controller provides remote configuration, dynamic allocation, and PCIe compliance without requiring an actual switch. As a result, no changes are required on the host side, and therefore there are no additional costs.

[0033] In one or more embodiments, a virtual PCIe hot-plug controller may virtually hot-add or hot-remove either a single PCIe physical function or a group of PCIe physical functions as a device via a native PCIe hot-plug mechanism without a physical switch. In one or more embodiments, FPGA soft logic and / or software intervention may be used to provide remote configuration, dynamic assignment, and PCIe compliance.

[0034] Therefore, in one or more embodiments where the extended hardware capabilities provide multiple virtual devices on a single physical card, and thus each does not have its own PCIe switch, a virtual hot-plug controller emulates hot-plug functionality for any of those virtual devices on the physical card. In some embodiments, the devices may be virtual cards fabricated by soft logic on a single FPGA card outside the IE. In one or more embodiments, each virtual device behaves as if it had its own data mover, processing engine, etc. Each virtual device is associated with a physical function provided by the IE so that it can be coupled to a device driver on the host and managed by the device driver.

[0035] One or more embodiments can be implemented on any FPGA card with sufficient soft logic to build multiple devices that may be desired by the user. Thus, one or more embodiments provide the ability to change the topology using the same card. Furthermore, these changes may be made remotely. This approach is faster for obvious reasons, as the person specifying the changes does not need to physically go to the data center to add or remove PCIe switches or cards.

[0036] Figure 1 shows an exemplary PCIe integrated endpoint system ("IES") 110 provided within an FPGA according to one embodiment. In one or more embodiments, the PCIe integrated endpoint system dynamically manages a pool of physical functions 152 and facilitates the hot removal or hot-add of any of the physical functions (PFs) across a set of virtual devices to which the pool of physical functions (PFs) is assigned. Referring to Figure 1, the exemplary PCIe IES 110 has three main blocks. These include a PCIe upstream port 120, a virtual hot-plug controller ("VHPC") 130 implemented as one or more PCIe downstream ports, and an integrated endpoint ("IE") 150.

[0037] In one or more examples, the PCIe upstream port 120 is used to connect to a host 105 via a PCIe link, such as PCIe link 101. The host 105 may be, for example, any CPU in a data center. Generally, the PCIe IES 110 can be understood as a single physical box or card that physically implements a pool of physical functions ("PFs"). However, all of those PFs can be allocated across multiple virtual PCIe devices as needed. In one or more embodiments, the VHPC 130 includes logic to provide information to the host 105 so that the host 105 treats any of the virtual devices on the PCIe IES 110 as if it were a full-fledged, standalone PCIe device with full hot-plug capabilities. This is true even when there is no physical switch, as in the conventional case.

[0038] In one or more embodiments, the information provided to the host 105 by the logic within the VHPC 130 is of the conventional type that is sent to the host when a device is either hot-added to or hot-removed from a PCIe link by inserting a card into a slot or, for example, removing a card from a physical slot. In one or more embodiments, as described below, when a PF is virtually added to or removed from a virtual downstream port, and the actual physical card is not inserted into or removed from a physical slot, the VHPC provides essential information that the host needs to see in order to emulate a physical hot-add or hot-remove event. In one or more embodiments, this information includes PCI / PCIe capabilities as defined in the specification. Key capabilities relevant to this description are the number of PFs and the number of address spaces required by the hot-add device. This flow is known as “device discovery” or “bus enumeration”.

[0039] In one or more embodiments, the virtual hotplug controller 130 may have three subblocks, as shown. These include, for example, a virtual hotplug handler 139, a PCIe topology mapper 132, and a PCIe interrupt generator 134. The virtual hotplug handler 139 receives hotplug configurations remotely from, for example, a remote management agent 103. The remote management agent 103 is located away from the data center where both the host 105 and the IES 110 are physically located. The remote management agent 103 may be coupled to a PCIe integrated endpoint system 110, for example, via an Ethernet connection.

[0040] Continuing to refer to Figure 1, in one or more embodiments, the hot-plug configuration instructions sent by the remote management agent 103 may be, for example, "add Physical Function #3-6" or, for example, "remove Physical Function #21-23". In one or more embodiments, the remote management agent 103 may or may not "see" the vDSP and vEP, in the latter case simply seeing a single integrated IES card and causing the PCIe topology mapper 132 to generate vEPs and distribute PFs among them. Which approach is taken depends on how much of the topology handling a given implementation needs to be processed on the card. Generally, assuming the topology mapper is closer and faster, it is often preferred to have the topology mapper generate and distribute various virtual entities and have the remote management agent provide higher-level instructions. However, for completeness, both approaches are possible in one or more embodiments.

[0041] Upon receiving a configuration instruction, the VHPC 130 configures the PCIe topology mapper 132 and triggers a PCIe interrupt via the PCIe interrupt generator 134 to notify the host 105 about the hot-plug action. Note that the topology (e.g., how many vDSPs to provide on the IE) is defined during a reset and is expected to remain fixed until the next reset. Based on the predefined topology and the hot-plug instruction received from the remote management agent 103, the topology mapper 132 adds / removes one or more vEPs and their respective PFs to one or more vDSPs.

[0042] In one or more embodiments, the PCIe topology mapper 132 informs the PCIe host 105 of the current topology by responding to configuration requests from the host 105. Note that the PCIe host 105 needs to know the topology so that it can allocate resources, such as address space. Also note that in the embodiments described herein, all configuration requests are sent by the host and are all PCIe-defined configuration requests.

[0043] In one or more embodiments, the IES 110 is initially seen as a switch (USP) by the host 105 by responding to a configuration read request from the host to the device / port type field of the PCI Express Capabilities Register. As a result, the host 105 continues to enumerate downstream ports and devices by sending more configuration read requests.

[0044] Continuing to refer to Figure 1, the PCIe topology mapper 132, in response to the received configuration instructions, distributes the various PFs within the pool of physical functions 152 among the various virtual endpoints ("vEPs") in the topology, with each vEP associated with a corresponding vDSP in the topology. Thus, a mapping of PFs 189 exists between each set of vEPs in a particular topology. Two such exemplary topologies are shown in Figure 1, and these exemplary topologies assume that there are a total of N PFs within the pool of physical functions 152. The first topology, topology 180, represents one extreme example of overall "centrality," having a single vEP that owns all N physical functions. The single vEP 180B is connected to a single virtual downstream port ("vDSP") 180A. At the other extreme, topology N 185 represents one PF for each vEP, thus N instances of vEP 185B, with each vEP connected to its corresponding vDSP 185A of N vDSPs. As shown, topology N 185 implements the maximum "function distribution" technique. In one or more embodiments, the topology can be any of the two extremes between these two.

[0045] After the PCIe topology mapper 132 performs its mapping and generates a topology in response to the received configuration instructions, the VHPC 130 may receive a hot-plug instruction, for example, from the remote management agent 103, which indicates adding or removing at least one vEP from a vDSP, and the VHPC 130 may respond by adding or removing at least one vEP from a vDSP. It is recalled here that the initial configuration instruction determines which topology should be implemented. Referring to Figure 1, for example, it could be a centralized topology 180, a distributed topology 185, or any two combinations between these two options. Thus, the configuration instruction determines how many vDSPs should be present in the topology.

[0046] Continuing to refer to Figure 1, when a hot-plug instruction is received, one or more vEPs are removed or added to a vDSP, but this does not change the total number of vDSPs in the topology, even if one or more vDSPs are currently left without vEPs as a result of the hot-plug instruction. This is because connections to vDSPs may be dangling, and vDSPs may remain in use for a while. The number of vDSPs remains constant until a new configuration instruction is received.

[0047] Following the implementation of the hot-plug instruction, the virtual hot-plug handler 139 triggers specific events. For example, it may cause the PCIe topology mapper 132 to update the status in the topology regarding which vEP is connected to which vDSP. Alternatively, it may cause the PCIe interrupt generator 134 to send a PCIe interrupt to the host, such as MSI or MSI-X, to bind a device driver and allocate resources for hot-adding the vEP and its assigned PF, or to unbind the device driver and release resources for hot-removing the vEP and its assigned PF, allowing resources to be dynamically allocated. It should also be noted that in one or more embodiments, any communication between the IE system and the host adheres strictly to the PCIe specification. Thus, the IE system "mimics" the behavior of a PCIe switch so that any conventional host can interoperate with the IE system naturally and easily.

[0048] Finally, referring to Figure 1, the third subblock is the integrated endpoint ("IE") 150, which has a pool of physical functions 152. This subblock receives configuration from the PCIe topology mapper 132 and accordingly groups the PFs from the pool of physical functions 152 into their respective vEPs.

[0049] In one or more embodiments, to achieve adaptability, the PCIe topology mapper 132 subblock may be implemented in two ways. The first method is to implement the PCIe topology mapper 132 in a programmable unit, such as FPGA soft logic, to load different topologies. Examples of such soft logic include user-programmable logic (CLB, BRAM, etc.) shown in Figure 4 and described below.

[0050] Therefore, this first method utilizes conventional FPGA methods for loading different bitstreams. The second method involves implementing the PCIe topology mapper 132 via a bypass interface so that the firmware kernel can directly respond to host configuration requests to create various topologies. This second method will now be illustrated with reference to Figure 2.

[0051] Figure 2 shows an alternative exemplary FPGA implementation of the exemplary system, which is a PCIe integrated endpoint system 210, to illustrate the second method described above, namely the “firmware intervention” method. Figure 2 maps the three subblocks of the PCIe integrated endpoint system 110 shown in Figure 1 to elements of an alternative exemplary FPGA, via light shading and different lines around the drawn boxes. Such exemplary FPGA is shown in Figure 4 and described below.

[0052] Referring to Figure 2, the key in the lower right of the figure indicates which of the three subblocks from the example in Figure 1—namely, PCIe upstream port 120, VHPC 130, and IE 110—maps to the various elements of the exemplary FPGA in Figure 2. Thus, PCIe upstream port 120, shown with a dotted boundary on the left side of Figure 2, maps to PCIe hard block 121 in Figure 2. This is the element that communicates with host 105. Similarly, VHPC 130, shown with a light shading and a solid boundary, maps to eight elements in Figure 2, also shown with light shading and a solid boundary, and also to firmware topology kernel 141, shown outside and to the right of the exemplary system 210. These nine VHPC elements have a total of index numbers 131 to 141. Finally, integration endpoint 150, shown with a dashed boundary, maps to seven elements in Figure 2, also shown with a dashed boundary. Considering the shading used in Figure 2, various elements can be easily identified, including each of the groups of elements that implement VHPC 130 and the integrated endpoint 150, respectively.

[0053] Continuing to refer to Figure 2, starting with the elements of system 210 that map to the PCIe upstream port 120 subblock in Figure 1, Figure 2 has only one such element, namely the PCIe hard block 121. The PCIe hard block 121 communicates with the host 105 and also with several other elements of system 210, which will be described next.

[0054] Next, the VHPC 130 is mapped to nine separate elements, as shown in the figure. Each of these nine elements in Figure 2 is shown as a block with light shading, and each such block is surrounded by a solid line boundary. Starting from the top of the figure, these include the PCIe interrupt generator 134, the requester request (RQ) MUX 131, the completer completion (CC) MUX 133, the completer request (CQ) MUX 135, the configuration bypass 136, the vDSP register 137, the vDSP processing 138, and the virtual hotplug handler 139.

[0055] Finally, the integrated endpoint 150 subblock in Figure 1 is mapped to seven distinct elements in Figure 2. Each of these elements is depicted in Figure 2 as an unshaded block with a dashed border. These seven elements, starting from the top of Figure 2, include the Requester Completion (RC) process 151, error message generation 153, CC process 155, RC data path process 157, CC data path register 156, vEP data path register 158, and vEP process 159.

[0056] As can be understood by comparing Figure 2 with the previous Figure 1, the PCIe topology mapper in Figure 1 is divided into two parts in the example in Figure 2: namely, the configuration bypass 136 and the FW topology kernel 141. Thus, in this exemplary implementation, all configuration requests (PCIe completer request (CFG) 176 sent from host 105) are forwarded to CQ mux 135 via PCIe hard block 121, from there to configuration bypass 136 via link 176 which is for PCIe completer requests (configuration), and from configuration bypass 136 to firmware ("FW") topology kernel 141. They are sent from the FW topology kernel 141 through essentially the same path, namely from the FW topology kernel 141 back to the configuration bypass 136, and upon receipt at the configuration bypass 136, they are sent via link 174 to the CC mux 133, which then sends a completion signal to the PCIe hard block 121, which in turn forwards the completion signal to the host 105.

[0057] In the case of a read request, the FW topology kernel 141 collects the status from the status register of the configuration bypass 136, maps the status to the current topology, and returns completion to the host 105 via the configuration bypass 136, along the link 174 to the CC mux 133, and through the PCIe hard block 121.

[0058] In the case of a write request, the FW topology kernel 141 maps it to the current topology, writes it to the corresponding register, and returns completion to the host 105. For non-data path-related access targeting virtual link-related registers, the FW topology kernel 141 may, for example, host those registers themselves, meaning that it is not necessary to read the status from the configuration bypass 136 or write the configuration to the configuration bypass 136.

[0059] As shown in Figure 2, the configuration bypass 136 has register access to the vDSP register 141 and the vEP data path register 158, and exchanges configuration information with the virtual hot-plug handler 139. Similarly, the vDSP register 137 and the vEP data path register 158 exchange control and status information with the vDSP process 138 and the vEP process 159. As described above, PCIe completer requests from the host 105 are sent via the CQ mux 135. These may be configured PCIe completer requests sent via link 176, or they may be unconfigured PCIe completer requests sent via link 177, for example.

[0060] It should be noted that any links or signal paths shown in the exemplary system 210 in Figure 2, but not explicitly described above, are standard PCIe links and protocols and are not closely related to the VHPC aspects that are the focus of this disclosure. Those skilled in the art will readily understand the standard PCIe structures and protocols used when responding to hot-plug instructions.

[0061] Figure 3 is a flowchart of method 300 for managing a pool of PFs in a PCIe IE according to one embodiment. Method 300 includes blocks 310-350. In alternative embodiments, method 300 may include more or fewer blocks. Method 300 begins in block 310, where a configuration instruction is received from a remote manager, which specifies the topology of the integrated endpoint for the PCIe connection. For example, the instruction may be received on an FPGA on which the PCIe integrated endpoint system 110 in Figure 1 is implemented. For example, the configuration instruction may be received from a remote management agent 103, as further shown in Figure 1. Finally, for example, the configuration instruction may instruct the integrated endpoint for the PCIe connection to divide the IE into K vDSPs, where K is an integer from 1, shown with respect to topology 1 180, to some upper limit N, shown with respect to topology N 185, both shown in Figure 1.

[0062] From block 310, method 300 proceeds to block 320, where the topology is implemented within the IE. For example, the topology implemented according to the configuration instructions may be more centralized, such as in the case of topology 1 180 in Figure 1, where there is a single vDSP to which all available PFs are associated, or the topology may be completely decentralized, such as in the exemplary topology N 185 in Figure 1, where there are N vEPs, each having only one PF.

[0063] From block 320, method 300 proceeds to block 330, where a hot-plug instruction is received from the remote manager, which instructs the VHPC to add one or more vEPs to the vDSP, remove one or more vEPs from the vDSP, or perform any combination of these two actions.

[0064] From block 340, method 300 proceeds to block 350, where at least one PCIe interrupt is initiated to notify the PCIe-connected hosts of the revised topology. For example, the PCIe interrupt may be sent from the PCIe interrupt generator 134 in Figure 1, or from the PCIe interrupt generator 134 of an alternative PCIe integrated endpoint system 210, for example, shown in Figure 2.

[0065] From block 350, method 300 proceeds to block 360, where a configuration request is received from the host in response to a PCIe interrupt initiated in block 350. In one or more embodiments, the configuration request sent by host 105 is a standard PCIe configuration request, and the host polls the DSP for PCIe hot-plug status information. Method 300 terminates in block 360.

[0066] As described above, in one or more embodiments, FPGA soft logic and / or software intervention may be used to provide remote configuration, dynamic allocation, and PCIe compliance. Figure 4 shows an exemplary FPGA architecture that may be used in this way. Referring to Figure 4, FPGA 400 includes a multi-gigabit transceiver ("MGT") 1, a configurable logic block ("CLB") 2, blocks of random access memory ("BRAM") 3, an input / output block ("IOB") 4, configuration and clocking logic ("CONFIG / CLOCKS") 5, a digital signal processing block ("DSP") 6, dedicated input / output blocks ("I / O") 7 (e.g., configuration port and clock port), and a number of different programmable tiles including other programmable logic such as a digital clock manager, analog-to-digital converter, and system monitoring logic 8. Some FPGAs also include a dedicated processor block ("PROC") 10. The FPGA 400 may also include one or more instances of the PCIe integrated endpoint system 100 or the PCIe integrated endpoint system 210, as also mentioned above.

[0067] In some FPGAs, each programmable tile may include at least one programmable interconnect element ("INT") 11 having connections to input and output terminals 20 of a programmable logic element within the same tile, as shown in the example included at the top of Figure 5. Each programmable interconnect element 11 may also include connections to interconnect segments 22 of adjacent programmable interconnect elements within the same tile or other tiles. Each programmable interconnect element 11 may also include connections to interconnect segments 24 of a general-purpose routing resource between logic blocks (not shown). A general-purpose routing resource may include routing channels between logic blocks (not shown) containing tracks of interconnect segments (e.g., interconnect segment 24) and switch blocks (not shown) for connecting interconnect segments. The interconnect segments of a general-purpose routing resource (e.g., interconnect segment 24) may span one or more logic blocks. The programmable interconnect elements 11, together with the general-purpose routing resource, implement a programmable interconnect structure ("programmable interconnect") for the illustrated FPGA.

[0068] In an exemplary implementation, CLB2 may include a single programmable interconnect element ("INT") 11 in addition to a configurable logic element ("CLE") 12 that can be programmed to implement user logic. BRAM3 may include one or more programmable interconnect elements in addition to a BRAM logic element ("BRL") 13. Typically, the number of interconnect elements included in a tile depends on the height of the tile. In the embodiment of the depiction, the height of the BRAM tile is the same as that of five CLBs, but other numbers (e.g., four) are also available. DSP tile 6 may include a suitable number of programmable interconnect elements in addition to a DSP logic element ("DSPL") 14. IOB4 may include, for example, one instance of a programmable interconnect element 11 in addition to two instances of input / output logic ("IOL") 15. As will be apparent to those skilled in the art, for example, the actual I / O pads connected to the I / O logic element 15 are typically not limited to the area of ​​the input / output logic element 15.

[0069] In the illustrated embodiment, a horizontal region near the center of the die (shown in Figure 5) is used for configuration, clock, and other control logic. A vertical column 9 extending from this horizontal region or column is used to distribute clock and configuration signals across the width of the FPGA.

[0070] Some FPGAs utilizing the architecture shown in Figure 4 include additional logic blocks that interrupt the regular columnar structure that makes up the majority of the FPGA. These additional logic blocks may be programmable blocks and / or dedicated logic. For example, processor block 10 spans several rows of CLB and BRAM. Processor block 10 can be a variety of components ranging from a single microprocessor to a complete programmable processing system including microprocessors, memory controllers, and peripherals.

[0071] Please note that Figure 4 is intended to show only illustrative FPGA architectures. For example, the number of logic blocks in a row, the relative width of the row, the number and order of the rows, the type of logic blocks contained in the row, the relative size of those logic blocks, and the interconnection / logic implementation configuration included in the top of Figure 4 are merely illustrative. For example, in a real FPGA, wherever a CLB appears, it typically contains two or more adjacent rows of CLBs to facilitate efficient implementation of user logic, although the number of adjacent CLB rows will vary depending on the overall size of the FPGA.

[0072] Therefore, in one or more embodiments where the extended hardware capabilities provide multiple virtual devices on a single physical card, and thus each does not have its own PCIe switch, a virtual hot-plug controller emulates hot-plug functionality for any of those virtual devices on the physical card. In such embodiments, the virtual PCIe hot-plug controller can virtually hot-add or hot-remove either a single PCIe physical function or a group of PCIe physical functions as a device via a native PCIe hot-plug mechanism without a physical switch. In one or more embodiments, FPGA soft logic and / or software intervention may be used to provide remote configuration, dynamic assignment, and PCIe compliance.

[0073] The above describes embodiments of the present invention, but other embodiments and further embodiments of the present invention can be devised without departing from the basic scope of the present invention, and the scope of the present invention is determined by the following "Claims".

Claims

1. A method for managing a pool of physical functions ("PF") in a Peripheral Component Interconnect Express (PCIe) integrated endpoint, wherein the PCIe integrated endpoint includes a virtual hot-plug controller (VHPC), and the method Receiving configuration instructions from the remote management agent that specify the topology for the PCIe-connected integrated endpoint (IE), Implementing the topology on the aforementioned IE, Receiving a hot-plug command from the aforementioned remote management agent, Adding or removing a virtual endpoint (vEP) and its corresponding PF from a virtual downstream port (vDSP) on the IE, at least in part based on the hot-plug instruction, the VHPC generates a PCIe-compliant interrupt to notify the host, the PCIe-compliant interrupt binds or unbinds a device driver on the host, and allocates or releases resources for hot-adding or hot-removing the vEP, The topology defines the number of vDSPs provided on the IE, and the number of vDSPs in the topology remains fixed until a new configuration command is received from the remote management agent. method.

2. Based on the aforementioned hot-plug command, Adding one or more PFs to vEP, Removing one or more PFs from vEP, or The method according to claim 1, further comprising at least one of any combination of adding PF to vEP and removing it from vEP.

3. The method according to claim 1, wherein the PCIe-connected integrated endpoint is provided to a PCIe integrated endpoint system, and a host is coupled to the PCIe integrated endpoint system via a PCIe link.

4. The method according to claim 3, wherein both the host and the PCIe integrated endpoint system are located within a data center.

5. The method according to claim 1, wherein each of the vDSPs has an associated vEP.

6. The method according to claim 5, wherein each of the vEPs is coupled to each of the vDSPs.

7. The method according to claim 5, wherein implementing the topology further includes grouping the PFs into each of the vEPs.

8. A Peripheral Component Interconnect Express (PCIe) integrated endpoint system, A PCIe-connected integrated endpoint ("IE") with a pool of PFs, A virtual hot-plug controller ("VHPC"), The remote management agent receives a configuration command indicating the topology of the IE, The topology is implemented on the aforementioned IE, Upon receiving a hot-plug command from the aforementioned remote management agent, Based on the hot-plug instruction, the VHPC is configured to add or remove at least one virtual endpoint (vEP) and its corresponding PF from a virtual downstream port (vDSP) on the IE, the VHPC generates a PCIe-compliant interrupt via a PCIe interrupt generator to notify the host, the PCIe-compliant interrupt binds or unbinds a device driver on the host, and allocates or releases resources for hot-adding or hot-removing the vEP, Equipped with VHPC, The topology defines the number of vDSPs provided on the IE, and the number of vDSPs in the topology remains fixed until a new configuration command is received from the remote management agent. PCIe integrated endpoint system.

9. The system according to claim 8, further comprising a PCIe upstream port coupled to the VHPC and the host, configured to communicate at least one PCIe interrupt to the host.

10. The system according to claim 8, wherein the VHPC receives the configuration command from the remote management agent.

11. The aforementioned VHPC is The system according to claim 8, further comprising an input interface configured to receive the configuration command and the hot-plug command.

12. The system according to claim 8, wherein the topology includes a set of vDSPs and corresponding vEPs, each vEP having one or more PFs.

13. A method for managing a pool of physical functions ("PFs") in a PCIe integrated endpoint system, Receiving a configuration instruction that specifies a topology for a PCIe-connected integrated endpoint (IE), wherein the topology includes a set of virtual downstream ports (vDSPs) and corresponding virtual endpoints (vEPs), and at least one of the vEPs includes a plurality of PFs. Implementing the topology on the aforementioned IE, Receiving a hot-plug command, The process includes, at least in part, adding a vEP to or removing a vDSP on the IE based on the aforementioned hot-plug instruction, method.

14. A PCIe-connected integrated endpoint ("IE") with a pool of PFs, A virtual hot-plug controller ("VHPC"), A configuration command is received from a remote manager, which represents the topology of the IE, wherein the topology includes a set of virtual downstream ports (vDSPs) and corresponding virtual endpoints (vEPs), and at least one of the vEPs includes multiple PFs. The topology is implemented on the aforementioned IE, Upon receiving a hot-plug command from the aforementioned remote manager, Based on the hot-plug command, at least one vEP is configured to be added to or removed from the vDSP of the IE. Equipped with VHPC, PCIe integrated endpoint system.

Images