Peripheral component interconnect express peripheral sharing
By introducing a peripheral agent subsystem into the SOC system, the complexity of multiple hosts sharing peripheral devices is solved, enabling efficient peripheral device sharing and management in non-multi-root sensing environments, simplifying multi-root operations, and improving system flexibility and availability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TEXAS INSTRUMENTS INC
- Filing Date
- 2020-12-28
- Publication Date
- 2026-07-10
AI Technical Summary
In existing technologies, sharing peripheral devices between SoCs involves significant overhead and complex system configuration, which limits the availability and deployment of PCIe peripheral devices. In particular, when multiple hosts need to share peripheral devices, the implementation complexity of SR-IOV and MR-IOV is high, making it difficult to effectively share peripheral devices.
A peripheral agent subsystem is introduced to perform virtual function mapping and management between multiple hosts and SR-IOV peripheral devices. The peripheral agent subsystem presents the virtual functions or multi-functional peripheral devices of SR-IOV peripheral devices to the hosts, realizes multi-root I/O virtualization, and simplifies the development of multi-root domains.
It enables the effective sharing and management of SR-IOV peripheral devices through the peripheral agent subsystem in the absence of multiple root sensing devices and switches, simplifying multi-root operations and improving the availability and deployment efficiency of peripheral devices.
Smart Images
Figure CN115023692B_ABST
Abstract
Description
Background Technology
[0001] The demand for increased processing power is endless. As vehicles become more autonomous and industrial processes rely more heavily on neural networks, the need for higher levels of processing power only grows. While significant progress has been made in integrating high-performance computing systems into systems-on-a-chip (SoCs), limitations remain. There is also a need to share peripherals across various SoCs. Although SoCs can include Ethernet capabilities and have… (Fast interconnection of peripheral components) ports, but sharing peripherals outside the SoC presents many problems. Sharing via Ethernet incurs significant overhead for the devices involved and involves complex system configurations. While PCIe enables the sharing of peripherals between systems with individual root controllers, this sharing introduces numerous difficulties for the peripherals and intervening components. These difficulties limit the availability and deployment of shared PCIe peripherals, leading to greater challenges in combining high-performance SoCs with shared peripherals. Summary of the Invention
[0002] A peripheral agent subsystem is placed between multiple hosts, each with its own root controller, and a single-root I / O virtualization (SR-IOV) peripheral device to be shared. The peripheral agent subsystem provides a root controller for endpoints coupled to the SR-IOV peripheral device or devices, and multiple endpoints for the root controller coupled to the hosts. The peripheral agent subsystem maps virtual functions of the SR-IOV peripheral device to the multiple endpoints as needed to allow the virtual functions to be assigned to the hosts. The physical functions of the SR-IOV peripheral device are managed by the peripheral agent subsystem to enable a required number of virtual functions. The peripheral agent subsystem can also map individual physical functions of multi-functional peripheral devices to the multiple endpoints in the peripheral agent subsystem as needed to allow individual functions to be assigned to the hosts. The virtual functions of the SR-IOV peripheral device or the individual physical functions of the multi-functional peripheral device are then presented to the appropriate hosts as physical or virtual functions.
[0003] This allows SR-IOV peripherals or multi-function peripherals to be used as multi-root I / O virtualization (MR-IOV) peripherals, enabling the development of multi-root domains without the hassle of finding MR-IOV peripherals and multi-root sensing (MRA) switches. Attached Figure Description
[0004] Detailed descriptions of various examples are now provided with reference to the accompanying drawings, in which:
[0005] Figure 1 It is a high-order block diagram of two hosts that share two peripheral devices using PCIe.
[0006] Figure 2 yes Figure 1 A more detailed block diagram.
[0007] Figure 3 This is a block diagram of an instance used for sharing SR-IOV peripheral devices.
[0008] Figure 4 It is based on Figure 2 A diagram of a more complex instance.
[0009] Figure 5 It is based on Figure 3 A diagram of a more complex instance.
[0010] Figure 6 yes Figure 3 The flowchart shows the operation of the peripheral agent device.
[0011] Figure 7 This is an explanation Figure 3 A block diagram of an instance of the PCIe register in an example.
[0012] Figure 8 This is an explanation Figure 3 A block diagram of an example of a data transfer operation from a peripheral device to the host.
[0013] Figure 9 This is an explanation Figure 3 A block diagram of an example of a data transfer operation from the host to a peripheral device.
[0014] Figure 10 This is an explanation Figure 3 A block diagram of an instance of an MSI operation in the example.
[0015] Figure 11 It is a diagram illustrating the possible configurations of the instance system.
[0016] Figure 12 yes Figure 3 A hardware block diagram of an instance of the peripheral agent subsystem.
[0017] Figure 13 This is a hardware block diagram of an example of a SoC. Detailed Implementation
[0018] For reference Figure 1 This describes computer system 100, in which host 1 102 and host 2 104 are connected to PCIe switch 106, and PCIe switch 106 enables PCIe NVM Express. TM (NVMe TMThis is an example of a configuration where memory 108 and PCIe 25G Ethernet controller 110 are connected to PCIe switch 106. Ideally, this configuration would allow host 1 102 and host 2 104 to access NVMe memory 108 for shared memory storage and utilize the 25G Ethernet controller 110 for high-speed access to external systems. However, in practice, this configuration is not like... Figure 1 It's not as simple as it appears. Problems arise because PCIe devices can be shared by multiple root controllers. Hosts (e.g., Host 1 102 and Host 2 104) have a PCIe root controller that acts as the top of the PCIe tree or network. PCIe provides two methods for sharing peripherals. The first is called Single Root I / O Virtualization (SR-IOV). The second is called Multi Root I / O Virtualization (MR-IOV), and the device is multi-root aware (MRA). SR-IOV allows multiple virtual machines on a physical processor chip, operated through a single PCIe root controller, and the host operating system or hypervisor to access the shared peripheral. The shared peripheral presents physical functions and multiple virtual function register sets at the PCIe interface. Each virtual machine or kernel is assigned one or more virtual functions, while the physical functions are assigned to the operating system or hypervisor. However, SR-IOV is useless when two hosts (each containing a PCIe root controller) must share the same PCIe peripheral. To enable multiple hosts with multiple root controllers to share PCIe peripherals, MR-IOV peripherals can be used, provided that any intervening PCIe switch is also MR-IOV compliant. While many SR-IOV-compliant peripherals exist, MR-IOV-compliant peripherals and MR-IOV-compliant switches are very rare. This is due to the added complexity required in both the peripheral and the switch. To be MR-IOV compliant, an MRRA PCIe switch must know the address space used by the host to properly route transactions. MR-IOV-compliant peripherals can have more complex interfaces to provide basic functionality and present multiple virtual endpoints, each with one physical function and several virtual functions. These additional requirements limit the availability of MR-IOV-compliant switches and peripherals.
[0019] Figure 2This is an example of a computer system 200 containing hosts 1 202 and 2 204 connected to an MR-IOV compliant MRA PCIe switch 206. An SR-IOV peripheral device 208 is connected to the MRA PCIe switch 206. This SR-IOV peripheral device 208 can be used by only one of the two hosts 202 and 204. An MR-IOV peripheral device 210 is connected to the MRA PCIe switch 206 and can be used by both hosts 202 and 204. The SR-IOV peripheral device 208 presents a physical function (PF) zero (PF0) 212 and two virtual functions (VFs), VF1 214 and VF2 216, at the endpoint (EP) PCIe interface EP0 217. In contrast, the MR-IOV peripheral device 210 presents at EP1 219 a basic function (BF) 244 for managing multiple root features of the MR-IOV peripheral device 210 and two virtual endpoints (VEPs), VEP2 218 and VEP3 220. VEP2 218 presents PF0 222, while VEP3 220 presents PF1 224 and VF1 226. In the illustrated embodiment, the instance host 1 202 has control over the SR-IOV peripheral device 208 and enables the illustrated module 228 to interact with PF0 212, module 230 to interact with VF1 214, and module 232 to interact with VF2 216. The host 1 202 also has control over VEP2 218 and enables module 236 to interact with PF0 222. Host 2 204 controls VEP3 220 and enables module 240 to interact with PF1 224 and module 242 to interact with VF1 226. MRA PCIe switch 206 includes a multi-root PCI manager (MR-PCIM) 246 responsible for discovering and configuring virtual hierarchies within a multi-root topology (in the indicated instance, only MR-IOV peripherals 210). MRA PCIe switch 206 manages the multi-root aspect, ensuring that host 1 202 and host 2204 only see SR-IOV, multi-function, or single-function peripherals.
[0020] Figure 3 This describes an example computer system 300 that provides individual root controllers for multiple hosts to share SR-IOV peripheral devices. Host 1 302 and Host 2 304 are connected to a peripheral agent subsystem 306. The peripheral agent subsystem 306 is connected to an SR-IOV peripheral device 308. Figure 3The peripheral agent subsystem 306 provides a PCIe endpoint interface EP2 310 to cooperate with the root controller (RC) interface RC1 312 of the host 1302. The peripheral agent subsystem 306 also provides a PCIe endpoint interface EP3 314 to cooperate with the root controller PCIe interface RC2 316 of the host 2 304. The peripheral agent subsystem 306 provides a root controller PCIe interface RC3 318 to cooperate with the PCIe endpoint interface EP0 320 on the SR-IOV peripheral device 308, which includes PF0 321, VF1 322, and VF2 324. These are controlled by RC3 318 in the peripheral agent subsystem 306. The peripheral agent subsystem 306 has a module 326 for interacting with PF0 321, a module 328 for interacting with VF1 322, and a module 330 for interacting with VF2 324. The peripheral agent subsystem 306 presents a first cloned example of PF0 321, labeled PF0'332, at EP2 310 coupled to RC1 312. Figure 3 In the example, PF0'332 is VF1 322 of peripheral device 308, which is presented as a PCIe endpoint physical function by peripheral agent subsystem 306. Similarly, EP3 314 presents a second cloned example of PF0 321, denoted as PF0”334, in Figure 3 In the example, PF0”334 is VF2 324 of peripheral device 308. In this way, the virtual functionality of peripheral device 308 can be arbitrarily divided among the cloned examples PF0'332 and PF0”334, and each of the cloned examples of physical functionality presents an independent subset of virtual functionality.
[0021] The peripheral agent subsystem 306 contains various manager modules. The policy manager 336 stores configuration allocation information regarding specific physical and virtual functions of attached peripheral devices to various attached hosts. The configuration manager 338 contains a physical function manager 340 that manages the configuration of PF0 321 of the peripheral device 308. The endpoint manager 342 contains and performs remaining endpoint management functions within the configuration manager 338 to allow hosts to recognize and be interrupted by shared peripheral devices. The mapping and forwarding module 344 manages the memory mapping and data forwarding routes contained in the peripheral agent subsystem 306.
[0022] Figure 4This illustrates a more complex example of a multi-root PCIe system 400. Host 1 402 and Host 2 404 are connected to an MRA PCIe switch 406. An MR-IOV peripheral 408 is connected to the MRA PCIe switch 406 and presents two virtual endpoints, VEP2 410 and VEP3 412. This combination creates a multi-root domain 414. A non-MRA PCIe switch 416 is connected to the MRA PCIe switch 406 and connects an SR-IOV peripheral 418 and a single-function peripheral 420 to it. The MR-IOV peripheral 408 is shared between Host 1 402 and Host 2 404, while the SR-IOV peripheral 410 and single-function peripheral 420 are assigned to either Host 1 402 or Host 2 404 because they are not multi-root aware.
[0023] Figure 5 This is an example of a more complex system 450 that includes a peripheral agent subsystem. Host 1 452 is connected to a first PCIe switch 456, which is not multi-root aware. Host 2 454 is connected to a second PCIe switch 458. PCIe switch 456 is connected to the PCIe endpoint interface EP2 460 of the peripheral agent subsystem 462, while PCIe switch 458 is connected to the PCIe endpoint interface EP3 464. The root controller interface RC3 466 of the peripheral agent subsystem 462 is connected to PCIe switch 468. A first SR-IOV peripheral device 470 is connected to PCIe switch 468. A second SR-IOV peripheral device 472 is also connected to PCIe switch 468. This set of devices together forms a multi-root domain 474 in system 450. This is then compared with... Figure 4 Compared to the multi-root domain 414, it should be noted that the hosts, switches, or peripherals in the multi-root domain 474 are non-multi-root aware devices. However, the peripheral agent subsystem 462 allows the use of more commonly available SR-IOV peripherals and ordinary PCIe switches, while still providing multi-root operation. Single-function peripheral 476 is connected to PCIe switch 456, while single-function peripheral 478 is connected to PCIe switch 458 and single-function peripheral 480 is connected to PCIe switch 468. In this configuration, the virtualization capabilities of SR-IOV peripherals 470 and 472 can be shared between hosts 452 and 454. Single-function peripheral 480 can be utilized by either host 1 452 or host 2 454. Single-function peripheral 476 is dedicated to host 1 452, while single-function peripheral 478 is dedicated to host 2 454. With the use of the peripheral agent subsystem 462, multi-root domains 474 can be developed using more commonly available SR-IOV devices and non-multi-root sensing switches without the need for multi-root sensing devices.
[0024] For reference Figure 6,illustrate Figure 3 This is an instance of the initialization of the peripheral agent subsystem 306. In operation 602, hardware link negotiation is performed between RC3 318 and EP0 320. In operation 604, the PCIe software stack in the peripheral agent subsystem 306 enumerates PF0 321. In operation 606, the PCIe driver in the physical function manager 340 is bound to PF0 321, making the SR-IOV peripheral device 308 configurable. In operation 608, the policy manager 336 determines the number of virtual functions to be enabled in the SR-IOV peripheral device 308. In operation 610, the physical function manager 340 writes this number of virtual functions into the peripheral device register of PF0 321. In operation 612, the PCI software stack enumerates the virtual functions of EP0 320. In operation 614, the PCIe endpoint function driver in endpoint manager 342 is bound to EP2 310 and EP3 314 to allow the initialization of endpoint instances connected to host 1 302 and host 2 304 using appropriate virtual function configuration space data. In operation 616, policy manager 336 determines the endpoint instances connected to the hosts and the virtual functions that should be used to initialize each endpoint instance. In some instances, physical functions also provide functionality beyond just management, and in those cases, physical functions may also be mapped to endpoint instances. In other instances, as described below, peripheral devices are directly assigned to endpoint instances. In operation 618, endpoint manager 342 initializes the endpoint instance using virtual function configuration space data and / or the relevant portion of the physical function configuration space used to provide the physical function. For example, if VF1 322 is to be mapped to EP2 310, then the values in the VF1 322 configuration space data are copied to EP2 310 to present the configuration space data for the appropriate cloned function (e.g., PF0'332 or virtual function). The following describes the mapping of basic physical function data. In operation 620, the mapping and forwarding module 344 configures various Translation Back Buffers (TLBs) and transaction forwarders in the peripheral agent subsystem 306 to perform the translation of memory addresses in memory transactions and forward those memory transactions from the virtual functions in the SR-IOV peripheral device 308 to EP2 310 or EP3 314. In operation 622, hardware link negotiation is performed between RC1 312 and EP2 310 and between RC2 316 and EP3 314. In operation 624, normal software execution begins in RC1 312 and RC2 316. Thus, the peripheral agent subsystem 306 is initialized and ready for operation.
[0025] The routing mechanism in the peripheral agent subsystem 306 provides a way to forward transactions in one direction from virtual functions VF1 322 and VF2 324 in the SR-IOV peripheral device 308 to host 1 302 and host 2 304 respectively, and in the other direction from host 1 302 and host 2 304 to VF1 322 and VF2 324 in the SR-IOV peripheral device 308 respectively. The peripheral agent subsystem provides a peripheral virtualization unit (PVU) (812), virtual ID mapping logic 810, and outbound address translation units 818 and 820 in the routing mechanism to forward transactions from VF1 322 and VF2 324 in the SR-IOV peripheral device 308 to host 1 302 and host 2 304 respectively. The peripheral agent subsystem provides a base address register, inbound and outbound address translation unit (described below) in the routing mechanism to forward transactions from host 1 302 and host 2 304 to VF1 322 and VF2 324 in SR-IOV peripheral device 308, respectively.
[0026] Figure 7 This describes the register configuration performed by endpoint manager 342. It describes the registers contained in the cloned examples PF0'332 and PF0"334 of the peripheral agent subsystem and the corresponding PF0 321 of the device to be shared. Endpoint manager 342 initializes the configuration spaces of endpoint examples EP2 310 and EP3 314. Endpoint manager 342 initializes the device identification capabilities in PF0'332 and PF0"334 using values from the standard configuration space header and the SR-IOV capability in the extended configuration space of EP0 320. Endpoint manager 342 does not need to initialize any other capabilities existing in PF0'332 and PF0"334 besides the MSI capability (which depends on the capability of PF0 321 in EP0 320), because this operation is performed in step 618 if necessary. Regarding the MSI capability, certain fields are initialized based on the corresponding physical or virtual function of the mapped SR-IOV peripheral 308, such as a multi-message capability field indicating the number of interrupt vectors required. If the physical function presented to the root controller is mapped from a virtual function, then the VF configuration space data has been copied to the PF configuration space presented in step 618.
[0027] Figure 8This describes the PCI memory transfers from SR-IOV peripheral device 308 to host 1 302 and host 2 304. Internal addresses are reserved or assigned when VF1 322 and VF2 324 are enumerated. The PCIe driver of host 1 302 contains memory buffer MEM(A) 802, and the PCIe driver of host 2 304 contains memory buffer MEM(B) 804. The PCIe drivers in host 1 302 and host 2 304 initialize the buffer addresses of their memory buffers MEM(A) 802 and MEM(B) 804 in the corresponding buffer address register 806 of VF1 322 and the buffer address register 808 of VF2 324. Therefore, VF1 322 contains access to memory buffer MEM(A) 802, and VF2 324 contains access to memory buffer MEM(B) 804. When the host address is unknown, entries in TLB 814 and TLB 816 can be programmed to translate from the host address to a reserved internal address and destination endpoint. Similarly, outbound address translation units 818 and 820 are programmed to translate from an internal address back to a host address. The steps described above are initialization steps and do not need to be performed for each transaction. VF1 322 and VF2 324 each have a requester ID developed during the initialization operation. The requester ID (ReqID) is used in a transaction to identify a specific PF or VF. When a PF or VF provides a memory transaction, the memory transaction contains the requester ID and the host memory address. For example, a memory transaction from VF1 322 would contain a requester ID of 01x01 and an address in register memory MEM(A) 802. The peripheral agent subsystem 306 includes virtual ID mapping logic 810. The virtual ID mapping logic 810 converts the requester ID in a transaction from a VF to a virtual ID (virtID) and forwards the transaction to PVU 812. PVU 812 contains TLBs 814 and 816, one per virtID or VF. Entries in TLBs 814 and 816 translate memory addresses in a transaction to internal addresses of the peripheral agent subsystem 306 and serve the transaction to the designated endpoint. The selection of a TLB based on the virtID and the serving of the translated transaction to the designated endpoint work together to route the transaction to the appropriate endpoint. TLB 814 contains a translation back buffer entry for virtID0 corresponding to VF1 322, while TLB 816 contains a translation back buffer entry for virtID1 corresponding to VF2 324. When PVU 812 receives a transaction from virtual ID mapping logic 810, it uses the virtID to select the appropriate TLB 814 or 816. TLBs 814 and 816 translate host addresses in the transaction to the internal outbound addresses of the peripheral agent subsystem 306 for the relevant endpoint EP2 310 or EP3 314.Transactions with the currently translated addresses are provided to the corresponding outbound address translation unit (ATU) 818 in PF0'332 or the outbound ATU 820 in PF0'334. The outbound ATU 818 in PF0'332 or the outbound ATU 820 in PF0'334 translates the internal outbound address back to the appropriate host address to provide the transaction to the specific host and provide the appropriate ReqID for the specific function based on the internal address. After this final transaction address translation, a transaction is provided from EP2 310 to RC1 312 for host 1 302 or from EP3 314 to RC2 316 for host 2 304.
[0028] In summary, for memory operations from peripheral device 308 to host 1 302 or host 2 304, peripheral device 308 provides the transaction with the appropriate function (e.g., PF0 321, VF1 322, or VF2 324) and the host memory address. The ReqID is translated to virtID in the virtual ID mapping logic 810. The TLB in PVU 812 translates the host memory address to an internal outbound memory address. The outbound ATU translates the internal outbound memory address back to the host memory address and provides the transaction to the host. In this way, each of the cloned physical functions PF0'332 and PF0"334 presents itself to its respective host as a separate and independent SR-IOV peripheral device function assigned only to that host.
[0029] Figure 9This describes the memory transfer operations from the host to the peripheral device. During initialization, each virtual function uses an inbound ATU to map its base address register (BAR) to an internal memory address. VF1 322 has BAR 907 mapped to internal memory location access (A) 916 of peripheral device 308 using inbound ATU 903. VF2 324 has BAR 909 mapped to internal memory location access (B) 918 using inbound ATU 903. Then, during the root controller enumeration of the virtual function, memory is allocated from the outbound address space and initialized within the virtual function. RC3 318 allocates memory block 908 for VF1 BAR 907 in outbound address space 906. RC3 318 allocates memory block 910 for VF2 BAR 909 in outbound address space 906. RC3318 sets memory block 908 to access BAR 907 and memory block 910 to access BAR 909. Next, EP2 310 uses inbound ATU 930 and EP3 314 uses inbound ATU 932 to map their base address registers 902 and 904 to outbound address blocks 908 and 910. After this, host 1 302 and host 2 304 allocate MEM(C) 912 and MEM(D) 914 respectively, the host calls back the memory buffer, and initializes those addresses in BAR 902 of PF0'332 and BAR 904 of PF0"334. This completes the initialization of the peripheral agent subsystem 306 and peripheral device 308 to receive memory write transactions.
[0030] To perform a write operation, host 1 302 uses RC1 312 and the address mapped in MEM(C)912 to provide a memory transaction to EP2 310. EP2 310 receives the transaction, translates the host 1 302 address to an internal address in memory block 908 allocated for VF1 322, and provides the transaction to RC3 318. RC3 318 provides the transaction to EP0 320. Inbound ATU 903 translates the address according to BAR 907 and provides the transaction to designated memory location access (A)916. Similarly, host 2 304 uses RC2 316 and the address mapped in MEM(D)914 to provide a memory transaction to EP3 314. EP3 314 receives the transaction, translates the host 2 304 address to an internal address in memory block 910 allocated for VF2 324, and provides the transaction to RC3 318. RC3 318 provides the transaction to EP0 320. Inbound ATU 903 translates the address according to BAR 909 and provides the transaction to the specified memory location access (B)918. The mapping of internal addresses to BARs of virtual functions of peripheral devices and the mapping of inbound ATUs to those internal addresses are used to route host transactions to the appropriate physical or virtual functions in the peripheral device.
[0031] PCIe uses Message Signaling Interrupts (MSI). MSI is used to deliver interrupt messages to host memory locations. Figure 10This illustrates an example of MSI operation. Host 1 302 has a reserved memory block MSI(P)1006 for MSI use, and host 2 304 has a reserved memory block MSI(Q)1008 for MSI use. EP2 310 contains outgoing MSI memory block (X)1002, and EP3 314 contains outgoing MSI memory block (Y)1004. The addresses of outgoing MSI memory blocks (X)1002 and (Y)1004 are programmed into the MSI registers of VF1 322 and VF2 324, respectively. Host 1 302 programs the address of memory block MSI(P) 1006 into the MSI register of PF0'332, while host 2 304 programs the address of memory block MSI(Q) 1008 into the MSI register of PF0'334. This completes the initialization operation. When VF1 322 or VF2 324 wants to send an MSI transaction, the virtual function provides the transaction with its requester ID and the address stored in outbound MSI memory block (X) 1002 or (Y) 1004, respectively. Virtual ID mapping logic 810 maps the requester ID to a virtual ID and provides the transaction to PVU 812. If necessary, PVU 812 translates the MSI address to the internal outbound address and routes the MSI transaction. Outbound ATUs 818 and 820 translate the internal outbound MSI address to host memory block MSI(P) 1006 or MSI(Q) 1008 and EP2 310 or EP3 314 delivers the transaction to host 1 302 or host 2 304.
[0032] Figure 11This describes the flexibility of mapping physical and virtual functions to the endpoints of the peripheral agent subsystem 306 and thus to the host. In the first instance, the peripheral agent subsystem 306 manages PF0 321, maps VF1 322 to PF0'332 of PCIe endpoint 310, maps VF2 324 to PF0'", and maps VF3 1108 to VF1 1104. This provides MR-IOV operation for the SR-IOV peripheral device 308. In the second instance, PF0 321 is managed by the peripheral agent subsystem 306, where VF1 322 is mapped to PF0', VF2 324 is mapped to PF1 1102, and VF3 1008 is mapped to PF2 (not shown). This maps all virtual functions to host 1 302 as physical functions, making the peripheral device 308 appear as a multi-functional PCIe endpoint. In the third instance, PF0 321 is managed by the peripheral agent subsystem 306, mapping VF1 322 to PF0'332, mapping VF2 324 to PF0'332, and mapping VF2 324 to PF0'332, and mapping VF3 1108 to VF1 1104. PF0 324 is mapped to VF1 1004 and VF3 1008 is mapped to VF2 1106. This allows different virtual machines in host 1 302 to utilize the three virtualization functions as normally in SR-IOV operation. In the fourth instance, SR-IOV peripheral 308 is directly assigned to EP2 310, such that PF0 321 is mapped to PF0'332 of PCIe endpoint 310, VF1 322 is mapped to VF1 1104, VF2 324 is mapped to VF2 1106 and VF3 1108 is mapped to VF3 (not shown). This mapping flexibility is achieved through PVU The 812 and TLB are used for routing peripheral device memory transfers and MSI operations, as well as for internal memory space allocation and BAR value copying of PF and VF configuration space data for host memory transfer operations to achieve the required functionality. These are just four examples of the flexibility of using the peripheral agent subsystem 306 to map physical and virtual functions, and other mappings can be easily performed.
[0033] Figure 12 It is the peripheral agent subsystem 306 (e.g.) Figure 3The hardware block diagram of the peripheral agent subsystem 306 is shown below. A processor 1202 (e.g., an ARM R5F core) provides basic processing functions within the peripheral agent subsystem 306. SRAM 1204 is provided as operational memory and serves as a memory block used by various devices within the peripheral agent subsystem 306. Various I / O ports (e.g., SPI 1208, UART 1210, and I2C 212) are present to allow external communication with and management by other devices in the system. Alternatively, management of the peripheral agent subsystem 306 can be performed through the extended capabilities of a PF presented at one endpoint. A non-volatile (NV) RAM interface 1214 connects to a non-transitory NVRAM 1216 containing firmware modules for providing the functionality of the peripheral agent subsystem 306. For example, a policy manager 336, a configuration manager 338 with its physical function manager 340 and endpoint manager 342, and a mapping and forwarding module 344 are contained in NVRAM 1216 for execution by the processor 1202. As with the previously mentioned PCIe software stack 1219, the operating system and / or hypervisor 1217 are stored in NVRAM 1216.
[0034] The first PCIe endpoint 1222 serves as an endpoint (e.g., EP2 310) and includes an outbound ATU 818, an outbound MSI memory 1002, an MSI capability register 1010, an inbound ATU 930, and a base address register 902. A second PCIe endpoint 1224 is provided to serve as a second endpoint (e.g., EP3 314) and includes an outbound ATU 820, an outbound MSI memory 1004, an MSI capability register 1012, an inbound ATU 932, and a base address register 904.
[0035] A PCIe root controller 1218 is presented as the root controller for SR-IOV peripheral devices to be shared among hosts connected to PCIe endpoints 1222 and 1224. An example of the PCIe root controller 1218 is the RC3 318. The PCIe root controller 1218 includes an inbound address space 1220, which contains various virtual function base address register spaces. Virtual ID mapping logic 810 is provided, as well as PVU 812 with various virtual ID TLBs 814 and 816 and others as needed. More than the two endpoints and one root controller shown in the various figures may exist on the peripheral agent subsystem 306. Each controller and endpoint will operate as described above.
[0036] Multiple SR-IOV peripheral devices can be connected to the root controller of the peripheral agent subsystem 306 and each can be mapped and managed as described above.
[0037] Figure 13This is a block diagram of an exemplary SoC 500 that can form host 1 302 and 2 304. A series of more powerful microprocessors 502 (e.g. The A72 or A53 core forms the main general-purpose processing block of the SoC 500, while the digital signal processor (DSP) 504 provides dedicated computing capabilities. A simpler microprocessor 506 (e.g., an ARM R5F core) provides general control capabilities within the SoC 500. A high-speed interconnect 508 connects the microprocessor 502, DSP 504, and microprocessor 506 to various other components within the SoC 500. For example, a shared memory controller 510, containing onboard memory or RAM 512, is connected to the high-speed interconnect 508 to function as onboard RAM for the SoC 500. A DDR memory controller system 514 is connected to the high-speed interconnect 508 and serves as an external memory interface to external DRAM memory. Similarly, a video acceleration module 516 and a radar processing accelerator (PAC) module 518 are connected to the high-speed interconnect 508. A vision processing accelerator module 520 is connected to the high-speed interconnect 508, as is a depth and motion PAC module 522. A graphics acceleration module 524 is connected to the high-speed interconnect 508. The display subsystem 526 is connected to the high-speed interconnect 508 and includes conversion logic 528 and output logic 530 to allow operation and connection with various video monitors where appropriate. A system service block 532 provides for normal SoC 500 operation and includes items such as a DMA controller, memory management unit, general-purpose I / O, mailbox, and the like. A serial connectivity module 534 is connected to the high-speed interconnect 508 and is included as a normal module within the SoC. A vehicle connectivity module 536 provides interconnects for external communication interfaces, such as a PCIe block 538, a USB block 540, and an Ethernet switch 542. The capture / MIPI module 544 includes a four-channel CSI-2 compliant transport block 546 and a four-channel CSI-2 receiver module and hub. Further details regarding the CSI-2 receiver module and hub are provided below.
[0038] MCU Island 560 is provided as a secondary subsystem and handles the operation of the integrated SoC 500 when other components are powered down to conserve energy. Processor 562 (e.g., one or more ARM R5F cores) operates as the master device and is coupled to high-speed interconnect 508 via isolation interface 561. MCU general-purpose I / O (GPIO) block 564 operates as a slave device. MCU RAM 566 is provided as local memory for MCU ARM processor 562. CAN bus block 568 (additional external communication interface) allows operation in a conventional CAN bus environment within a vehicle. Ethernet MAC (Media Access Control) block 570 is provided for further connectivity within the vehicle. Non-volatile memory (NVM) is connected to MCU ARM processor 562 via external NVRAM interface 569.
[0039] This is the instance host configuration, and many other host configurations can be utilized.
[0040] While the examples above describe using virtID mapping logic to translate ReqID to virtID and using TLB to translate host memory addresses to internal memory addresses, other examples use ReqID to route transactions from peripheral devices to the host and do not require virtID mapping logic. Still other examples do not use internal memory space and do not require TLB.
[0041] In the example described above, a TLB is provided for each VF in peripheral device 308. In another example, only a single TLB is provided, but the TLB is expanded to include either ReqID or virtID as a lookup value and provide the desired endpoint as an output value.
[0042] The term "coupled" is used throughout this specification. The term may encompass a connection, communication, or signaling path that achieves a functional relationship consistent with this description. For example, if device A generates a signal to control device B to perform an action, then in a first instance, device A is coupled to device B, or in a second instance, device A is coupled to device B via an intervening component C, such that device B is controlled by device A via a control signal generated by device A, provided that the intervening component C does not substantially alter the functional relationship between device A and device B.
[0043] Modifications to the embodiments are feasible within the scope of the claims, and other embodiments are also feasible.
Claims
1. A peripheral agent subsystem, comprising: Subsystem Peripheral Component Interconnect Fast PCIe Root Controller, which is used to couple to the peripheral device Peripheral Component Interconnect Fast PCIe endpoint; A first subsystem PCIe endpoint is used to couple to a first PCIe root controller of a first host. The first subsystem PCIe endpoint presents a first physical function to the first PCIe root controller. The first physical function is mapped to a first peripheral device virtual function. The second subsystem PCIe endpoint is used to couple to the second PCIe root controller of the second host. The second subsystem PCIe endpoint presents the second physical function to the second PCIe root controller. The second physical function is mapped to the second peripheral device virtual function. and A routing mechanism for routing PCIe memory transactions to the virtual function between the subsystem PCIe root controller and the corresponding endpoints from the first subsystem PCIe endpoint and the second subsystem PCIe endpoint, based on whether the virtual function selected from the first peripheral device virtual function and the second peripheral device virtual function is mapped to the first physical function or the second physical function.
2. The peripheral agent subsystem according to claim 1, wherein: The first physical function has a configuration space for providing physical function configuration space from PCIe peripheral devices and virtual function configuration space of the first peripheral device virtual function mapped to the first physical function, and The second physical function has a configuration space for providing the physical function configuration space from the PCIe peripheral device and the virtual function configuration space of the second peripheral device virtual function mapped to the second physical function.
3. The peripheral proxy subsystem according to claim 1, wherein the routing mechanism comprises: A peripheral virtualization unit (PVU) coupled between the subsystem PCIe root controller and the first and second subsystem PCIe endpoints translates memory addresses in a transaction from PCIe peripheral devices to internal memory addresses and routes the transaction to the first subsystem PCIe endpoint or the second subsystem PCIe endpoint based on the mapping of the virtual functions of the first and second peripheral devices and the virtual function of the PCIe peripheral device that initiated the transaction.
4. The peripheral proxy subsystem according to claim 3, wherein the routing mechanism comprises: Virtual ID mapping logic is used to translate the requester ID in a transaction from the PCIe peripheral device to a virtual ID, and The PVU utilizes the virtual ID when translating memory addresses.
5. The peripheral agent subsystem according to claim 1, wherein: The first and second subsystem PCIe endpoints include a base address register (BAR). The virtual functions of the first and second peripheral devices each include a BAR, and The routing mechanism directs the first and second endpoint BARs to the first and second internal memory addresses to be written into the first and second peripheral device virtual function BARs.
6. The peripheral agent subsystem according to claim 1, wherein: The virtual functions of the first and second peripheral devices utilize message signaling interrupt MSI, and The routing mechanism routes MSI transactions from the peripheral device virtual functions between the subsystem PCIe root controller and the first or second subsystem PCIe endpoint to which the peripheral device virtual function is mapped, selected from the first and second peripheral device virtual functions.
7. The peripheral agent subsystem according to claim 1, wherein: The first subsystem PCIe endpoint presents virtual functions to the PCIe root controller of the first host, and the virtual functions of the first subsystem PCIe endpoint are mapped to the virtual functions of the third PCIe peripheral device.
8. A computer system comprising: The first host has a first peripheral component interconnect fast PCIe root controller; The second host has a second PCIe root controller; Peripheral device, which has a peripheral device PCIe endpoint; and The peripheral agent subsystem includes: The subsystem PCIe root controller is coupled to the peripheral device's PCIe endpoint; A first PCIe endpoint coupled to the first PCIe root controller of the first host presents a first physical function to the first PCIe root controller, and the first physical function is mapped to a first peripheral device virtual function. A second PCIe endpoint, coupled to the second PCIe root controller of the second host, presents a second physical function to the second PCIe root controller, and the second physical function is mapped to a second peripheral device virtual function; and A routing mechanism for routing PCIe memory transactions to the virtual function between the subsystem PCIe root controller and the corresponding endpoint from the first and second PCIe endpoints, based on a virtual function selected from the first and second peripheral device virtual functions and mapped to the first physical function or the second physical function.
9. The computer system according to claim 8, wherein: The peripheral device has configuration space for the physical functions and each virtual function. The first physical function has a configuration space for providing data from the peripheral device physical function configuration space and the virtual function configuration space of the first peripheral device virtual function mapped to the first physical function, and The second physical function has a configuration space for providing data from the peripheral device physical function configuration space and the virtual function configuration space of the second peripheral device virtual function mapped to the second physical function.
10. The computer system of claim 8, wherein the routing mechanism comprises: A peripheral virtualization unit (PVU) coupled between the root controller and the first and second PCIe endpoints translates memory addresses in a transaction from the peripheral device to internal memory addresses and routes the transaction to the first PCIe endpoint or the second PCIe endpoint based on the mapping of the virtual functions of the first and second peripheral devices and the virtual function of the peripheral device that initiated the transaction.
11. The computer system of claim 10, wherein the routing mechanism comprises: Virtual ID mapping logic is used to translate the requester ID in a transaction from the peripheral device to a virtual ID, and The PVU utilizes the virtual ID when translating memory addresses.
12. The computer system according to claim 8, wherein: The first and second PCIe endpoints include a base address register (BAR). The virtual functions of the first and second peripheral devices each include a BAR, and The routing mechanism directs the first and second PCIe endpoint BARs to the first and second internal memory addresses to be written to the first and second peripheral device virtual function BARs.
13. The computer system according to claim 8, wherein: The virtual functions of the first and second peripheral devices utilize message signaling interrupt MSI, and The routing mechanism routes MSI transactions from the peripheral device virtual function between the PCIe root controller and the first or second PCIe endpoint to which the peripheral device virtual function is mapped.
14. The computer system according to claim 8, wherein: The peripheral device has a third virtual function, and The first PCIe endpoint presents the virtual function to the first PCIe root controller of the first host, and the virtual function of the first PCIe endpoint is mapped to the third virtual function.
15. A non-transitory memory storing a program to cause a processor or processors to perform a method of operating a peripheral agent subsystem, the peripheral agent subsystem comprising: Subsystem peripheral component interconnection fast PCIe root controller, which is used to couple to the PCIe endpoints of peripheral devices; The first PCIe endpoint is used to couple to the first PCIe root controller of the first host; The second PCIe endpoint is used to couple to the second PCIe root controller of the second host; and A routing mechanism is used to route PCIe memory transactions between the subsystem's PCIe root controller and the first and second PCIe endpoints. The method includes: Configure the first PCIe endpoint to present a first physical function to the PCIe root controller, the first physical function being mapped to a first peripheral device physical function; Configure the second PCIe endpoint to present a second physical function to the PCIe root controller, the second physical function being mapped to a second peripheral device physical function; and The routing mechanism is configured to route PCIe memory transactions between the subsystem PCIe root controller and selected endpoints of the first and second PCIe endpoints based on PCIe memory transactions directed to virtual functions selected from the first and second peripheral device virtual functions.
16. The non-transitory memory according to claim 15, wherein: The first physical function has configuration space, and The second physical function has configuration space. The method further includes: Configure the first physical function configuration space using data from the peripheral device physical function configuration space and the virtual function configuration space of the first peripheral device virtual function mapped to the first physical function, and Configure the second physical function configuration space using data from the peripheral device physical function configuration space and the virtual function configuration space of the second peripheral device virtual function mapped to the second physical function.
17. The non-transitory memory of claim 15, wherein the routing mechanism comprises: A peripheral virtualization unit (PVU), coupled between the subsystem PCIe root controller and the first and second PCIe endpoints, translates memory addresses in memory transactions from peripheral devices to internal memory addresses and routes memory transactions to the first or second PCIe endpoint based on the mapping of the virtual functions of the first and second peripheral devices and the virtual function of the peripheral device initiating the memory transaction. The method further includes: Assign internal addresses to the virtual functions of the first and second peripheral devices; Configure the PVU to transfer from the host memory address to the assigned internal address of the first and second peripheral device virtual functions in the memory transaction; and Configure the PVU to route memory transactions to the first or second subsystem PCIe endpoint based on the mapping from the first and second peripheral device virtual functions to the first and second subsystem PCIe endpoints.
18. The peripheral agent subsystem according to claim 15, wherein: The first and second PCIe endpoints include a base address register (BAR). The first and second PCIe peripheral device virtual functions each include a BAR. The method further includes: The first and second PCIe endpoint BARs are directed to the first and second internal memory addresses to be written to the first and second PCIe peripheral device virtual function BARs.
19. The peripheral agent subsystem according to claim 15, wherein: The virtual functions of the first and second peripheral devices utilize message signaling interrupt MSI, and The routing mechanism routes MSI transactions for virtual functions from a single-root I / O virtualized SR-IOV PCIe peripheral device between the root controller and the first or second PCIe endpoint. The method further includes: Configure the routing mechanism to route MSI transactions between the subsystem PCIe root controller and the first or second PCIe endpoint to which the SR-IOV PCIe peripheral device virtual function is mapped.
20. The peripheral agent subsystem according to claim 15, wherein: The method further includes: The first PCIe endpoint is configured to present virtual functionality to the third PCIe root controller, and the first peripheral device virtual functionality is mapped to the third virtual functionality.