Zoned Accelerator Embedded Processing

Abstract

In embodiments herein, end-to-end bindings are described to create zones that span across various components in an SoC, such as I / O gateways, processor subsystems, NoCs, storage and data accelerators, and programmable logic. Each zone can be assigned to a different domain controlled by a tenant, such as an external host or software running on the host. In embodiments herein, end-to-end bindings are created between the acceleration engines, I / O gateways, and embedded cores of an SoC. Rather than treating these components as separate monolithic components, bindings divide hardware and memory resources among the components that make up the SoC into various zones. These zones can then have unique bindings for multiple tenants. Bindings can be configured as bridges between components to divide resources into zones and allow tenants of these zones to utilize dedicated resources that are protected from other tenants.

Description

[Technical field]

[0001] SUMMARY OF THE DISCLOSURE Embodiments of the present disclosure generally relate to end-to-end zoning of hardware resources in a system-on-chip (SoC). [Background technology]

[0002] Current acceleration devices, such as Smart Network Interface Cards / Controllers (SmartNICs), contain various components, such as I / O gateways, processor subsystems, Network-on-Chips (NoCs), storage and data accelerators, data processing engines, and programmable logic (PL). In the prior art, these various components in an acceleration device are treated as single, integrated components that simply pass processing or transform data before forwarding it to the next component. These components either interface with each other via a common bus protocol, such as Advanced eXtensible Interface (AXI), or have a protocol bridging function that converts the protocol of one component, such as PCIe, to the protocol of another component, such as AXI.

[0003] When sending data from one component to the next, properties or attributes of the data specific to one component or to a tenant that services a function for that tenant are not propagated to the next component, so that the two components are invisible to each other even if both components may perform a specific function that services the same tenant. In other words, different components in the same accelerator device act independently of each other, even though they may perform operations that are related to each other. Some prior art involves software intermediation (such as a hypervisor or firmware of a component) to reacquire tenant-specific attributes across various components (although bridges may create fuzzy boundaries) so that each time one component processes or receives data from another component, the receiving component can reacquire the tenant properties through the intermediation software. However, this software intermediation introduces overhead, especially when enabling services across components of multiple tenant functions. Furthermore, some tenants may not trust the hypervisor or other software entity that intervenes on their behalf (as in bare metal deployments). Summary of the Invention

[0004] One embodiment is a system on a chip (SoC) including an I / O gateway configured to couple to an external host, and a processor subsystem connected to the I / O gateway and including a processing core and an interconnect communicatively coupled to the processing core, further comprising: circuitry within the I / O gateway, the processing core, and the interconnect partitioned into multiple zones having end-to-end bindings such that each of the multiple zones is isolated from one another, and circuitry within the I / O gateway configured to extract the end-to-end bindings.

[0005] One embodiment described herein is a Smart Network Interface Controller (SmartNIC) including an I / O gateway configured to couple to an external host, and a processor subsystem connected to the I / O gateway, the processor subsystem including a processing core and an interconnect communicatively coupled to the processing core. Further, the circuitry, processing core, and interconnect within the I / O gateway are partitioned into multiple zones having end-to-end bindings such that each of the multiple zones is isolated from one another, and the circuitry within the I / O gateway is configured to extract the end-to-end bindings.

[0006] One embodiment described herein includes a method including: receiving, at an SoC, an encrypted packet assigned to a first zone of a plurality of zones within the SoC; decrypting the encrypted packet; extracting, at an I / O gateway of the SoC, an end-to-end binding from the decrypted packet; adding, at the I / O gateway, a stashing attribute to a transaction corresponding to the decrypted packet based on the end-to-end binding; routing, at an interconnect within the SoC, the transaction based on the stashing attribute to a first processing core within the SoC assigned to the first zone; and processing the transaction at the first processing core. [Brief description of the drawings]

[0007] In a manner in which the above-recited features may be understood in detail, a more particular description briefly summarized above may be made by reference to exemplary implementations, some of which are illustrated in the accompanying drawings. It should be noted, however, that the accompanying drawings illustrate only typical example implementations and therefore should not be considered as limiting the scope thereof. [Figure 1] FIG. 2 illustrates an SoC including a zone with end-to-end binding according to one embodiment. [Diagram 2]FIG. 13 illustrates an example zone binding property type for a PCIe packet according to an embodiment. [Diagram 3] FIG. 1 illustrates an example transaction flow and associated binding attributes according to one embodiment. [Figure 4] 1 illustrates a flow diagram for sending a transaction through a SoC with end-to-end binding according to one embodiment.

[0008] For ease of understanding, wherever possible, identical reference numbers have been used to designate identical elements common to the figures. It is contemplated that elements of one example may be beneficially incorporated in other examples. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Various features are described below with reference to the drawings. It should be noted that the drawings may or may not be drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. It should be noted that the drawings are intended only to facilitate the description of features. They are not intended as an exhaustive description of features or as limitations on the scope of the claims. In addition, the illustrated examples do not necessarily have all aspects or advantages shown. An aspect or advantage described in connection with a particular embodiment is not necessarily limited to that embodiment and can be implemented in any other embodiment, even if not so illustrated or explicitly described.

[0010] Embodiments herein describe end-to-end binding to create domains starting from the host and extending between various components in the SoC, such as I / O gateways, processor subsystems, NoCs, storage and data accelerators, and PLs. That is, the hardware of the SoC is divided into multiple zones, and each zone can be assigned to a different domain controlled by a tenant, such as an external host, or software (e.g., a virtual machine, operating system, or software application) running on the host. That is, a domain includes an external agent, such as an external host, or software running on the host, and one of the zones in the SoC. Embodiments herein create end-to-end bindings between the external agent and the acceleration engines, I / O gateways, and embedded cores in acceleration devices, such as SmartNICs, to create domains. Rather than treating the components of the SoC as separate monolithic components, the binding divides the hardware and memory resources into various zones among the components that make up the acceleration device. These zones can then have bindings specific to their respective external agents (e.g., tenants) to form domains. Bindings can be configured as bridges between components within the SoC, such as processor subsystem bridges, to partition resources into zones and allow tenants of these zones to utilize dedicated resources protected from other external agents assigned to different zones within the SoC. Access to these zones can be controlled using programmable tables in the bridges such that bindings propagate from the I / O gateway to the processor subsystem and further to the remaining components of the acceleration device (e.g., input / output memory management units (IOMMUs), processing cores, caches with designated cache injections, NoCs, memory destinations in the NoCs).In general, the end-to-end binding described herein is applicable to a diverse set of storage, networking, artificial intelligence (AI), and other acceleration devices.

[0011] FIG. 1 illustrates a computing system 100 including a SoC 115 with a zone having end-to-end binding, according to one embodiment. The computing system 100 includes multiple hosts 105 coupled to the SoC 115 (e.g., an integrated circuit). The hosts 105 (e.g., a server) may include processors that execute software 110 and memory. The software 110 may include one or more virtual machines (VMs), an operating system (OS), or software applications (e.g., user applications or tenant applications). The hosts 105 may offload tasks to the SoC 115. For example, the SoC 115 may be a SmartNIC with an accelerator engine, or may be an accelerator device, such as an AI accelerator, a crypto accelerator, a compression accelerator, or a graphics accelerator.

[0012] In embodiments herein, an end-to-end setup is described with either pre-boot configuration options and / or driver configuration options to program multiple components in the SoC 115 over an external agent such that a collection of resources across these components forms a zone. These zones can then provide performance and / or isolation between them, whether that be security isolation from other tenants using the SoC 115, performance isolation from other tenants, and / or error isolation from errant or malicious tenants. Isolation zones are created through bindings between these components, functions within the components, various protocols used to create or signal binding associations with specific zones, or component attributes. The configuration and binding setup described in this section, among others, can provide zoning isolation during runtime or during bare metal deployment without a hypervisor without incurring the overhead of an intervening hypervisor. For example, software running on the embedded processor subsystem of the SoC cannot access attributes of the hardware bus PCIe protocol that have binding attributes (described below). Typically, the only information from the PCIe protocol exposed to the processor subsystem is the read and write data, meaning that providing binding attributes to the processor subsystem software involves encapsulating the entire PCIe packet and forwarding it to the subsystem, but a hardware solution can avoid this overhead entirely.

[0013] In one embodiment, each domain (i.e., a combination of an external agent and a zone in the SoC 115) provides isolation and resources for each host 105, each bare metal OS in the host 105, each VM in the host 105, each application in the host 105, or each external PCIe accelerator card that interacts with a corresponding zone in the SoC 115. Although FIG. 1 illustrates an example transaction flow across four zones, the concepts applied thereto may be similarly applied to creating a single zone or four or more zones using the same binding techniques described herein. Furthermore, while FIG. 1 illustrates an example transaction flow in one direction, from the software 110 in the host 105 to the I / O gateway 120 to the processor subsystem 145 to the NoC 170, and to the memory controller 175, the concepts applied thereto may be similarly applied to transaction flows in different directions or across various agents, for example, from the core 150 to the interconnect 155 to the I / O gateway 120 to the host 105. Furthermore, this concept can be used for intra-SoC communications from cores 150 to interconnect 155, to programmable fabric 160, or to accelerators (not shown) within SoC 115.

[0014] The following discussion uses Figure 1 to illustrate four example domains and describes various components that are either configured to generate the four domains, have the ability to interpret binding attributes, or are programmed with binding attributes. The following list also describes the transaction flow and identification mechanisms that distinguish transactions between one domain and another.

[0015] In one embodiment, there are four foreign agents binding to the four zones shown in FIG. 1 respectively. The bindings between the foreign agents and the zones create four domains. Examples of foreign agents that may be bound to such zones may include: (i) four server hosts 105, (ii) four VMs or four OSes within the hosts 105, (iii) four user / tenant applications running within the hosts 105, or (iv) four PCI peer devices or four peer accelerators (not shown). All such examples may function as foreign agents for the four zones illustrated in FIG. 1.

[0016] An external agent (e.g., host 105, a VM, an OS, a user / tenant application, or a peer device) can initiate a transaction that is routed to I / O gateway 120. I / O gateway 120 acts as a PCIe, Compute Express Link™ (CXL™), or Cache Coherent Interconnect for Accelerators (CCIX) external communication gateway to internal components in SoC 115, such as processor subsystem 145 and programmable fabric 160, including the PL.

[0017] The communication between the host 105 and the I / O gateway 120 is shown using four arrows, each arrow representing a transaction associated with a different domain. In some embodiments, these transactions may be PCIe or CXL transactions. Transactions issued by an external agent include binding attributes that are used to identify the domain to which the transaction belongs. These binding attributes are described below in subsequent figures.

[0018] In one embodiment, bindings can be created between four domains and four unique encrypted PCIe traffic streams. A first option is to prevent any domain from accessing the PCIe integrity and data encryption (IDE) traffic keys and properties. In this option, the PCIe IDE transaction layer packet (TLP) prefix can include attributes of the link IDE stream to create the bindings.

[0019] A second option is to have bindings between the selectable IDE streams and the domains such that access to the keys and properties of PCIe IDE encrypted traffic is restricted to selectable IDE streams that have bindings for those domains. In the four domain example of FIG. 1, there can be either 1:1 bindings for a total of four selectable IDE streams, assuming PCIe controller 125 supports 32 selectable IDE streams, or there can be 4:1 bindings between the 32 selectable IDE streams and the four domains. In this option, the PCIe IDE attribute in the TLP prefix portion of the PCIe packet received from host 105 includes a binding attribute that identifies the selectable IDE stream.

[0020] A third option, which may be applied in addition to or instead of the second option, is to form bindings between the four domains in a transaction and a requestor ID (RID) attribute such that access to the keys and properties of the PCIe IDE encrypted traffic is restricted not only to the selectable IDE streams that have bindings for a particular domain, but also to a particular bus:device.function (BDF) value or set of BDF values ​​or set of function numbers. In the four domain example of FIG. 1, assuming that the PCIe controller 125 is a Gen5x16 PCIe controller supporting 32 selectable IDE streams and also 32 physical functions (PFs), there may be a 1:1 binding between the four PFs and the four domains, or a 4:1 binding between the 32 PFs and the four domains. In the third option, the PCIe requestor ID, which may be included in the header portion of a PCIe packet received from the host 105, includes a binding attribute that identifies the PF. And the 32 PFs may have 1:1 bindings with the 32 selectable IDE streams, regardless of the PF-to-domain bindings.

[0021] Next, a composable scalable interconnect (CSI) bridge 130 receives the PCIe IDE decoded packet from the PCIe controller 125. The CSI bridge 130 uses the collection of PCIe packet attributes along with the CSI attributes to then extract domain-specific properties based on pre-configured bindings in the CSI bridge 130 to determine the domain or zone. Additionally, the CSI bridge 130 determines, based in part on the match with the CSI attributes, whether the PCIe transaction (and its metadata) should be routed to a host access handler (HAH) 135 or whether the transaction can skip HAH 135 and proceed directly to the processor subsystem bridge 140.

[0022] In one embodiment, the CSI bridge 130 extracts a source property for binding with a zone in the SoC. The source property can be one or more attributes of an external agent connection. The CSI bridge 130 uses the source property to identify the source of the PCI packet and binds the data received from that source to the corresponding zone. In a situation where transactions for four domains are received on multiple PCIe controllers 125, the CSI_ID is one such source property that communicates the PCIe controller connection. This includes when transactions are received from four hosts 105 and when transactions are received from four separate PCIe peer / accelerator sources. Thus, the CSI_ID, or PCIe controller number, is used to identify the domains:<PCIe Controller #> This provides one criterion for extracting source property bindings that have

[0023] Alternatively, as another example of a source property, if transactions for four domains are received at a single PCIe controller 125, the transactions may be sent to the following domains: <bdf>, four function numbers, or four requester ID groups (e.g., PFs and their associated virtual functions (VFs)).<PCIe Controller #、BDF> A unique combination of may also form the basis for extracting a source property binding assigned to a domain. A multi-core host, various cores, or various applications communicatively coupled to a PCIe device may express source properties by transmitting unique BDF values ​​in a PCIe TLP. Thus, the BDF values ​​may be selected as binding properties for the host's cores, VMs, or applications that are external agents of the domain.

[0024] After extracting the source properties of the domain, the CSI bridge 130 extracts the destination properties for binding with the corresponding zone of the SoC. In one embodiment, the destination property is a routing attribute such as a destination address, or a range of destination addresses, such as PCIe BAR mapping memory. For example, in the case of addressless transactions such as PCIe Vendor Defined Messages (VDM), the destination property can be a destination BDF, or a range of BDFs, or multiple functions with a common Bus#:Device#.

[0025] Using the destination property, the CSI bridge 130 binds the destination of the PCIe packet to the same domain as the source. Based on the address of the request TLP in the header of the received PCIe packet, the CSI bridge 130 then determines whether the source address is the same as the destination PCIe base address register (BAR) for the function.<Function#,PCIeBAR#,Aperture> or<BDF,PCIeBAR#,Aperture> The destination function for that domain, or zone, may be determined based on a programmed aperture in the memory space of that PCIe BAR (e.g., the portion of the BAR that corresponds to the destination function) that matches the PCIe BAR's address. An example of a destination function may be a particular acceleration function, such as, for example, a cryptographic function or an artificial intelligence function. For example, one tenant may use the cryptographic acceleration function of the SoC and another tenant may use the machine learning acceleration function of the SoC, but in different zones. Furthermore, the destination function may be a predefined chain of acceleration functions, where one tenant may use the cryptographic acceleration function first and then the SoC's restoration acceleration function, and another tenant may use the restoration acceleration function first and then the machine learning acceleration function.

[0026] Rather than an input address identifying the aperture of the BAR to which it applies, the destination function for a domain / zone may instead be based on a source address offset that matches an address offset within the aperture associated with a particular function.<PCIeController#,BDF,PCIeBAR,Aperture,Offset> A unique combination of can form the criteria for extracting source property bindings with a zone, along with different offsets that identify different zones, or different functions of the same zone.

[0027] In addition to using a match of the PCIe BAR and / or source address offset, the Process Address Space ID (PASID) in the TLP prefix of the received PCIe packet can determine the destination function of a zone in cases where different PASID values ​​identify different zones or different functions of the same zone.

[0028] In one embodiment, the binding attributes may be further determined by the HAH 135. The HAH 135 may receive, along with the PCIe packet, from the CSI bridge 130:<PCIe Controller #,BDF,PCIe BAR,Aperture,Offset> The binding properties may be received in the original form, or in an encoded form, or as some relevant subset of the binding properties. For example, if the address offset is not a binding property of CSI bridge 130 but is a binding property of HAH 135, CSI bridge 130 propagates to HAH 135 a set of attributes that HAH 135 uses for further binding to a particular zone, or alternatively, for further binding to a specific function (e.g., a specific storage crypto function) to be performed for a zone already identified by CSI bridge 130. HAH 135 can then use the additional PCIe packet attributes to further extract zone-specific properties based on the pre-configured bindings in HAH 135 in conjunction with the attributes propagated by CSI bridge 130 to continue to identify a particular zone, or a function within a zone, if that zone has not already been identified in a previous step.

[0029] These previous steps may include the HAH 135 determining, based on the address offset of the request TLP within the target aperture, the specific service that the external agent is attempting to access, as well as the specific zone within the SoC 115 to which the external agent is binding. Alternatively, the HAH 135 may also use a transaction processing hint (TPH) within the TLP prefix portion of the received PCIe packet to determine the specific zone within the SoC 115 to which the external agent is binding. When using TPH in binding, even if the complete TPH namespace is available across various sources, there may be a limitation that only unique TPH values ​​can be provided and aliasing is not allowed. In that case, the TPH is uniquely provisioned across the four hosts 105, four VMs / OS, four user / tenant applications, or four PCIe peer devices / accelerators to ensure that the same TPH value is not issued by multiple external agents.

[0030] Additionally, these previous stages may also include embodiments in which a requestor ID and CSI_ID combination that is part of the PCIe header is associated with a zone of the SoC. A particular set of requestor ID and CSI_ID combinations may be further reduced to a smaller set of Extended Multi-Tenant ID (XMTI) binding attributes. While the HAH 135 itself may not use XMTI binding attributes, components further downstream in the SoC 115 may use the XMTI binding attributes to achieve isolation and access control.

[0031] The processor subsystem bridge 140 can receive transactions from the HAH 135 or directly from the CSI bridge 130 and can further add routing and stashing attributes before propagating the transaction to the processor subsystem 145. In one embodiment, the processor subsystem bridge 140 uses the TPH to reference a programmable lookup table, or indexed table, to obtain additional zone binding attributes to be included with transactions routed from the I / O gateway 120 to the coherent interconnect 155 in the processor subsystem 145. This programmable table can be programmed such that each TPH value can indicate a specific set of cache injection (e.g., stashing) attributes, such as a stash node ID (StashNID) and a logical processor ID (StashLPID), that can uniquely identify a stash destination within the processor subsystem 145. Transactions to an aperture for a particular processor subsystem may not be enabled to have a stashing attribute and command type (e.g., a transaction mapped to a first memory controller 175 is enabled to have stashing, but a transaction mapped to a second memory controller 175 is not (enabled to not have stashing)), or alternatively, a particular TPH value may have a lookup result indicating that the stashing attribute does not require the addition of a stash attribute before forwarding to the processor subsystem 145.

[0032] 1 illustrates the transfer of four transactions from I / O gateway 120 to coherent interconnect 155. Optionally, these transactions may flow through the IOMMU for access control checks to the local processor subsystem 145 or to perform address translation from external BAR-mapped PCIe addresses to the address map of processor subsystem 145. Additionally, the address map may be considered to populate page tables accessed by the IOMMU and the address map of processor subsystem 145. As shown, circuitry within coherent interconnect 155, such as snoop filters and last level caches (LLCs), are assigned to four zones to provide isolation for transactions as they flow through interconnect 155.

[0033] In the case where the external agent is a four-host 105, address-based host-initiated MMIO transactions may have already undergone translation from the host MMU to the PCIe address space. Upon arrival at the I / O gateway 120, the request address matching a BAR-mapped memory in the PCIe physical address space becomes the property used to direct the MMIO transaction to the processor subsystem 145. Thus, the page table translation occurring in the IOMMU performs a translation from the PCIe address space to the processor subsystem's physical address space, rather than a typical processor subsystem's virtual address space to physical address space translation. The processor subsystem's physical address space may then be a function of the SoC's DDR and peripheral physical address map provided to the boot CPU (e.g., one of the cores 150) via the device tree, while stashing addresses that are in the coherent address space may be restricted to main memory (DDR, or on-chip memory (OCM)).

[0034] Similar considerations may apply when the external agents are four external PCIe peer devices or accelerators that initiate transactions using PCIe physical address space, an intervening PCIe switch directs the transactions to SoC 115 for BAR-mapped memory, and I / O gateway 120 functions as a PCIe gateway.

[0035] After flowing through the IOMMU, coherent interconnect 155 routes the transaction to one of four processor zones. That is, the resources of processor subsystem 145 can be divided into four zones. If the stashing attribute is enabled, interconnect 155 can route the transaction to a more granular destination within each of the four processor zones.

[0036] The transaction may be routed to a processing zone defined by processing cores 150, which in this example are divided into four clusters: core 150A in a first processor zone, core 150B in a second processor zone, core 150C in a third processor zone, and core 150D in a fourth processor zone. The data portion of the transaction forwarded to core 150 includes metadata used in downstream communication.

[0037] If the stashing attributes StashNID and StashLPID indicate a particular L2 cache stashing destination in one of the cores 150 (and assuming each processor zone has a cluster of four cores 150), then the stash can be targeted to one of the four L2s per processor zone for a total of 16 L2 stash destinations. In one embodiment, this provides the lowest possible access latency for cache injection data associated with a transaction issued by an external agent, or for cache injection data plus metadata associated with the transaction after it has been received from I / O gateway 120.

[0038] If the stashing attributes StashNID and StashLPID indicate a particular L3 cache stashing destination (again, assuming each processor zone has a cluster of four cores 150), the stash can be targeted to the L3 stash destination in each of the four processor zones. If the L2 stash operation is not accepted, the L3 cache for that zone can allocate the data of the L2 stash operation instead to provide the next best access latency for the cache injection data.

[0039] If a write stash operation cannot be allocated to the target stash destination for that zone, the LLC for that zone can attempt to allocate the write to a cache instead. In one embodiment, coherent interconnect 155 includes four LLC slices with four associated home nodes. Each home node is then mapped to a pair of memory controllers 175, and if a stash write to the four zones cannot be allocated to any of the four L2 or L3 caches for those zones, an allocation to the corresponding LLC slice is attempted. However, if cache injection or allocation is not possible, the write operation is completed to the DDR memory destination for the zone using memory controller 175.

[0040] At the system level, to consolidate the pairwise relationships between the external domain and the processor zones executing on the four clusters of cores 150 to form a single domain, in one embodiment, a set of four transaction-initiating external agents (e.g., host 105, software 110, or PCIe-accelerator domain) includes a corresponding set of four SoC zones, and a software driver, application, or OS with multiple applications executing in each of the four embedded processing zones executes on the cluster of cores 150. Furthermore, as transactions flow through various address domains within the SoC, software communicating between the external domain and the four embedded processing zones executing on cores 150 can consider address space translation and thus source address domain to destination address domain bindings, including intermediate address domains, during its setup. For example, the software can handle address space transitions from four host domain virtual address spaces to four host domain physical address spaces, transitions from the four host domain physical address spaces to either the four domain PCIe address spaces in a multi-host case or a common PCIe address space for one of the four domains in a single-host case, and transitions from the four domain PCIe address spaces to the processor subsystem zone physical address space.

[0041] In one embodiment, communication of transactions from host 105 to CSI bridge 130 allows PCIe IDE encrypted data communication on a per-domain basis. However, transfer of transactions from CSI bridge 130 to processor subsystem 145 may include decrypted data. If either security or isolation is required between transactions downstream of CSI bridge 130, cryptographic extensions of core 150 may be enabled to use a common cipher, such as Advanced Encryption Standard (AES), between I / O gateway 120 and core 150.

[0042] Additionally, while the above-described relationship was a pairwise relationship between an external agent and an embedded processing zone executing on core 150, the four domains are not limited to a pairwise relationship and can be extended to other components within the system, including dividing the programmable fabric into four PL zones and four accelerator zones, such that core 150, the L2 cache, and the L3 cache communicate with the four PL zones and four accelerator zones. 2 is a diagram illustrating zone binding property types of a PCIe packet, according to an embodiment. Specifically, FIG. 2 shows one PCIe packet 205 of a domain transaction 210 that may be received from an external agent bound to one of the four zones illustrated in FIG. 1. For example, the external agent may be four hosts 105, four VMs / OS, four user / tenant applications, four PCIe peer devices, accelerators, etc. Such an external agent may initiate the domain transaction 210 that is received at an I / O gateway of an SoC (e.g., I / O gateway 120 of SoC 115 of FIG. 1). The PCIe packet 205 includes a TLP prefix, a header, and a data payload. As discussed above, the PCIe packet 205 may include a binding attribute for binding the packet 205 to a particular zone of the SoC. In one embodiment, the IDE field of the TLP prefix may include an attribute of the link IDE stream for creating the binding, i.e., for identifying which of the four domains the PCIe packet 205 (and the external agent that created the packet 205) belongs to. In another embodiment, the PASID field of the TLP prefix may be used to bind the packet 205 to one of the four zones of the SoC. For example, the PASID field may indicate a destination function of a domain, where different PASID values ​​identify different domains or different functions of the same domain. In another embodiment, the TPH field of the TLP prefix may be used to create the binding. The use of TPH in the binding may be limited in that only unique TPH values ​​can be provided and aliasing is not allowed, even if the complete TPH namespace is available to different sources (i.e., different external agents). In that case, the TPH is uniquely provisioned across the four foreign agents and the same TPH value cannot be issued by multiple foreign agents.

[0043] Instead of using the TLP prefix field as the binding, the requestor ID field of the header can be used. The requestor ID field can contain a BDF value that indicates to the I / O gateway the zone to which the packet 205 should be bound. In another embodiment, the address field of the header can be used to create the binding. The address field indicates the destination PCIe BAR of the function, and then the source address.<Function#、PCIe BAR#、Aperture> or<BDF、PCIe BAR#、Aperture> The PCIe BAR may include a programmed aperture in the memory space of the PCIe BAR that corresponds to the

[0044] Additionally, in one embodiment, the data in the data payload of the packet 205 may include binding attributes used to create the binding. For example, the data payload may include metadata indicating the zone to which the packet 205 corresponds.

[0045] A more detailed description of various embodiments relating to binding a domain transaction 210 received from an external actor using various portions of the PCIe packet 205 illustrated in FIG. 2 has been provided above and will not be repeated here.

[0046] FIG. 3 is a diagram illustrating transaction flows and associated binding attributes according to one embodiment. That is, unlike FIG. 2, which shows the flow of domain transactions from an external agent to the SoC, FIG. 3 illustrates the flow of domain transactions once they enter the SoC (e.g., SoC 115 of FIG. 1). That is, FIG. 3 shows the various transformations and operations that are performed by various components within the SoC (e.g., CSI bridge 130, HAH 135, processor subsystem bridge 140, and coherent interconnect 155) to create end-to-end bindings of external actors to zones that span the SoC. In this way, rather than these components functioning as separate monolithic components, the components bind together to create multiple zones.

[0047] Packet 205 is received at PCIe controller 125. As shown, PCIe controller 125 is logically divided into four zones. This may be done using a single PCIe controller 125 shared by the four zones, or there may be a separate physical controller 125 for each zone. In one embodiment, the PCIe controller decrypts packet 205. That is, packets 205 of four domain transactions may be encrypted such that packets 205 of one domain cannot be read by an external agent that does not belong to the same domain. However, in other embodiments, the data may remain encrypted within the SoC.

[0048] The PCIe controller 125 forwards the packet 205 to the CSI bridge 130, which in turn can be logically or physically assigned to the four zones in the SoC. In one embodiment, the CSI bridge 130 first extracts the source property for binding to a domain. That is, the CSI bridge 130 identifies the source of the PCIe packet 205 and binds the packet 205 to the domain corresponding to the source. For example, the CSI bridge 130 can be configured at boot time to know which external agent (e.g., source) is bound to which domain. Thus, when the CSI bridge 130 receives a packet from that external agent, it can identify the domain to bind the packet 205 to. The CSI bridge 130 can use any of the information shown in FIG. 2 to identify the source and domain of the received packet 205.

[0049] After extracting the source property of the domain, the CSI bridge 130 extracts the destination property for binding with a zone of the SoC. Based on the address of the request TLP in the header of the received PCIe packet 205, the CSI bridge 130 can determine the destination function of the zone. This destination function can be identified by the CSI bridge 130 using many different attributes, such as the PCIe controller number (PCIe#), BDF, PCIe BAR, full address (Addr), address offset (Addr Offset), and PASID.

[0050] Optionally, the binding attributes can be further evaluated by HAH 135. HAH 135 can receive the aperture number (aperture#), address offset, function type of the destination function, or other binding properties from CSI bridge 130 along with PCIe packet 205, either in the original form, or in the encoded form, or as some relevant subset of the binding properties. HAH 135 can then use the additional PCIe packet attributes, in conjunction with the attributes propagated by CSI bridge 135, to further extract domain-specific properties based on pre-configured bindings at HAH 135 to continue identifying a particular zone, or a function within a zone, if that zone has not already been identified in a previous step.

[0051] Processor subsystem bridge 140 can receive transactions from HAH 135 or directly from CSI bridge 130 and can add routing and stashing attributes to the transactions before propagating them to processor subsystem 145. As shown, processor subsystem bridge 140 can identify the stashing attributes through a programmable table using information in the TPH field or aperture# of packet 205.

[0052] Coherent interconnect 155 receives transactions from processor subsystem bridge 140 and further uses the stash attribute to forward the transactions to a core, an L2 cache within a core, an L3 cache, or a memory controller for storage in main memory.

[0053] As transactions traverse PCIe controller 125, CSI bridge 130, HAH 135, processor subsystem bridge 140, and coherent interconnect 155, these hardware components generate end-to-end bindings for transactions in each zone to provide dedicated hardware resources for each zone. That is, the circuitry in PCIe controller 125, CSI bridge 130, HAH 135, and processor subsystem bridge 140 can be logically and / or physically partitioned among the four zones such that transactions in one zone cannot use resources allocated to a different zone. Generating these bindings in hardware advantageously avoids the overhead associated with software-defined zones in an SoC.

[0054] 4 is a flow diagram of a method 400 for sending transactions through an SoC with end-to-end binding, according to one embodiment. At block 405, the SoC receives encrypted packets from an external agent that are assigned to zones of the SoC. That is, the SoC is pre-configured to establish multiple zones, each with end-to-end bindings, from the external agent through various components in the SoC (e.g., I / O gateways, coherent interconnect, cores, processor subsystems, PL fabric, etc.). In one embodiment, these bindings are controlled using hardware in the SoC (e.g., without relying on a hypervisor or OS in the SoC to maintain domains).

[0055] The SoC can receive packets (e.g., PCIe packets) from different types of external agents. In other words, various types of external agents can be bound to zones established in the SoC. Examples of the external agents can include (i) a server host, (ii) a VM or OS in the host, (iii) a user / tenant application running in the host 105, or (iv) a PCI peer device or four peer accelerators (not shown). In one embodiment, the packets are received at an I / O gateway of the SoC.

[0056] At block 410, the SoC decrypts the packet. In one embodiment, a PCIe controller (e.g., PCIe controller 125 of FIG. 1) of the I / O gateway decrypts the packet. The above description describes three implementations or options for binding PCIe traffic streams to various zones established within the SoC. Briefly, these options include (i) not allowing any domain to access the keys and properties of the PCIe IDE traffic, (ii) having bindings between selectable IDE streams and domains such that access to the keys and properties of the PCIe IDE encrypted traffic is restricted to selectable IDE streams that have bindings of those domains, and / or (iii) forming bindings between domains and requestor ID attributes in a transaction such that access to the keys and properties of the PCIe IDE encrypted traffic is restricted not only to selectable IDE streams that have bindings of a particular domain, but also to a particular BDF value or set of BDF values ​​or set of function numbers.

[0057] In block 411, circuitry in the I / O gateway extracts the end-to-end bindings. This extraction can be performed by the CSI bridge, the HAH, the processor subsystem bridge, and combinations thereof, all of which are circuitry within the I / O gateway.

[0058] Optionally, at block 415, the SoC extracts source properties for binding with the domain. In one embodiment, a CSI bridge (e.g., CSI bridge 130 of FIG. 1) identifies the source of the PCI packet and binds data received from that source to the corresponding domain. In a situation where transactions for a domain in the SoC are received at multiple PCIe controllers, the CSI_ID is one such source property that communicates the PCIe controller connection. This includes the case where a transaction is received from four hosts and the case where a transaction is received from four separate PCIe peer / accelerator sources. Thus, the CSI_ID, or PCIe controller number, is used to bind the domain:<PCIe Controller #> This provides one criterion for extracting source property bindings that have

[0059] Alternatively, if transactions for a domain of an SoC are received by a single PCIe controller, the transactions are <bdf>may represent a distinct requestor ID (BDF) source, a function number, or a requestor ID group (e.g., PFs and their associated VFs) for extracting source property bindings with<PCIe Controller #、BDF> The unique combination of may also form the criteria for extracting source property bindings assigned to a domain.

[0060] Optionally, in block 420, the SoC extracts the destination property for binding with a zone of the SoC. In one embodiment, the CSI bridge binds the destination of the PCIe packet to the same domain as the source. Based on the address of the request TLP in the header of the received PCIe packet, the CSI bridge then determines the destination PCIe BAR for the function, and<Function#,PCIe BAR#、Aperture> or<BDF、PCIe BAR#、Aperture> Based on the programmed aperture in the memory space of that PCIe BAR that matches the destination function for that zone, the destination function for that zone can be determined.

[0061] Similarly, the destination function for a zone may be based on a source address offset that matches an address offset within an aperture associated with a particular function.<PCIeController#,BDF,PCIeBAR,Aperture,Offset> A unique combination of can form the criteria for extracting source property bindings with a zone, along with different offsets that identify different zones, or different functions of the same zone.

[0062] In addition to using a match of the PCIe BAR and / or source address offset, the PASID in the TLP prefix of the received PCIe packet can determine the destination function of a zone if different PASID values ​​identify different zones or different functions of the same zone.

[0063] Optionally, at block 425, the HAH (e.g., HAH 135 of FIG. 1) identifies binding attributes. That is, in one embodiment, the binding attributes can be further determined by HAH 135. HAH 135 can receive the binding properties from the CSI bridge along with the PCIe packet in original form, or in encoded form, or as some relevant subset of the binding properties. For example, if the address offset is not a binding property of the CSI bridge but is a binding property of the HAH, the CSI bridge propagates a set of attributes to the HAH that the HAH uses to further bind to a particular zone. The HAH can then use the additional PCIe packet attributes, in conjunction with the attributes propagated by the CSI bridge, to further extract domain-specific properties based on the pre-configured bindings at the HAH to continue to identify a particular zone, or a function within a zone, if that zone has not yet been identified in a previous step.

[0064] At block 430, the SoC adds routing stashing attributes to the transaction corresponding to the packet based on the end-to-end binding. In one embodiment, the routing stashing attributes are added by a processor subsystem bridge in the I / O gateway (e.g., processor subsystem bridge 140 of FIG. 1). In one embodiment, the stashing attributes are identified by using a CSI bridge or HAH to extract the binding at blocks 415, 420, and / or 425, as described above. However, in another embodiment, the stashing attributes can instead be identified directly without using a CSI bridge and HAH. For example, FIG. 3 illustrates that when a TPH is used to provide the binding, the processor subsystem bridge 140 can directly determine the stashing attributes using the TPH field of the PCIe packet 205, without the need for a CSI bridge and HAH to extract the binding and identify the domain / zone corresponding to the transaction. In this scenario, blocks 415, 425, and 425 can be omitted from method 400.

[0065] Thus, method 400 illustrates using circuitry in an I / O gateway (whether that circuitry is a CSI bridge, a HAH, a processor subsystem bridge, or a combination thereof) to extract end-to-end bindings from data in received PCIe packets.

[0066] In one embodiment, the processor subsystem bridge uses the TPH to reference a programmable lookup table, or indexed table, to obtain additional domain binding attributes to be included with transactions routed from the I / O gateway to a coherent interconnect in the processor subsystem (e.g., coherent interconnect 155 in processor subsystem 145 of FIG. 1). This lookup table can be programmed such that each TPH value can indicate a specific set of cache injection (e.g., stashing) attributes, such as StashNID and StashLPID, that can uniquely identify a stash destination in the processor subsystem. Transactions to apertures destined for a particular processor subsystem may not be enabled to have stashing attributes and command types, or alternatively, a particular TPH value may have a lookup result that indicates that the stashing attributes do not require the addition of stashing attributes before forwarding to the processor subsystem.

[0067] At block 435, the interconnect of the processor subsystem routes the transaction using the stashing attribute. In one embodiment, after passing through the IOMMU in the interconnect, the interconnect routes the transaction to one of the processor core zones. This may include routing the transaction to a particular L2 cache in one of the cores of the corresponding core zone or to an L3 cache of the corresponding core zone.

[0068] At block 440, the corresponding core zone processes the transaction, which may include using a processing core (e.g., one of the cores 150 in FIG. 1) to perform an operation on the data in the transaction or to store the data in an L2 or L3 cache. Additionally, the interconnect may route the transaction outside of the processor subsystem, such as to a NoC, a memory controller, or a programmable fabric.

[0069] In the foregoing, reference is made to the embodiments presented in this disclosure. However, the scope of the disclosure is not limited to the specific described embodiments. Instead, any combination of the described features and elements, whether related to different embodiments or not, is contemplated for implementing and practicing the contemplated embodiments. Furthermore, the embodiments disclosed herein may achieve other possible solutions or advantages over the prior art, but whether or not a particular advantage is achieved by a given embodiment does not limit the scope of the disclosure. Thus, the above-mentioned aspects, features, embodiments, and advantages are merely illustrative and are not considered elements or limitations of the appended claims unless expressly recited in the claims.

[0070] As will be appreciated by one of ordinary skill in the art, embodiments disclosed herein may be embodied as a system, method, or computer program product. Accordingly, aspects may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, microcode, etc.), or an embodiment combining software and hardware aspects, all of which may be referred to generally herein as a "circuit," "module," or "system." Furthermore, aspects may take the form of a computer program product embodied in one or more computer-readable medium(s) having computer-readable program code embodied therein.

[0071] Any combination of one or more computer readable media may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer readable storage media include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this specification, a computer readable storage medium is any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0072] A computer-readable signal medium may include a propagated data signal in which computer-readable program code is embodied, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electromagnetic, optical, or any suitable combination thereof. A computer-readable signal medium is not a computer-readable storage medium but may be any computer-readable medium that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0073] The program code embodied on the computer readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, fiber optic cable, RF, etc., or any suitable combination of the foregoing.

[0074] Computer program code for carrying out operations of aspects of the present disclosure may be written in any combination of one or more programming languages, including, for example, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The program code may execute completely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0075] Aspects of the present disclosure are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments presented in the present disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, a special purpose computer, or other programmable data processing apparatus such that the instructions, executed via a processor of the computer or other programmable data processing apparatus, cause a machine to create means for implementing the functions / acts specified in the blocks of the flowchart and / or block diagrams.

[0076] These computer program instructions may also be stored on a computer-readable storage medium, where the instructions stored on the computer-readable storage medium can direct a computer, programmable data processing apparatus, and / or other device to function in a particular manner to produce an article of manufacture including instructions that implement aspects of the functions / acts specified in the flowchart and / or block diagram blocks.

[0077] Computer program instructions may also be loaded into a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be executed on the computer, other programmable apparatus, or other device to generate a computer-implemented process, such that the instructions executing on the computer or other programmable apparatus provide a process for implementing the functions / acts specified in the flowchart and / or block diagram blocks.

[0078] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may occur out of the order described in the figures. For example, two blocks shown in succession may in fact be executed substantially simultaneously, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flow diagram illustrations, as well as combinations of blocks in the block diagrams and / or flow diagram illustrations, may be executed by a dedicated hardware-based system that performs the specified functions or acts, or a combination of dedicated hardware and computer instructions.

[0079] While the above is directed to particular examples, other and further examples may be devised without departing from the basic scope thereof, which is determined by the following claims.< / bdf> < / bdf>

Claims

1. A system on chip (SoC), comprising: an I / O gateway configured to couple to an external host; A processor subsystem connected to the I / O gateway, A processing core; a processor subsystem including the processing cores and an interconnect communicatively coupled to the processing cores; A SoC, wherein circuitry in the I / O gateway, the processing cores, and the interconnect are partitioned into a plurality of zones having end-to-end bindings such that each of the zones is isolated from the others, and the circuitry in the I / O gateway is configured to extract the end-to-end bindings.

2. The SoC of claim 1 , wherein a first cluster of the processing cores is assigned to a first zone of the plurality of zones and a second cluster of the processing cores is assigned to a second zone of the plurality of zones.

3. The I / O gateway includes: decrypting packets received from the external host; and extracting a source property for binding the packet to one of the plurality of zones; extracting a destination property for binding the packet to one of the plurality of zones; and adding a routing-stashing attribute to a transaction corresponding to the packet.

4. The SoC of claim 3 , wherein the interconnect is configured to use the stashing attribute to route the transaction to one of the processing cores.

5. 5. The SoC of claim 4, wherein extracting the end-to-end bindings is performed by at least one of a Composable Scalable Interconnect (CSI) bridge, a Host Access Handler (HAH), or a processor subsystem bridge in the I / O gateway.

6. The SoC of claim 1 , wherein the end-to-end binding between the I / O gateway, the interconnect, and the processing cores is performed without the use of a hypervisor or an operating system.

7. A smart network interface controller (Smart NIC), an I / O gateway configured to couple to an external host; A processor subsystem connected to the I / O gateway, A processing core; a processor subsystem including the processing cores and an interconnect communicatively coupled to the processing cores; A Smart NIC, wherein circuitry in the I / O gateway, the processing cores, and the interconnect are divided into a plurality of zones having end-to-end bindings such that each of the zones is isolated from the others, and the circuitry in the I / O gateway is configured to extract the end-to-end bindings.

8. The Smart NIC of claim 7 , wherein a first cluster of said processing cores is assigned to a first domain of said plurality of zones and a second cluster of said processing cores is assigned to a second domain of said plurality of zones.

9. memory controllers, each of the memory controllers being assigned to one of the plurality of zones; The SoC of claim 1 or the Smart NIC of claim 7, further comprising: a network-on-chip (NoC) configured to communicatively couple the memory controller to the interconnect.

10. The I / O gateway includes: decrypting packets received from the external host; and extracting a source property for binding the packet to one of the plurality of zones; extracting a destination property for binding the packet to one of the plurality of zones; and adding a routing stash attribute to a transaction corresponding to the packet.

11. The Smart NIC of claim 10 , wherein the interconnect is configured to use the stashing attribute to route the transaction to one of the processing cores.

12. 12. The Smart NIC of claim 11, wherein extracting the end-to-end bindings is performed by at least one of a Composable Scalable Interconnect (CSI) bridge, a Host Access Handler (HAH), or a Processor Subsystem bridge in the I / O gateway.

13. The Smart NIC of claim 7 , wherein the end-to-end binding between the I / O gateway, the interconnect, and the processing cores is performed without the use of a hypervisor or an operating system.

14. 1. A method comprising: receiving, at a SoC, an encrypted packet assigned to a first zone of a plurality of zones within the SoC; decrypting the encrypted packet; and extracting end-to-end bindings from the decrypted packets at an I / O gateway of the SoC; and adding, at the I / O gateway, a stash attribute to a transaction corresponding to the decrypted packet based on the end-to-end binding; Routing, in an interconnect within the SoC, the transaction based on the stashing attribute to a first processing core within the SoC assigned to the first zone; processing the transaction in the first processing core.

15. 15. The method of claim 14, wherein the I / O gateway circuitry, the interconnect, and the processing cores of the SoC are partitioned into a plurality of zones having end-to-end bindings such that each of the zones is isolated from one another.

Images