Tunneling of peripheral bus protocol traffic over consistent fabric and inter-chip or inter-die bus

Abstract

The invention relates to tunneling of peripheral bus protocol traffic over a consistent fabric and an inter-chip or inter-die bus. A peripheral device includes: a master chip, the master chip including a coherent interconnect; one or more first peripheral bus modules; an inter-chip (C2C) or inter-die (D2D) interface; and one or more tunnel adapters. A coherency interconnect is used to connect electronic components of a peripheral device according to a coherency interconnect protocol. The one or more first peripheral bus modules are configured to communicate according to a peripheral bus protocol. The C2C or D2D interface is to communicate with an auxiliary chip comprising one or more second peripheral bus modules over a C2C or D2D bus. The one or more tunnel adapters are configured to transmit peripheral bus packets of a peripheral bus protocol between the first and second peripheral bus modules over the coherency interconnect and the C2C or D2D bus by transitioning between the peripheral bus packets, messages of the coherency interconnect protocol, and C2C or D2D bus messages.

Description

TECHNICAL FIELD

[0001] The present invention relates generally to computing and communication systems, and more particularly to tunneling peripheral bus communications over an interconnect fabric and chip-to-chip (C2C) or die-to-die (D2D) buses in peripheral devices. BACKGROUND

[0002] Computing systems often include peripheral devices connected to a host over a peripheral bus. The peripheral devices can include, for example, network adapters, storage devices, accelerators, and graphics processing units (GPUs). The peripheral bus (also referred to as a system bus) can include, for example, Peripheral Component Interconnect Express (PCIe), Advanced eXtensible Interface (AXI), Compute Express Link (CXL), Nvlink, or Nvlink chip-to-chip (Nvlink-C2C). SUMMARY

[0003] Embodiments described herein provide a peripheral device including a primary chip. The primary chip includes a coherent interconnect, one or more first peripheral bus modules, a chip-to-chip (C2C) or die-to-die (D2D) interface, and one or more tunnel adapters. The coherent interconnect is to connect electronic components of the peripheral device according to a coherent interconnect protocol. The one or more first peripheral bus modules are to communicate according to a peripheral bus protocol. The C2C or D2D interface is to communicate with a secondary chip including one or more second peripheral bus modules over a C2C or D2D bus. The one or more tunnel adapters are to transmit peripheral bus packets of the peripheral bus protocol between the first and second peripheral bus modules over the coherent interconnect and the C2C or D2D bus by converting between the peripheral bus packets, messages of the coherent interconnect protocol, and C2C or D2D bus messages.

[0004] In some embodiments, a given tunnel adapter is to (i) receive one or more messages of the coherent interconnect protocol from the coherent interconnect carrying one or more peripheral bus packets, (ii) convert the one or more messages of the coherent interconnect protocol into one or more of C2C or D2D bus messages, and (iii) send the one or more of the C2C or D2D bus messages to the C2C or D2D bus over the C2C or D2D interface.

[0005] In some embodiments, a given tunnel adapter is to (i) receive one or more of C2C or D2D bus messages from the C2C or D2D bus over the C2C or D2D interface carrying one or more peripheral bus packets, (ii) convert the one or more of the C2C or D2D bus messages into one or more messages of the coherent interconnect protocol, and (iii) send the one or more messages of the coherent interconnect protocol to the coherent interconnect.

[0006] In one embodiment, a given one of the first peripheral bus modules is configured to communicate with a host over a peripheral bus in accordance with a peripheral bus protocol. In one disclosed embodiment, a data size of a message of the coherent interconnect protocol is smaller than a data size of a peripheral bus packet, and a given one of the tunnel adapters is configured to convert one peripheral bus packet into a plurality of messages. In one embodiment, a data size of a message of the coherent interconnect protocol is smaller than a data size of a peripheral bus packet, and a given one of the tunnel adapters is configured to identify a plurality of messages corresponding to a peripheral bus packet and reconstruct the peripheral bus packet from the identified plurality of messages.

[0007] In one embodiment, the coherent interconnect protocol does not guarantee unconditional in-order delivery of messages, and a given one of the tunnel adapters is configured to select two or more messages and cause the coherent interconnect to deliver the selected messages in order when converting a peripheral bus packet into messages. In one example embodiment, the given tunnel adapter is configured to cause the coherent interconnect to deliver the selected messages in order by assigning the selected messages the same hash value, thereby causing the coherent interconnect to route the selected messages over the same route.

[0008] In some embodiments, a given one of the tunnel adapters is configured to: (i) receive messages corresponding to peripheral bus packets originating from a plurality of different peripheral bus modules; (ii) maintain a respective context for each of the plurality of different peripheral bus modules; and (iii) reconstruct the peripheral bus packets originating from each of the plurality of different peripheral bus modules using the respective context. In some embodiments, at least one of the tunnel adapters is configured to control flow of the messages by applying credit-based flow control.

[0009] According to one embodiment described herein, there is also provided a method in a peripheral device comprising a host chip. The method comprises: interconnecting electronic components of the peripheral device with each other in accordance with a coherent interconnect protocol over a coherent interconnect in the peripheral device. Communicating in accordance with a peripheral bus protocol using one or more first peripheral bus modules. Communicating with a secondary chip comprising one or more second peripheral bus modules over an inter-chip (C2C) or intra-die (D2D) bus. Transferring peripheral bus packets of the peripheral bus protocol between the first peripheral bus modules and the second peripheral bus modules over the coherent interconnect and the C2C or D2D bus using one or more tunnel adapters by converting between the peripheral bus packets, messages of the coherent interconnect protocol, and C2C or D2D bus messages.

[0010] The present application will become more fully understood from the detailed description given herein below, and accompanied by the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 FIG. 1 is a block diagram schematically illustrating a computing system including a peripheral device that employs PCIe traffic tunneling over a coherent interconnect, according to an embodiment of the present application;

[0012] Figure 2 FIG. 2 is a diagram schematically illustrating a structure of a coherent hub interface (CHI) flow control unit (FLIT) for tunneling PCIe traffic over a coherent interconnect, according to an embodiment of the present application;

[0013] Figure 3 FIG. 3 is a diagram schematically illustrating a conversion between a PCIe transaction layer packet (TLP) and a plurality of CHI FLITs, according to an embodiment of the present application;

[0014] Figure 4 FIG. 4 is a flowchart schematically illustrating a method for converting a PCIe TLP to a CHI FLIT, according to an embodiment of the present application;

[0015] Figure 5 FIG. 5 is a flowchart schematically illustrating a method for converting a CHI FLIT to a PCIe TLP, according to an embodiment of the present application;

[0016] Figure 6 FIG. 6 is a block diagram schematically illustrating a computing system including a peripheral device that employs PCIe traffic tunneling over a coherent interconnect and chip-to-chip (C2C), according to an embodiment of the present application. DETAILED DESCRIPTION

[0017] SUMMARY

[0018] Embodiments of the present application described herein provide methods and systems for tunneling peripheral bus protocol traffic over a coherent interconnect in a peripheral device. The disclosed technology is applicable to various types of peripheral devices, such as network adapters, storage devices, storage controllers, graphics processing units (GPUs), accelerators, etc.

[0019] In some embodiments, the peripheral device communicates with a host over a peripheral bus using a peripheral bus protocol. Embodiments described herein are primarily illustrated with the Peripheral Component Interconnect (PCI) protocol as an example. The disclosed technology is applicable to any other suitable peripheral bus protocol, such as Compute Express Link (CXL) or Nvlink.

[0020] The peripheral device additionally contains powerful compute resources, such as multiple processing cores, memory controllers for storing data in memory, multiple coherent caches, and a coherent interconnect connecting these components. In an example configuration, the coherent interconnect and its connected various components communicate according to a Coherent Hub Interconnect (CHI) protocol. For example, CHI is specified in the “CHI Architecture Specification,” published September 2022. Alternatively, other suitable coherent interconnect protocols can be used. One non-limiting example is the TileLink protocol. 5CHI Architecture Specification.” Alternatively, other suitable coherent interconnect protocols can be used. One non-limiting example is the TileLink protocol.

[0021] In one example implementation, the processing cores, memory controllers, coherent caches, and coherent interconnect are densely distributed in a system-on-a-chip (SoC). The various components are connected to the coherent interconnect using CHI links.

[0022] The SoC also includes multiple PCIe modules (also referred to as peripheral bus modules) that communicate with a host over a PCIe bus. The PCIe modules can include, for example, a PCIe endpoint (EP) coupled to the PCIe bus and one or more PCIe EP devices. The PCIe EP devices can be located anywhere in the SoC. In principle, a dedicated PCIe connection can be routed between each PCIe EP device and the PCIe EP. However, in practice, due to the dense layout and interconnects between the processing cores, coherent caches, and coherent interconnect, routing PCIe connections across the SoC is extremely challenging.

[0023] In embodiments of the invention, PCIe traffic is transmitted over the coherent interconnect (“tunneling”). In one example embodiment, the peripheral device contains multiple “tunnel adapters” that are used to connect the PCIe modules to the coherent interconnect. Each tunnel adapter includes circuitry for converting between peripheral bus data packets (e.g., PCIe transaction-level packets - TLPs) and coherent interconnect protocol messages (e.g., CHI flow control units - FLITS). When using this architecture, there is no need to use dedicated PCIe connections, and the overall design of the peripheral device is also significantly simplified.

[0024] In some embodiments, one or more of the PCIe modules are located in a secondary chip, such as a field programmable gate array (FPGA). The secondary chip is connected to the SoC over an inter-chip (C2C) or die-to-die (D2D) bus. One example of a C2C bus is the Universal Chip Interconnect Express TM (UCIe) bus. In these embodiments, the tunneling is performed both over the coherent interconnect and over the C2C or D2D bus.

[0025] Example implementations of the disclosed tunneling techniques are described herein, including examples of mapping between PCIe TLPs and CHIFLITs. Techniques for ensuring that selected FLITs are transmitted in order over a coherent interconnect (e.g., to comply with ordering requirements of the PCIe protocol) are also described herein.

[0026] System Description

[0027] Figure 1 is a block diagram in accordance with an embodiment of the present invention, which schematically illustrates a computing system 20 that includes a peripheral device 24 that employs PCIe traffic tunneling over a coherent interconnect. The peripheral device 24 serves a host 28, such as a server or other computer. The peripheral device communicates with the host using the PCIe protocol over a PCIe link 32. Alternatively, any other suitable peripheral bus and peripheral bus protocol can be used, such as Compute Express Link (CXL), Nvlink, and Ethernet and InfiniBand (IB) protocols, among others. The system 20 also includes host memory, in this case dynamic random access memory (DRAM) 40. TM

[0028] In this example, the peripheral device 24 is a high-performance network adapter for connecting the host 28 to a packet network 36. Such high-performance network adapters with strong internal processing capabilities are also referred to as “smart NICs” or data processing units (DPUs). Alternatively, the peripheral device 24 can include a storage device, a GPU, an accelerator, or any other suitable type of peripheral device.

[0029] The peripheral device 24 includes a system on a chip (SoC) 44 and a device memory, in this case DRAM 48. The DRAM 48 (used as device memory) is not to be confused with the DRAM 40 (used as host memory). The SoC 44 can be viewed as performing two main tasks: (i) internal processing; (ii) PCIe communication with the host 28. The internal processing can involve any suitable type of processing, such as, for a network adapter, packet processing; for an accelerator, mathematical computations and / or offloading operations, etc.

[0030] ​To perform internal processing, the SoC 44 includes one or more processing cores (in this example, multiple ARM cores 52), one or more cache memories 56, a coherent interconnect 56 (also referred to as a “coherence fabric”), and a memory controller 60. The cores 52 store data in the DRAM 48 using the memory controller 60, and can cache portions of the data in the cache memories 56. The cores 52 communicate with the cache memories 56 and the memory controller 60 through the coherent interconnect 64 (also referred to as a coherence fabric). The cores 52, the memory controller 60, and the cache memories 56 are collectively referred to herein as “electronic components” that are connected through the coherent interconnect 64.

[0031] In this example, the coherent interconnect 64 operates according to the CHI protocol. The basic unit of data for the CHI protocol is a 32-byte message, referred to as a flit (flow control unit). More generally, CHI is considered herein as a non-limiting example of a coherent interconnect protocol, and a flit is considered herein as a non-limiting example of a coherent interconnect protocol message.

[0032] The interconnect 64 includes multiple ports and multiple switches that forward each flit from an input port (the input port through which the flit enters the interconnect) to an output port (the port through which the flit exits the interconnect to its destination). In general, there are multiple different physical routes through the interconnect 64 between a given input port and a given output port. One of the properties of a flit is a hash value that the switches use to select a physical route for the flit.

[0033] To perform PCIe communications, the SoC 44 includes a PCI EP 76 that is used to handle PCIe communications with the host 28 over the PCIe link 32. The SoC 44 also includes a NIC 72 that is used as a PCIe EP to communicate using Ethernet over the network 36. The NIC 72 communicates with the PCIe EP 76 using PCIe. The NIC 72 also connects to the coherent interconnect 64 through a PCIe Request Node - Full (PRNF) module 68. The PRNF 68 translates between the PCIe and the coherent interconnect protocols, including maintaining transaction ordering as required by the PCIe.

[0034] In addition, the SoC 44 includes one or more PCIe EP devices 80. Each PCIe EP device 80 performs certain specified processing tasks, and communicates with the PCIe EP 76 using PCIe. Examples of tasks that the PCIe EP devices 80 can perform include direct memory access (DMA), cryptographic operations, compression and / or decompression, various acceleration or offload tasks, or any other suitable tasks.

[0035] As part of the operation of peripheral device 24, PCIe EP device 80 is to send PCIe packets to PCIe EP 76 and receive PCIe packets from PCIe EP 76, which in turn sends PCIe packets to host 28 and receives PCIe packets from host 28. For clarity, PCIe EP device 80 and PCIe EP 76 are referred to as "PCIe modules" or "peripheral bus modules," which send and receive PCIe packets to each other. The following description primarily takes PCIe transaction-level packets (TLPs) as an example of PCIe packets. TLPs can vary in size, e.g., 128, 256, or 512 bytes. More generally, PCIe packets are considered a non-limiting example of peripheral bus packets.

[0036] As noted above, PCIe EP device 80 can be scattered throughout SoC 44. It is extremely challenging to route dedicated PCIe connections between PCIe EP device 80 and PCIe EP 76. Instead, in embodiments of the invention, PCIe traffic (e.g., TLPs) between PCIe EP device 80 and PCIe EP 76 is "tunneled" through coherent interconnect 64.

[0037] In some embodiments, SoC 44 includes a number of tunnel circuits referred to as tunnel adapters 84. A given tunnel adapter 84 is coupled between a respective PCIe module (PCIe EP device 80 or PCIe EP 76) and a port of coherent interconnect 64. The tunnel adapters convert between PCIe packets and CHI FLITs. In transmission, tunnel adapter 84 receives PCIe packets from its respective PCIe module, converts each PCIe packet into one or more CHI FLITs, and sends the FLITs to coherent interconnect 64. In reception, tunnel adapter 84 receives CHI FLITs from coherent interconnect 64, reconstructs PCIe packets from the FLITs, and sends the PCIe packets to the PCIe module. Logically, these operations can be viewed as sending PCIe packets through a "tunnel" 86 in coherent interconnect 64. The tunneling technique, including the detailed operations of tunnel adapters 84, are further described below.

[0038] Example tunneling scheme

[0039] In various embodiments, tunnel adapters 84 can convert between PCIe packets (e.g., TLPs) and CHI FLITs in different ways.

[0040] Figure 2is a diagram that schematically illustrates the structure of a CHIFLIT for tunneling PCIe traffic over a coherent interconnect 64, according to an embodiment of the application. In the present example, the FLIT includes a FLIT header 90, a routing field 94 (also referred to as a uniform coherence fabric (UCF) routing decision field), and a payload 98.

[0041] The FLIT header 90 includes header fields that affect the establishment, teardown, and termination of tunnels 86 by the tunnel adapter 84. The Flit header fields can contain, for example:

[0042] A tunnel port ("TNLPortID") field. For example, a receiving tunnel adapter 84 that serves multiple PCIe EP devices can use this field to identify to which PCIe EP device a PCIe packet is addressed. FLITs that belong to the same PCIe packet should be assigned the same tunnel port value.

[0043] A "Subtype" field indicates the type of tunneled PCIe packet being transmitted in the FLIT. Example types include "PCIe TLP", "Credit message", etc.

[0044] The routing field 94 specifies how the FLIT is to be routed by the coherent interconnect 64. The routing field 94 can contain, for example:

[0045] A packet identifier ("PktID") field. In certain embodiments, the PktID field is set to a value indicating "tunneled" in FLITs that are transmitting PCIe traffic.

[0046] A hub port identifier ("HUBPortID") field, which specifies the port of the core 64 to which the FLIT should be routed.

[0047] A tunnel hash ("TnlHash") field. This field is used to ensure in-order delivery of selected FLITs, as will be detailed below. The tunnel hash field is considered valid only when the PktID field is set to "tunneled".

[0048] A source identifier ("SrcID") field and a target identifier ("TgtID") field.

[0049] Figure 3is a diagram that schematically illustrates the conversion between a PCIe transaction layer packet (TLP) 100 and a plurality of CHI FLITs 104, according to an embodiment of the present application, the process being performed by the tunnel adapter 84. As noted above, the fixed size of a FLIT 104 is 32 bytes. The size of a TLP 100 can vary, for example, 128, 256, or 512 bytes, and is larger than a FLIT 104. Thus, the tunnel adapter 84 typically converts a given TLP 100 into a plurality of FLITs 104, and vice versa. In Figure 3 the example, the TLP 100 is tunnel-transmitted over four FLITs, labeled FLIT 0...FLIT 3.

[0050] According to the PCIe specification, the TLP 100 contains a TLP header 108 and TLP data 112. When converting the TLP 100 into FLITs 104, the tunnel adapter 84 inserts the TLP header 108 into FLIT 0, and inserts the TLP data 112 into the four FLITs. In addition, according to the CHI protocol, the tunnel adapter 84 fills in a route field 116 and a FLIT header 120 of each FLIT 104. The tunnel adapter 84 also inserts TLP metadata into FLIT 0.

[0051] Ensuring in-order delivery of selected FLITs

[0052] In general, the coherent interconnect 64 does not guarantee in-order delivery of FLITs (i.e., it does not guarantee that a sequence of FLITs sent from a certain input port to a certain output port will exit the output port in the order that they were provided to the input port). For example, as noted above, the coherent interconnect 64 can include multiple different physical routes between a given input port and a given output port. The different routes can have different latencies, for example, because they pass through different numbers of switches and "hops." A sequence of FLITs distributed over two or more routes can arrive out of order.

[0053] However, for certain sequences of FLITs, in-order arrival is very important. For example, it can be very important that the FLITs corresponding to the same TLP (e.g., FLIT 0-FLIT 3 in the example above) arrive in the order that they were sent. Figure 3

[0054] ​When multiple routing possibilities exist, the switch opportunity in interconnect 64 selects a route for a FLIT based on the value of the FLIT's Tunnel Hash ("Tnl Hash") field. In some embodiments, tunnel adapter 84 ensures that FLITS in a selected group are transmitted in order to a peer tunnel adapter at the far end of interconnect 64 by assigning the same Tunnel Hash value for the FLITS in the group. For example, in some embodiments, tunnel adapter 84 assigns the same Tunnel Hash value for FLITS carrying the same TLP.

[0055] Different PCIe EP devices 80 can have different requirements for in-order transmission of FLITS, depending on the functionality of the EP device. The ordering mechanism described above enables PCIe EP devices 80 to specify ordering as needed.

[0056] In general, for FLITS that do not require in-order transmission, tunnel adapter 84 will aim to assign different Tunnel Hash values. Assigning different Tunnel Hash values improves the distribution ("multipathing") of FLITS among different routes in a coherent interconnect, thereby enhancing throughput and load balancing.

[0057] In certain embodiments, certain PCIe modules can require in-order transmission of certain PCIe packets (not to be confused with in-order transmission of FLITS belonging to a PCIe packet). The sending tunnel adapter ensures packet-level ordering by assigning the same Tunnel Hash value for all FLITS belonging to all PCIe packets in a group. In one example embodiment, the interface between the sending tunnel adapter and a locally-coupled PCIe module enables the PCIe module to specify the following for each PCIe packet transmitted to the tunnel adapter:

[0058] Order Enable: a field indicating whether this PCIe packet requires in-order transmission in a group of multiple PCIe packets.

[0059] Order ID: a five-bit index identifying the group of PCIe packets that require in-order transmission. This index enables the PCIe module and tunnel adapter to handle multiple different groups of PCIe packets and ensures in-order transmission within each group.

[0060] Flow Control Using End-to-End Credits

[0061] Suppose a tunnel adapter 84 (referred to herein as a "sending tunnel adapter") sends CHI FLITS to a peer tunnel adapter 84 (referred to as a "receiving tunnel adapter") over the interconnect 64. (A given tunnel adapter 84 can potentially act as a "sending tunnel adapter" for one or more streams of FLITS and as a "receiving tunnel adapter" for one or more other streams of FLITS simultaneously. For simplicity, the following description will focus on a particular stream.)

[0062] At some point, the receiving tunnel adapter can be unable to handle the data bandwidth sent by the sending tunnel adapter, e.g., due to a buffer overflow. Accordingly, in some embodiments, the tunnel adapters 84 support a credit-based flow control mechanism to ensure that the sending tunnel adapter does not exceed the data bandwidth that the receiving tunnel adapter can handle.

[0063] In one example embodiment, the receiving tunnel adapter allocates "credits" to the sending tunnel adapter by sending credit messages over the interconnect 64. The allocated credits indicate a data quota that the sending tunnel adapter can send. When the receiving tunnel adapter approaches a point at which it cannot handle additional bandwidth, it will stop allocating new credits or allocate fewer credits. Accordingly, the sending tunnel adapter will reduce its transmission bandwidth. When the receiving tunnel adapter is again able to handle new traffic, it will allocate new credits, enabling the sending tunnel adapter to resume transmission.

[0064] In some embodiments, the receiving tunnel adapter receives FLITS from two or more sending tunnel adapters. In such a configuration, the receiving tunnel adapter typically maintains a separate and independent credit mechanism for each sending tunnel adapter.

[0065] Tunnel adapter sending and receiving processing

[0066] The following description provides an example of the sending and receiving processing performed by the tunnel adapters 84. The sending processing refers to the process of converting a PCIe packet into CHI FLITS and sending the FLITS over the coherent interconnect 64. The receiving processing refers to the process of reconstructing a PCIe packet from CHI FLITS received over the coherent interconnect 64. Since communication between PCIe modules is typically bidirectional, a given tunnel adapter typically performs both sending and receiving processing simultaneously.

[0067] Figure 4is a flowchart that schematically illustrates an example sender process (i.e., a method for converting a PCIe TLP to a CHI FLIT) according to embodiments of the present application. The method begins with the tunnel adapter 84 receiving a PCIe TLP from a locally coupled PCIe module (e.g., PCIe EP device 80 or PCIe EP 76) at a TLP input stage 130. At a FLIT creation stage 134, the tunnel adapter 84 creates one or more FLITs that will transport the TLP header 108 and TLP data 112 of the TLP in question. The number of FLITs depends on the size of the TLP data 112.

[0068] At a FLIT fill stage 138, the tunnel adapter 84 fills the routing field 116 and FLIT header 120 of the FLITs. As mentioned above, the tunnel adapter 84 sets the packet ID ("PktID") to "Tunneled." In addition, the tunnel adapter 84 assigns the same tunnel hash value ("TnlHash") to all FLITs corresponding to the TLP. Thus, the switches in the coherent interconnect 64 will route all FLITs of the TLP through the same physical route. Accordingly, the FLITs of the TLP are guaranteed to arrive at the peer tunnel adapter 84 in order.

[0069] At a credit check stage 142, the tunnel adapter 84 checks whether there are credits available for sending the FLITs. If not, the tunnel adapter waits until sufficient credits are available. If sufficient credits are available, at a transmission stage 146, the tunnel adapter 84 sends the FLITs to the coherent interconnect 84.

[0070] Figure 5 is a flowchart that schematically illustrates an example receiver process (i.e., a method for converting a CHI FLIT to a PCIe TLP) according to embodiments of the present application. The method begins with the tunnel adapter 84 receiving a CHI FLIT from the coherent interconnect 64 at a FLIT reception stage 150. The received FLIT carries information pertaining to a specific PCIe TLP that is to be reconstructed.

[0071] At a new TLP check stage 154, the tunnel adapter 84 checks whether the received FLIT is the first FLIT in a new TLP that is to be reconstructed. If so, at a context creation stage 158, the tunnel adapter 84 creates a context for holding information of the new TLP. The context can include information such as the TLP source (the PCIe module that sent the TLP) and / or any other suitable information. If the TLP is not new, i.e., the received FLIT is not the first FLIT of the TLP, stage 158 is skipped.

[0072] In extraction phase 162, tunnel adapter 84 extracts data and metadata from the received FLIT. The extracted data and metadata may include those described above. Figure 2 and Figure 3 Any or all of the fields shown. In TLP population phase 166, tunnel adapter 84 populates the TLP being reconstructed with the extracted data and metadata. In this phase, the tunnel adapter can update the context of the relevant source to reflect the current connection phase.

[0073] In the final FLIT check phase 170, the tunnel adapter 84 checks whether the received FLIT is the last FLIT carrying the TLP information. If not, the method loops back to phase 150 to receive and process the next FLIT for the TLP. If the received FLIT is the last FLIT, the tunnel adapter 84 sends the reconstructed TLP to the locally coupled PCIe module (PCIe EP device 80 or PCIe EP 76) in the TLP output phase 174. (In practice, the tunnel adapter 84 typically knows the number of FLITs transmitting the TLP being reconstructed. In other words, the last TLP is usually not marked as the last.)

[0074] above Figure 4 and Figure 5 The method flow described herein is merely an example flow chosen for clarity of concept. In alternative embodiments, any other suitable flow may be used to implement sender and receiver processing in tunnel adapter 84.

[0075] PCIe tunneling via a consistent architecture and C2C / D2D bus.

[0076] Figure 6 To illustrate a block diagram of a computing system 180 according to an embodiment of the present invention, the computing system includes peripheral devices that employ PCIe traffic tunneling via a coherent interconnect and chip-to-chip (C2C) bus. In this embodiment, peripheral device 24 includes a main chip 184 and an auxiliary chip 188. The auxiliary chip 188 may include, for example, an FPGA.

[0077] The main chip 184 and the auxiliary chip 188 are connected via a chip-to-chip (C2C) bus 192, which in this example is a Universal Chip Interconnect Express. TM (UCIe) bus. In alternative embodiments, any other suitable C2C bus can be used. The embodiments described herein are illustrated using a C2C bus as an example. In alternative embodiments, the main chip 184 and the auxiliary chip 188 can be connected via a suitable die-to-die (D2D) bus. The techniques disclosed herein are also applicable to D2D buses.

[0078] Some components and functions of the main chip 184 are the same as those mentioned above. Figure 1 Similar to the SoC 44. However, in Figure 6 In this embodiment, one or more PCIe EP devices 80 are located in auxiliary chip 188, rather than in main chip 184. Therefore, PCIe traffic between PCI EP 76 and these PCIe EP devices 80 should be tunneled through both coherence interconnect 64 and C2C bus 192.

[0079] In some embodiments, a hybrid configuration is used, where one or more PCIe EP devices 80 reside in the main chip 184, while one or more other PCIe EP devices 80 reside in the auxiliary chip 188. In these configurations, Figure 1 The tunneling transmission scheme is used to communicate with the PCIe EP device 80 of the main chip 184, while Figure 6 The tunneling transmission scheme is used to communicate with the PCIe EP device 80 of the auxiliary chip 188. The main chip 184 typically includes components such as NIC 72 and PRNF 68, etc. Figure 1 As shown; purely for clarity, Figure 6 These components have been omitted.

[0080] In this example, the main chip 184 and the auxiliary chip 188 each include at least one C2C interface 196 for communication via the UCIe bus 192. (When a D2D bus is used instead of a C2C bus, interface 196 is referred to as a D2D interface.) Communication on the UCIe bus can be performed, for example, using the AXI protocol. The main chip 184 includes at least one tunnel adapter 198 for switching between CHI and AXI. The auxiliary chip 188 includes at least one tunnel adapter 200 for switching between AXI and PCIe.

[0081] In one embodiment, transferring outbound PCIe traffic from PCIe EP 76 (located in main chip 184) to PCIe EP device 80 (located in auxiliary chip 188) includes the following steps:

[0082] The tunnel adapter 84 in the main chip 184 receives PCIe packets (e.g., TLP) from the PCIe EP 76 and converts the PCIe packets to CHIFLIT (e.g., using...). Figure 2 and Figure 3 (Mapping in the middle), and send CHI FLIT on the consistency interconnect 64 via tunnel 86.

[0083] The tunnel adapter 198 in the main chip 184 receives the CHI FLIT from the coherence structure 64 and converts it into a UCIeFLIT.

[0084] The C2C interface 196 in the primary chip 184 transmits the UCIe FLITs over the UCIe bus 192 to a peer C2C interface 196 in the secondary chip 188, which in turn forwards the UCIe FLITs to the tunnel adapter 200.

[0085] The tunnel adapter 200 in the secondary chip 188 converts the UCIe FLITs to CHI FLITs, converts the CHI FLITs to PCIe packets (e.g., using the mapping of Figure 2 and Figure 3 ), and then forwards the PCIe packets to the PCIe EP device 80.

[0086] In one embodiment, transmitting inbound PCIe traffic from the PCIe EP device 80 (located in the secondary chip 188) to the PCIe EP 76 (located in the primary chip 184) includes the following steps:

[0087] The tunnel adapter 200 in the secondary chip 188 receives the PCIe packets (e.g., TLPs) from the PCIe EP device 80, converts the PCIe packets to CHI FLITs, converts the CHI FLITs to UCIe FLITs, and forwards the UCIe FLITs to the C2C interface 196.

[0088] The C2C interface 196 in the secondary chip 188 transmits the UCIe FLITs over the UCIe bus 192 to a peer C2C interface 196 in the primary chip 184, which in turn forwards the UCIe FLITs to the tunnel adapter 198.

[0089] The tunnel adapter 198 in the primary chip 184 converts the UCIe FLITs to CHI FLITs and sends the CHI FLITs over the coherent interconnect 64 via the tunnel 86.

[0090] The tunnel adapter 84 in the primary chip 184 receives the CHI FLITs from the coherent fabric 64, converts the CHI FLITs to PCIe packets (e.g., TLPs), and forwards the PCIe packets to the PCIe EP device 76.

[0091] In the present context, the terms “PCIe packet”, “CHI FLIT”, and “UCIe FLIT” are considered herein to be examples of “peripheral bus packet”, “message of a coherent interconnect protocol”, and “C2C bus message”, respectively. In alternative embodiments, any other suitable peripheral bus protocol and / or packet, coherent interconnect protocol and / or message, and / or C2C bus and / or message can be used.

[0092] As noted above, one of the tasks of the transmit tunnel adapter is to convert the CHI FLITs into UCIe FLITs. The transmit tunnel adapter typically performs this conversion by treating the CHI FLITs (including data and various header, metadata, and routing fields, see Figure 3 ) as data and stuffing this data into UCIe FLITs. In addition, the transmit tunnel adapter typically adds a cyclic redundancy check (CRC) to each UCIe FLIT. The receive tunnel adapter (located at the other end of the C2C bus 192) checks whether the CRC of a received UCIe FLIT is valid. If the CRC check fails, the receive tunnel adapter requests the transmit tunnel adapter to retransmit the UCIe FLIT. In response, the transmit tunnel adapter retransmits the UCIe FLIT.

[0093] In some embodiments, a pair of tunnel adapters 198 and 200 that communicate over the C2C bus 192 apply flow control based on end-to-end credits to the traffic they transmit and receive. This flow control scheme is similar to the one described in the section "Flow control using end-to-end credits" above.

[0094] In the example configuration of Figure 6 , the auxiliary chip 188 is part of the peripheral device 24. In alternative embodiments, the auxiliary chip 188 can be located outside the peripheral device, e.g., on the same circuit board but not part of the same package device.

[0095] As shown in Figure 1 and Figure 6 , the configuration of the systems 20 and 180, including the internal configuration of the peripheral device 24, the SoC 44, the main chip 184, and the auxiliary chip 188, is an example configuration chosen for conceptual clarity. In alternative embodiments, any other suitable configuration can be used. Elements that are not necessary for understanding the principles of the invention have been omitted from the figures for clarity.

[0096] The various elements of the systems 20 and 180, including the elements of the peripheral device 24, in particular the SoC 44, the main chip 184, and the auxiliary chip 188, can be implemented in hardware (e.g., one or more application-specific integrated circuits (ASICs) or FPGAs), software, or using a combination of hardware and software elements. In some embodiments, certain elements of the SoC 44, the main chip 184, and / or the auxiliary chip 188 can be implemented, in part or in whole, using one or more general-purpose processors programmed to perform the functions described herein. The software can be downloaded to any of the processors in electronic form (over the internet for example) or alternatively or additionally it can be provided and / or stored on non-transitory tangible media, such as magnetic, optical or electronic memory.

[0097] While the embodiments described herein are primarily directed to CHI and PCIe, the methods and systems described herein can be used to tunnel any other suitable protocol over any other suitable type of coherent interconnect.

[0098] It will therefore be appreciated that the above described embodiments are merely given by way of example and that the application is not limited to the specific details described above. On the contrary, the scope of the application is limited only by the claims and any equivalents thereof. The documents incorporated by reference are to be considered an integral part of the application and are hereby expressly incorporated by reference, but only to the extent that the incorporated material does not contradict the main teaching of the present application. In case of a contradiction, the present application will prevail.

Claims

1. A peripheral device comprising a main chip comprising: a coherent interconnect to connect electronic components of the peripheral device according to a coherent interconnect protocol; one or more first peripheral bus modules to communicate according to a peripheral bus protocol; an inter-chip C2C or intra-die D2D interface to communicate with a secondary chip comprising one or more second peripheral bus modules over a C2C or D2D bus; and one or more tunnel adapters to transfer peripheral bus packets between the first peripheral bus modules and the second peripheral bus modules over the coherent interconnect and the C2C or D2D bus by translating between peripheral bus packets of the peripheral bus protocol, messages of the coherent interconnect protocol, and C2C or D2D bus messages.

2. The peripheral device of claim 1, wherein, A given tunnel adapter is to: receive one or more of the messages of the coherent interconnect protocol carrying one or more of the peripheral bus packets from the coherent interconnect; translate the one or more of the messages of the coherent interconnect protocol into one or more of the C2C or D2D bus messages; and send the one or more of the C2C or D2D bus messages to the C2C or D2D bus via the C2C or D2D interface.

3. The peripheral device of claim 1, wherein, A given tunnel adapter is to: receive one or more of the C2C or D2D bus messages carrying one or more of the peripheral bus packets from the C2C or D2D bus via the C2C or D2D interface; translate the one or more of the C2C or D2D bus messages into one or more of the messages of the coherent interconnect protocol; and send the one or more of the messages of the coherent interconnect protocol to the coherent interconnect.

4. The peripheral device of claim 1, wherein, A given one of the first peripheral bus modules is to communicate with a host over a peripheral bus according to the peripheral bus protocol.

5. The peripheral device of claim 1, wherein, A data size of the messages of the coherent interconnect protocol is smaller than a data size of the peripheral bus packets, and wherein a given one of the tunnel adapters is to translate a peripheral bus packet into a plurality of the messages.

6. The peripheral device of claim 1, wherein, A data size of the messages of the coherent interconnect protocol is smaller than a data size of the peripheral bus packets, and wherein a given one of the tunnel adapters is to identify a plurality of the messages corresponding to a peripheral bus packet and to reconstruct the peripheral bus packet from the identified plurality of the messages.

7. The peripheral device of claim 1, wherein, The coherence interconnect protocol does not guarantee unconditional in-order delivery of the messages, and wherein, in converting the peripheral bus packets to the messages, a given one of the tunnel adapters is operative to select two or more of the messages, and to cause the coherence interconnect to deliver the selected messages in-order.

8. The peripheral device of claim 7, wherein, The given one of the tunnel adapters is operative to cause the coherence interconnect to deliver the selected messages in-order by assigning the selected messages the same hash value, thereby causing the coherence interconnect to route the selected messages over the same route.

9. The peripheral device of claim 1, wherein, A given one of the tunnel adapters is operative to: receive messages corresponding to peripheral bus packets originating from a plurality of different peripheral bus modules; maintain a respective context for each of the plurality of different peripheral bus modules; and reconstruct the peripheral bus packets originating from each of the plurality of different peripheral bus modules using the respective context.

10. The peripheral device of claim 1, wherein, At least one of the tunnel adapters is operative to control flow of the messages by applying credit-based flow control.

11. A method for use in a peripheral device comprising a host chip, the method comprising: interconnecting electronic components of the peripheral device with each other over a coherence interconnect in the peripheral device according to a coherence interconnect protocol; communicating according to a peripheral bus protocol using one or more first peripheral bus modules; communicating with a secondary chip comprising one or more second peripheral bus modules over an inter-chip C2C or intra-die D2D bus; and transferring the peripheral bus packets between the first peripheral bus modules and the second peripheral bus modules over the coherence interconnect and the C2C or D2D bus using one or more tunnel adapters that convert between peripheral bus packets of the peripheral bus protocol, messages of the coherence interconnect protocol, and C2C or D2D bus messages.

12. The method of claim 11, wherein, Transferring the peripheral bus packets includes, in a given tunnel adapter: receiving one or more of the messages of the coherence interconnect protocol carrying one or more of the peripheral bus packets from the coherence interconnect; converting the one or more of the messages of the coherence interconnect protocol to one or more of the C2C or D2D bus messages; and sending the one or more of the C2C or D2D bus messages to the C2C or D2D bus.

13. The method of claim 11, wherein, Transferring the peripheral bus packets includes, in a given tunnel adapter: receiving one or more of the C2C or D2D bus messages carrying one or more of the peripheral bus packets from the C2C or D2D bus; converting the one or more of the C2C or D2D bus messages to one or more of the messages of the coherence interconnect protocol; and sending the one or more of the messages of the coherence interconnect protocol to the coherence interconnect. sending the one or more of the messages of the coherent interconnect protocol to the coherent interconnect.

14. The method of claim 11, wherein, communicating according to the peripheral bus protocol includes communicating between a given one of the first peripheral bus modules and a host over a peripheral bus.

15. The method of claim 11, wherein, the data size of the messages of the coherent interconnect protocol is smaller than the data size of the peripheral bus packets, and wherein transmitting the peripheral bus packets includes converting, in a given one of the tunnel adapters, a peripheral bus packet into a plurality of the messages.

16. The method of claim 11, wherein, the data size of the messages of the coherent interconnect protocol is smaller than the data size of the peripheral bus packets, and wherein transmitting the peripheral bus packets includes identifying, in a given one of the tunnel adapters, a plurality of the messages corresponding to a peripheral bus packet and reconstructing the peripheral bus packet from the identified plurality of the messages.

17. The method of claim 11, wherein, the coherent interconnect protocol does not guarantee unconditional in-order delivery of the messages, and wherein converting the peripheral bus packets into the messages includes selecting two or more of the messages and causing the coherent interconnect to deliver the selected messages in-order.

18. The method of claim 17, wherein, causing the coherent interconnect to deliver the selected messages in-order includes assigning the selected messages the same hash value, thereby causing the coherent interconnect to route the selected messages over the same route.

19. The method of claim 11, wherein, transmitting the peripheral bus packets includes, in a given one of the tunnel adapters: receiving messages corresponding to peripheral bus packets originating from a plurality of different peripheral bus modules; maintaining a respective context for each of the plurality of different peripheral bus modules; and reconstructing the peripheral bus packets originating from each of the plurality of different peripheral bus modules using the respective context.

20. The method of claim 11, and comprising: controlling flow of the messages in at least one of the tunnel adapters by applying credit-based flow control.

Images