Parameter Exchange for Die-to-Die Interconnect

Abstract

The port includes an inter-die (D2D) adapter coupled to another die via a D2D link and determining from a set of registers a set of capabilities of the D2D adapter to advertise in negotiation with a link partner D2D adapter, where the D2D adapter is on the die and the link partner D2D adapter is disposed on a remote link partner die. A first capability advertisement message is sent to the link partner D2D adapter to advertise the set of capabilities to the link partner D2D adapter. A second capability advertisement message is received from the link partner D2D adapter, where the second capability advertisement message identifies the set of capabilities of the link partner D2D adapter. A final configuration of the D2D link is determined to couple the die to the link partner die.

Description

[Technical field]

[0001] Related Applications This application claims priority to U.S. Patent Application No. 17 / 855,687, entitled "Parameter Exchange for Die-to-Die (D2D) Interconnect," filed on June 30, 2022, which is an application claiming priority to Indian Provisional Patent Application No. 202141061716, filed on December 30, 2021, entitled "Software Visibility and Protocol Parameter Exchange Flow for Die-to-Die (D2D) Interconnect." The disclosures of the prior applications are deemed part of the disclosure of this application and are incorporated herein by reference in their entirety.

[0002] The present disclosure relates to computing systems, and particularly, but not exclusively, to physical interconnects and associated link protocols. [Background technology]

[0003] Advances in semiconductor processing and logic design have enabled an increase in the amount of logic that can be present in an integrated circuit device. As a natural consequence, computer system configurations have evolved from a single or multiple integrated circuits in a system to multiple cores, multiple hardware threads, and multiple logical processors on individual integrated circuits, as well as other interfaces integrated with such processors. A processor or integrated circuit typically comprises a single physical processor die, which may include any number of cores, hardware threads, logical processors, interfaces, memory, controller hubs, etc.

[0004] Smaller computing devices have become more popular as a result of their greater ability to fit more processing power in a smaller package. Smartphones, tablets, ultra-thin notebooks, and other user equipment have grown exponentially. However, these smaller devices rely on servers for both data storage and complex processing that exceeds their form factor. As a result, the demand in the high performance computing market (i.e., server space) has also increased. For example, in today's servers, there are typically multiple physical processors (also referred to as multiple sockets) to increase computing power, rather than just a single processor with multiple cores. However, as processing power increases along with the number of devices in a computing system, communication between the sockets and other devices becomes more important.

[0005] In fact, interconnects have grown from traditional multi-drop buses that primarily handled electrical communications to full-blown interconnect architectures that facilitate high-speed communications. Unfortunately, as demands for future processors that consume at much higher rates place corresponding demands on the capabilities of existing interconnect architectures. [Brief description of the drawings]

[0006] [Figure 1] 1 illustrates an embodiment of a computing system that includes an interconnect architecture.

[0007] [Diagram 2] 1 illustrates an embodiment of an interconnect architecture that includes a layered stack.

[0008] [Diagram 3] 1 illustrates an embodiment of a potential high performance inter-processor interconnect configuration.

[0009] [Figure 4]1 illustrates an embodiment of a layered protocol stack associated with an interconnect.

[0010] [Diagram 5] 1 shows a simplified block diagram of an exemplary computing system utilizing a link that conforms to the Compute Express Link (CXL) based protocol.

[0011] [Figure 6] 1 shows a simplified block diagram of an exemplary system-on-chip (SoC) package.

[0012] [Figure 7] 1 shows a simplified block diagram of a two-dimensional packaging substrate and interconnects for coupling two or more devices.

[0013] [Figure 8A] 1 shows a simplified block diagram of an exemplary package configuration. [Figure 8B] 1 shows a simplified block diagram of an exemplary package configuration. [Figure 8C] 1 shows a simplified block diagram of an exemplary package configuration. [Figure 8D] 1 shows a simplified block diagram of an exemplary package configuration.

[0014] [Figure 9] 1 is a simplified block diagram illustrating an example rack server having interconnected computing devices using a Universal Chiplet Interconnect Express (UCIe) based protocol.

[0015] [Figure 10] 1 is a simplified block diagram illustrating an example protocol stack for UCIe.

[0016] [Figure 11]1 is a simplified block diagram of an exemplary physical layer block.

[0017] [Figure 12A] 1 is a simplified block diagram of an exemplary protocol stack implementation. [Figure 12B] 1 is a simplified block diagram of an exemplary protocol stack implementation. [Figure 12C] 1 is a simplified block diagram of an exemplary protocol stack implementation.

[0018] [Figure 13] 1 is a simplified block diagram of a computing system including two packages connected by a UCIe-based interconnect.

[0019] [Figure 14] 1 is a representation of an example register for use with a UCIe-based link.

[0020] [Figure 15A] 1 is a diagram illustrating an example flow for initialization of a UCIe-based link. [Figure 15B] 1 is a diagram illustrating an example flow for initialization of a UCIe-based link. [Figure 15C] 1 is a diagram illustrating an example flow for initialization of a UCIe-based link.

[0021] [Figure 16] 1 is a diagram illustrating an example decision tree for selecting a particular flit format for use with a UCIe-based link.

[0022] [Figure 17A] 1 illustrates an exemplary data format for use with a UCIe-based link. [Figure 17B] 1 illustrates an exemplary data format for use with a UCIe-based link. [Figure 17C] 1 illustrates an exemplary data format for use with a UCIe-based link. [Figure 17D] 1 illustrates an exemplary data format for use with a UCIe-based link. [Figure 17E] 1 illustrates an exemplary data format for use with a UCIe-based link. [Figure 17F] 1 illustrates an exemplary data format for use with a UCIe-based link.

[0023] [Figure 18A] 1 illustrates an exemplary sideband message format. [Figure 18B] 1 illustrates an exemplary sideband message format. [Figure 18C] 1 illustrates an exemplary sideband message format. [Figure 18D] 1 illustrates an exemplary sideband message format.

[0024] [Figure 19] FIG. 1 illustrates an embodiment of a block diagram for a computing system including a multi-core processor.

[0025] [Figure 20] 1 illustrates a block diagram of an embodiment of a computing system including multiple processors. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0026] In the following description, numerous specific details are described to provide a thorough understanding of the present disclosure, such as examples of specific types of processors and system configurations, specific hardware structures, specific design and micro-design details, specific register configurations, specific instruction types, specific system components, specific dimensions / heights, specific processor pipeline stages and operations, etc. However, it will be apparent to those skilled in the art that these specific details need not be employed to implement the solutions provided in the present disclosure. In other instances, well-known components or methods, such as specific and alternative processor architectures, specific logic circuits / code for the described algorithms, specific firmware code, specific interconnect operations, specific logic configurations, specific manufacturing techniques and materials, specific compiler implementations, specific expressions for algorithms in the code, specific power-down and gating techniques / logic, and other specific operational details of computer systems, have not been described in detail to avoid unnecessarily obscuring the present disclosure.

[0027] Although the following embodiments may be described with respect to a particular integrated circuit, such as a computing platform or a microprocessor, other embodiments are applicable to other types of integrated circuits and logic devices. Similar techniques and teachings of the embodiments described herein may be applied to other types of circuits and semiconductor devices. For example, the disclosed embodiments are not limited to desktop computer systems or Ultrabooks, but may also be used in other devices, such as handheld devices, tablets, other thin notebooks, system-on-a-chip (SOC) devices, and embedded applications. Some examples of handheld devices include cellular phones, Internet Protocol devices, digital cameras, personal digital assistants (PDAs), and handheld PCs. Embedded applications typically include microcontrollers, digital signal processors (DSPs), system-on-a-chips, network computers (NetPCs), set-top boxes, network hubs, wide area network (WAN) switches, or any other system capable of performing the functions and operations taught below. The techniques and teachings of the embodiments described herein may also be applied at the server level, including rack-scale server platforms, blade server platforms, and other server architectures. Additionally, the apparatus, methods, and systems described herein are not limited to physical computing devices, but may also relate to software emulation and simulation of physical systems, such as the architectures discussed in the examples below. As will become readily apparent in the following description, the method, apparatus, and system embodiments described herein (whether referring to hardware, firmware, software, or a combination thereof) are important in enabling computing platforms to process increasing data rates while balancing power usage and physical footprint, among other example considerations.

[0028] As computing systems advance, the components therein become more complex. As a result, the interconnect architectures for coupling and communicating between components also become more complex to ensure that bandwidth requirements for optimal component operation are met. Furthermore, different market segments require different aspects of the interconnect architecture to meet market needs. For example, servers require higher performance while the mobile ecosystem may sometimes sacrifice overall performance for power savings. However, providing the highest possible performance with maximum power savings is the sole purpose of most fabrics. A number of interconnects described below could potentially benefit from aspects of the solutions described herein.

[0029] Modern interconnect fabric architectures enable components and devices from different vendors to interoperate in an open architecture that spans multiple market segments, client (desktop and mobile), server (standard and enterprise), and embedded communications devices. For example, high-performance general-purpose I / O interconnect protocols (e.g., Peripheral Component Interconnect Express (PCIe)) are being defined for a wide variety of future computing and communications platforms. Such protocols and corresponding architectures can leverage advances in point-to-point interconnects, switch-based technologies, and packetized protocols to provide new levels of performance and features. By way of example, power management, quality of service (QoS), hot-plug / hot-swap support, data integrity, and error handling are some of the advanced features supported by PCI Express.

[0030] Referring to FIG. 1, an embodiment of a fabric composed of point-to-point links interconnecting a set of components is shown. System 100 includes processor 105 and system memory 110 coupled to controller hub 115. Processor 105 includes any processing element, such as a microprocessor, host processor, embedded processor, co-processor, or other processor. Processor 105 is coupled to controller hub 115 through front side bus (FSB) 106. In one embodiment, FSB 106 is a serial point-to-point interconnect, as described below. In another embodiment, link 106 includes a serial differential interconnect architecture that conforms to a different interconnect standard. The interconnect protocols and features discussed below may be utilized to implement the fabric and links that couple the set of components introduced herein in FIG. 1.

[0031] System memory 110 includes any memory device, such as random access memory (RAM), non-volatile (NV) memory, or other memory accessible by multiple devices in system 100. System memory 110 is coupled to controller hub 115 through memory interface 116. Examples of memory interfaces include a double data rate (DDR) memory interface, a dual channel DDR memory interface, and a dynamic RAM (DRAM) memory interface.

[0032] In one embodiment, controller hub 115 may be implemented as a root hub, root complex, or root controller in a PCIe interconnect hierarchy. Examples of controller hub 115 include a chipset, a memory controller hub (MCH), a northbridge, an interconnect controller hub (ICH), a southbridge, and a root controller / hub. The term chipset often refers to two physically separate controller hubs, a memory controller hub (MCH) coupled to an interconnect controller hub (ICH). Note that current systems often include an MCH integrated into processor 105, but controller 115 communicates with I / O devices in a similar manner to that described below. In some embodiments, peer-to-peer routing is optionally supported through root complex 115.

[0033] Here, controller hub 115 is coupled to switch / bridge 120 via serial link 119. Input / output modules 117 and 121, which may also be referred to as interfaces / ports 117 and 121, contain / implement a layered protocol stack to provide communication between controller hub 115 and switch 120. In one embodiment, multiple devices may be coupled to switch 120.

[0034] The switch / bridge 120 routes packets / messages from upstream devices 125, i.e., one level up towards the root complex, to downstream controller hub 115, i.e., one level down away from the root controller, from processor 105 or system memory 110 to devices 125. In one embodiment, the switch 120 is referred to as a logical assembly of multiple virtual PCI-to-PCI bridge devices. Devices 125 include any internal or external device or component coupled to an electronic system, such as I / O devices, network interface controllers (NICs), add-in cards, audio processors, network processors, hard drives, storage devices, CD / DVD ROMs, monitors, printers, mice, keyboards, routers, portable storage devices, Firewire devices, Universal Serial Bus (USB) devices, scanners, and other input / output devices. In PCIe terminology, such devices are often referred to as endpoints. Although not specifically shown, devices 125 may include PCIe-to-PCI / PCI-X bridges to support legacy or other version PCI devices. Endpoint devices in PCIe are often categorized as legacy, PCIe or root complex integrated endpoints.

[0035] The graphics accelerator 130 is also coupled to the controller hub 115 via a serial link 132. In one embodiment, the graphics accelerator 130 is coupled to an MCH which is coupled to the ICH. The switch 120, and thus the I / O devices 125, are then coupled to the ICH. The I / O modules 131 and 118 also implement a layered protocol stack for communicating between the graphics accelerator 130 and the controller hub 115. As with the MCH described above, the graphics controller or graphics accelerator 130 itself may be integrated into the processor 105. Additionally, one or more links (e.g., 123) of the system may include one or more extension devices (e.g., 150), such as, for example, retimers, repeaters, etc.

[0036] Turning now to FIG. 2, an embodiment of a layered protocol stack is shown. The layered protocol stack 200 includes any form of layered communication stack, such as a Quick Path Interconnect (QPI) stack, a PCIe stack, a Compute Express Link (CXL) stack, or other high performance computing interconnect stack, or other layered stack. The following description of FIGS. 1-4 is in terms of a general interconnect protocol stack (e.g., PCIe), but the same concepts may be applied to other interconnect stacks. In one embodiment, the protocol stack 200 is a protocol stack including a transaction layer 205, a link layer 210, and a physical layer 220. Interfaces such as interfaces 117, 118, 121, 122, 126, and 131 in FIG. 1 may be represented as a communication protocol stack 200. The representation as a communication protocol stack may also be referred to as a module or interface that implements / contains the protocol stack.

[0037] Protocols may communicate information between components using packets. Packets are formed in the transaction layer 205 and data link layer 210 to carry information from a sending component to a receiving component. As transmitted packets flow through other layers, they are extended with additional information required for processing the packets at those layers. At the receiving end, reverse processing occurs, converting packets from their physical layer 220 representation to their data link layer 210 representation, and finally (for transaction layer packets) to a form that can be processed by the transaction layer 205 of the receiving device.

[0038] Transaction Layer

[0039] In one embodiment, transaction layer 205 provides an interface between the processing core of the device and the interconnect architecture, such as data link layer 210 and physical layer 220. In this regard, the primary role of transaction layer 205 is the assembly and disassembly of packets (i.e., transaction layer packets or TLPs). Translation layer 205 typically manages credit-based flow control for TLPs. For example, the protocol may implement split transactions, i.e., multiple transactions with a request and a response separated by time, allowing the link to carry other traffic while the target device collects data for the response. In some implementations, flow control may be implemented using a credit-based flow control scheme. In this scheme, the device advertises an initial amount of credit for each of multiple receive buffers in transaction layer 205. An external device at the other end of the link, such as controller hub 115 of FIG. 1, counts the number of credits consumed by each TLP. If the transaction does not exceed the credit limit, the transaction may be sent. Upon receiving the response, the credit amount is restored. The advantage of the credit scheme is that the latency of credit return does not impact performance unless a credit limit is encountered.

[0040] In one embodiment, the four transaction address spaces include a configuration address space, a memory address space, an input / output address space, and a message address space. Memory space transactions include one or more of read and write requests that transfer data to / from memory mapped locations. In one embodiment, memory space transactions can use two different address formats, e.g., a short address format, such as a 32-bit address, or a long address format, such as a 64-bit address. Configuration space transactions are used to access the configuration space of devices that support the protocol. Transactions to the configuration space include read and write requests. Message space transactions (or simply messages) are defined to support in-band communication between protocol agents that implement the protocol on the device. Thus, in one embodiment, the transaction layer 205 assembles the packet header / payload 206.

[0041] Link Layer

[0042] The link layer 210, also referred to as the data link layer 210, acts as an intermediate stage between the transaction layer 205 and the physical layer 220. In one embodiment, the role of the data link layer 210 is to provide a reliable mechanism for exchanging transaction layer packets (TLPs) between two components on a link. One side of the data link layer 210 accepts the TLPs assembled by the transaction layer 205, applies a packet sequence identifier 211, i.e., an identification number or packet number, calculates and applies an error detection code, i.e., CRC 212, and sends the modified TLPs to the physical layer 220 for transmission across the physical device to an external device.

[0043] physical layer

[0044] In one embodiment, physical layer 220 includes logical sub-block 221 and electrical sub-block 222 for physically transmitting packets to an external device, where logical sub-block 221 is responsible for the “digital” functions of physical layer 220. In this regard, logical sub-block 221 includes a transmit section for preparing outgoing information for transmission by physical sub-block 222 and a receive section for identifying and preparing received information before passing the received information to link layer 210.

[0045] The physical block 222 includes a transmitter and a receiver. The transmitter is provided with a number of symbols by the logical sub-block 221, which the transmitter serializes and transmits to a physical transmission medium that connects the transmitter to another external device. The receiver is provided with serialized symbols from an external device and converts the received signal into a bit stream. The bit stream is deserialized and provided to the logical sub-block 221. In one embodiment, an 8b / 10b transmission code is employed, where 10-bit symbols are transmitted / received. In other cases, 128b / 130b encoding may be utilized, while in still other implementations, flit-based transmission may be utilized, among other examples. In one example, a special symbol is used to frame the packets with the frame 223. Additionally, in one example, the receiver also provides a symbol clock recovered from the incoming serial stream.

[0046] The transmission medium may implement any transmission path for transmitting data, such as a transmission line, a copper line, an optical line, a wireless communication channel, an infrared communication link, or other communication path. The connection between two devices is referred to as a link. A link may be implemented using one or more constituent transmission paths or lanes. To scale the bandwidth, a link may aggregate multiple lanes, denoted by xN, where N is any supported link width, such as 1, 2, 4, 8, 12, 16, 32, 64, or more. In some implementations, differential signaling may be utilized, where a differential pair refers to two transmission paths for transmitting differential signals. As an example, in a differential pair, when a first line in the pair switches from a low voltage level to a high voltage level, i.e., a rising edge, the other line in the pair drives from a high logic level to a low logic level, i.e., a falling edge. Differential signals potentially exhibit better electrical characteristics, such as better signal integrity, i.e., cross-coupling, voltage overshoot / undershoot, ringing, etc. This allows for better timing windows and therefore faster transmission frequencies.

[0047] As mentioned above, the transaction layer 205, the link layer 210, and the physical layer 220 are described with respect to a specific embodiment (e.g., a conventional PCIe protocol stack), but the layered protocol stack is not so limited. In fact, any layered protocol may be included / implemented. By way of example, a port / interface may include logic implemented in hardware circuitry and / or firmware to implement various layers in a layered protocol, such as: (1) a first layer (e.g., a transaction or protocol layer) for assembling packets; a second layer (e.g., a link or data link layer) for sequencing packets; and a third layer (e.g., a physical layer) for transmitting packets, including layers of protocols discussed in more detail below.

[0048] In some implementations, the interconnect protocol may implement a cache coherent link. As an example, the Ultra-Path Interconnect™ (UPI™) may be used in high performance computing platforms, such as workstations or servers, in systems where PCIe or another interconnect protocol is typically used to connect processors, accelerators, I / O devices, etc. However, UPI is not so limited. Instead, UPI may be used in any system or platform described herein. Additionally, the ideas developed separately may be applied to other interconnects and platforms, such as PCIe, MIPI, QPI, etc.

[0049] To support multiple devices, in one exemplary implementation, the UPI can be instruction set architecture (ISA) independent (i.e., the UPI can be implemented in multiple different devices). In another scenario, the UPI can also be used to connect to high performance I / O devices, rather than simply to processors and accelerators. For example, a high performance PCIe device can be coupled to the UPI through an appropriate conversion bridge (i.e., UPI to PCIe). Furthermore, multiple UPI links can be used by multiple UPI-based devices, such as multiple processors, in various ways (e.g., star, ring, mesh, etc.). Indeed, interconnect protocols such as the UPI and others discussed herein can be used to facilitate interconnections between dies or packages. For example, FIG. 3 shows exemplary implementations of various potential multi-socket configurations. As shown, a two-socket configuration 305 can include two links, although in other implementations, one link can be used. For larger topologies, any configuration may be used as long as identifiers (IDs) can be assigned and some form of virtual path exists, among other additional or alternative features. As shown, in one example, a four-socket configuration 310 has links from each processor to another processor. However, in an eight-socket implementation shown in configuration 315, not all sockets are directly connected to each other through their respective links. However, such configurations are supported where virtual paths or channels exist between multiple processors. The range of supported processors includes 2 to 32 in the native domain. Larger numbers of processors may be achieved through the use of multiple domains or other interconnects between multiple node controllers, among other examples.

[0050] Like other interconnect architectures, the UPI architecture includes the definition of a layered protocol architecture, which in some examples includes multiple protocol layers (coherent, non-coherent, and optionally other memory-based protocols), a routing layer, a link layer, and a physical layer. In some implementations, each layer in the protocol stack may build and / or process information at their own level of granularity or quantum (e.g., a protocol layer using packets, a link layer using flits, a physical layer using phits or symbols, among other examples). Note that in some embodiments, a packet may include a partial flit, a single flit, or multiple flits, depending on the implementation.

[0051] In another example interconnect protocol, the Compute Express Link (CXL) interconnect protocol may be utilized to provide improved high-speed CPU-device and CPU-memory interconnects designed to accelerate next-generation data center performance, among other applications. CXL maintains memory coherency between the CPU memory space and memory on attached devices. This allows for resource sharing for higher performance, reduces software stack complexity, and lowers overall system costs, among other example benefits. CXL enables communication between a host processor (e.g., CPU) and a set of workload accelerators (e.g., Graphics Processing Units (GPUs), Field Programmable Gate Array (FPGA) devices, Tensor and Vector Processing Units, Machine Learning Accelerators, Specialized Accelerator Solutions, among other examples). Indeed, CXL is designed to provide a standard interface for high-speed communication as accelerators are increasingly used to complement CPUs in supporting new computing applications, such as artificial intelligence, machine learning, and other applications.

[0052] The CXL link can be a low-latency, high-bandwidth discrete or on-package link that supports dynamic protocol multiplexing of coherency, memory access, and input / output (I / O) protocols. Among other uses, the CXL link can enable accelerators to access system memory as a caching agent and / or host system memory, among other examples. CXL is a dynamic multi-protocol technology designed to support a vast spectrum of accelerators. Over a discrete or on-package link, CXL offers a rich set of protocols including I / O semantics similar to PCIe (CXL.io), cache protocol semantics (CXL.cache), and memory access semantics (CXL.mem). Based on the particular accelerator usage model, all of the CXL protocols or only a subset of these protocols can be enabled. In some implementations, CXL may be built on the well-established and widely adopted PCIe infrastructure (e.g., PCIe 5.0) and leverage the PCIe physical and electrical interface to provide advanced protocols for areas including I / O, memory protocols (e.g., allowing a host processor to share memory with accelerator devices) and coherency interfaces.

[0053] 4, a simplified block diagram 400 is shown illustrating an exemplary system utilizing a CXL link 450. For example, the link 450 may interconnect a host processor 405 (e.g., a CPU) to an accelerator device 410. In this example, the host processor 405 includes one or more processor cores (e.g., 415a-b) and one or more I / O devices (e.g., 418). A host memory (e.g., 460) may be provided with the host processor (e.g., on the same package or die). The accelerator device 410 may include accelerator logic 420 and, in some implementations, may include its own memory (e.g., accelerator memory 465). In this example, the host processor 405 may include circuitry for implementing coherence / cache logic 425 and interconnect logic (e.g., PCIe logic 430). CXL multiplexing logic (e.g., 455a-b) may also be provided to enable multiplexing of CXL protocols (e.g., I / O protocols 435a-b (e.g., CXL.io), cache protocols 440a-b (e.g., CXL.cache), and memory access protocols 445a-b (CXL.mem)). This allows data of any one of the supported protocols (e.g., 435a-b, 440a-b, 445a-b) to be sent in a multiplexed manner over link 450 between host processor 405 and accelerator device 410.

[0054] In some implementations of CXL, Flex Bus ports may be utilized in conjunction with CXL-compliant links to flexibly adapt the device to interconnect with a wide variety of other devices (e.g., other processor devices, accelerators, switches, memory devices, etc.). Flex Bus ports are flexible high-speed ports that are statically configured to support either PCIe or CXL links (and potentially links of other protocols and architectures). Flex Bus ports allow designers to choose to provide either native PCIe protocol or CXL over a high-bandwidth off-package link. The selection of the protocol applied at the port may occur during boot time via auto-negotiation and may be based on the device plugged into the slot. Flex Bus uses PCIe circuitry to be compatible with PCIe retimers and adheres to the PCIe form factor, which is a standard for add-in cards.

[0055] FIG. 5 is a simplified block diagram illustrating an example port architecture 500 (e.g., Flex Bus) utilized to implement a CXL link. For example, the Flex Bus architecture may be organized as multiple layers for implementing multiple protocols supported by the port. For example, a port may include transaction layer logic (e.g., 505), link layer logic (e.g., 510), and physical layer logic (e.g., 515) (e.g., implemented in whole or in part in a circuit). For example, the transaction (or protocol) layer (e.g., 505) may be subdivided into transaction layer logic 525 that implements PCIe transaction layer 555 and CXL transaction layer enhancements 560 (for CXL.io) of the base PCIe transaction layer 555, and logic 530 for implementing cache (e.g., CXL.cache) and memory (e.g., CXL.mem) protocols for the CXL link. Similarly, link layer logic 535 may be provided to implement a base PCIe data link layer 565 and a CXL link layer (for CXL.io), which represents an enhanced version of PCIe data link layer 565. CXL link layer 510 may also include cache and memory link layer enhancement logic 540 (e.g., CXL.cache and CXL.mem).

[0056] Continuing with the example of FIG. 5, the CXL link layer logic 510 may interface with a CXL arbitration / multiplexing (ARB / MUX) logic 520 that interleaves traffic from two logic streams (e.g., PCIe / CXL.io and CXL.cache / CXL.mem), among other example implementations. During link training, the transaction layer and link layer are configured to operate in either PCIe mode or CXL mode. In some cases, among other examples, a host CPU may support implementation of either PCIe mode or CXL mode, while other devices such as accelerators may only support CXL mode. In some implementations, a port (e.g., a Flex Bus port) may utilize a physical layer 515 based on a PCIe physical layer (e.g., PCIe electrical circuit PHY 550). For example, the Flex Bus physical layer may be implemented as a converged logical physical layer 545 that can operate in either PCIe mode or CXL mode based on the results of alternative mode negotiation during the link training process. In some implementations, the physical layer may support multiple signaling rates (e.g., 8GT / s, 16GT / s, 32GT / s, etc.) and multiple link widths (e.g., x16, x8, x4, x2, x1, etc.). In PCIe mode, the link implemented by port 500 may be fully compliant with native PCIe features (e.g., as defined in the PCIe specification), while in CXL mode, the link supports all features defined for CXL. Thus, among other examples, the Flex Bus port may provide a point-to-point interconnect capable of transmitting native PCIe protocol data or dynamic multi-protocol CXL data to provide I / O protocols, coherency protocols, and memory protocols over PCIe circuitry.

[0057] The CXL I / O protocol, CXL.io, provides a non-coherent load / store interface to I / O devices. Transaction types, transaction packet formats, credit-based flow control, virtual channel management, and transaction ordering rules in CXL.io may follow all or part of the PCIe definition. The CXL cache coherency protocol, CXL.cache, defines the interaction between devices and hosts as multiple requests, each with at least one associated response message and possibly a data transfer. The interface consists of three channels in each direction: request, response, and data.

[0058] The CXL memory protocol, CXL.mem, is a transaction interface between processors and memory that uses the CXL physical and link layers when communicating between dies. Among other examples, CXL.mem can be used for a number of different memory mounting options, including where the memory controller is located in the host CPU, where the memory controller is in an accelerator device, or where the memory controller is moved to a memory buffer chip. Among other example features, CXL.mem can be applied to transactions involving different memory types (e.g., volatile, persistent, etc.) and configurations (e.g., flat, hierarchical, etc.). In some implementations, the host processor's coherency engine can interface with the memory using CXL.mem requests and responses. In this configuration, the CPU coherency engine is considered a CXL.mem Master and the Mem device is considered a CXL.mem Subordinate. The CXL.mem Master is the agent responsible for sourcing CXL.mem requests (e.g., reads, writes, etc.), and the CXL.mem Subordinate is the agent responsible for responding to CXL.mem requests (e.g., data, completion, etc.). When a Subordinate is an accelerator, the CXL.mem protocol assumes the presence of a Device Coherency Engine (DCOH). This agent is assumed to be responsible for implementing coherency related functions such as snooping the device cache and updating metadata fields based on CXL.mem commands. In implementations where metadata is supported by device-attached memory, the metadata may be used by the host to implement a coarse-grained snoop filter for the CPU socket, among other example uses.

[0059] In some implementations, an interface may be provided for coupling a circuit or other logic (e.g., an intellectual property (IP) block or other hardware element) implementing a link layer (e.g., 510) to a circuit or other logic (e.g., an IP block or other hardware element) implementing at least a portion of a physical layer (e.g., 515) of a protocol. For example, an interface based on the Logical PHY Interface (LPIF) specification for defining a common interface between a link layer controller, module, or other logic and a module implementing a logical physical layer ("logical PHY" or "logPHY") facilitates interoperability, design, and validation reuse between one or more link layers and physical layers for an interface to a physical interconnect, such as in the example of FIG. 5. Additionally, as in the example of FIG. 5, an interface may be implemented with logic (e.g., 535, 540) for simultaneously implementing and supporting multiple protocols. Furthermore, in such implementations, an arbitration and multiplexer layer (e.g., 520) may be provided between the link layer (e.g., 510) and the physical layer (e.g., 515). In some implementations, each block in a multiple protocol implementation (e.g., 515, 520, 535, 540) may interface with the other blocks via an independent LPIF interface (e.g., 580, 585, 990). In cases where branching is supported, each branch port may also have its own independent LPIF interface, among other examples.

[0060] While the examples discussed herein may reference the use of an LPIF-based link layer logical PHY interface, it should be understood that the details and principles discussed herein may apply equally to non-LPIF interfaces. Similarly, while some examples may reference the use of a common link layer logical PHY interface to couple a PHY to a controller to implement CXL or PCIe, other link layer protocols may also utilize such interfaces. Similarly, while some references may be made to a Flex Bus physical layer, other physical layer logic may be similarly utilized in some implementations, utilizing a common link layer logical PHY interface such as that discussed herein, among other example variations that are within the scope of this disclosure.

[0061] Conventional die-to-die interconnects are either vendor-specific or application-specific (e.g., HBM connections are used to connect memory-on-package). Some die-to-die interconnects (e.g., Intel's AIB, HBI, and OCP Consortium's BoW) define only the physical layer and do not provide a mechanism to ensure interoperability between dies. In fact, there is no current solution for implementing a generic die-to-die interconnect that can be used for seamless interoperability between dies, providing an open innovation slot on the package where engineers can innovate. An improved interconnect architecture is introduced herein for implementing a standardized die-to-die interface through the Universal Chiplet Interconnect Express (UCIe) protocol. Not only does UCIe enable a solution for a generic die-to-die interconnect for on-package and off-package bonding of dies, but the standardization of the interface enables the interconnection of different devices from different vendors and different fabs across different technology nodes, using different package options, to improve existing computing systems and implement new systems.

[0062] UCIe provides high bandwidth, low latency, power efficient, cost-effective on-package connectivity between chiplets. It addresses the expected growth in computing, memory, storage, and connectivity demands across the entire computing continuum spanning cloud, edge, enterprise, 5G, automotive, high performance computing, and handheld segments. While Moore's law is a fact and has allowed computing blocks to become smaller and more powerful, the increase in on-package integration of multiple dies in mainstream commercial offerings such as client CPUs, server CPUs, and GP-GPUs has led to larger die sizes to meet the growing performance demands. However, this phenomenon has resulted in designs encountering die reticle limits. Examples include multi-core CPUs with hundreds of cores, or very large fan-out switches. Even when a die can fit within the reticle limit, multiple smaller dies connected in a package can be preferable for yield optimization and die reuse across multiple market segments. On-package connectivity on the same die enables these scale-up applications. Additionally, chiplet integration on a package also allows designers to make different tradeoffs for different market segments by selecting different numbers and types of dies. For example, different numbers of compute, memory, and I / O dies can be selected depending on the needs of the segment. This results in lower production SKU costs since there is no need to make different die designs for different segments.

[0063] On-package integration of chiplets enables a fast and cost-effective means to provide dedicated solutions. For example, different applications may have the same memory, cores, and I / O but require different acceleration capabilities. It also enables co-packaging of dies where optimal process node selection is made based on functionality. For example, memory, logic, analog, and co-packaging optics each require different process technologies that can be packaged with the chiplet. Applications requiring high bandwidth such as memory access (e.g., high bandwidth memory) are implemented as on-package integration because the package traces are short and provide high density routing.

[0064] UCIe is an open, multi-protocol, on-package interconnect standard for connecting multiple dies on the same package. UCIe enables the development of a vibrant ecosystem supporting disaggregated die architectures that can be interconnected using UCIe. UCIe supports multiple protocols, such as PCIe, CXL, Advanced Extensible Interface (AXI), UPI, and others, and a raw mode that can be used to map any protocol of choice over a common physical and link layer (as long as both link partners support it). UCIe can additionally encompass elements for system-on-chip (SoC) construction, such as application layer and package-related form factors (e.g., bump locations, power delivery, thermal solutions, etc.), among other features. UCIe features act to ensure interoperability across a wide range of devices with different performance characteristics. Well-defined debug and compliance mechanisms are provided to ensure interoperability. UCIe can additionally enable support of devices in a backward-compatible manner.

[0065] UCIe supports a wide range of usage models, but a subset is provided here as an illustrative example. As described above, some protocols may be explicitly mapped onto UCIe, such as PCIe and CXL. Such protocols may be mapped onto UCIe flit formats, including raw mode. By way of example, by replacing the PCIe SERDES PHY and PCIe / CXL logical PHY with, for example, link level retries, widely used protocols such as PCIe and CXL may be mapped onto UCIe for more on-package integration, improving power and enhancing performance, among other example features. UCIe raw mode is protocol agnostic, allowing other protocols to be mapped ad-hoc by the device, while enabling uses such as integrating standalone SERDES / transceiver tiles (e.g., Ethernet) on-package, among other example features.

[0066] 6 is a simplified block diagram 600 illustrating an example implementation of an SoC package 605 including multiple processor dies (e.g., 610, 615), accelerator dies (e.g., 620), and input / output (I / O) tiles (e.g., 625) interconnected using multiple UCIe links (e.g., 630, 635, 640, 645). In some implementations, the accelerator 620 and / or I / O tiles 625 may be connected to the CPU devices 610, 615 using CXL transactions running over UCIe, thereby leveraging the I / O, coherency, and memory protocols of CXL. The I / O tiles may provide external CXL, PCIe, and DDR pins (e.g., 650, 655) of the package 605. The accelerators may also be connected to the CPU using PCIe transactions running over UCIe. On-package inter-CPU connectivity may also use a UCIe interconnect implementing a coherency protocol, among other examples. In some implementations, all components on an example SoC package may be interconnected using UCIe links. In other cases, one or more blocks (e.g., memory blocks 660, 665, 670, 675) may be connected using other protocols or links (e.g., DDR, etc.), among other example implementations.

[0067] A variety of packages may support UCIe links. For example, standard two-dimensional (2D) packages, as well as 2.5D and 3D advanced package options, among other examples, may be utilized. FIG. 7 is a simplified block diagram illustrating an exemplary application using a standard 2D package. 2D packages may be used for low-cost, long-reach (e.g., 10 mm to 25 mm) interconnects using traces 705, 708 on an organic package / substrate 710, yet provide significantly better BER performance compared to off-package SERDES. As shown in FIG. 7, a first die 715 may be coupled to a second die 720 and a third die 725 by respective links implemented using traces (e.g., 705, 708) on the substrate 710.

[0068] 8A-8D are simplified block diagrams 800a-d illustrating an example application using the UCIe Advanced (e.g., 2D) option. Such packaging technology can be used for performance-optimized applications. As a result, the channel reach is expected to be short (e.g., less than 2 mm) and the interconnect is expected to be optimized for high bandwidth and low latency with best performance and power efficiency characteristics. In one example, the links can be implemented on silicon bridges 805, 810 (e.g., Embedded Multi-Die Interconnect Bridge (EMIB)) as shown in FIG. 8A. In the example of FIG. 8B, an interposer 815 on package 710 and UCIe links (e.g., 820, 825) can be provided on the interposer 815 (e.g., chip-on-wafer-on-substrate (CoWoS) application). In yet another example of an advanced package, shown in FIG. 8C, silicon bridges (e.g., 830, 835) may be provided in a fan-out organic interposer 840 provided on the package 710, and UCIe links may be implemented in the silicon bridges 830, 835. FIG. 8D illustrates another example where a combination of standard and advanced options is utilized. For example, a base die 850 may be coupled to a companion die 855 in a 2D manner by a first link utilizing standard package traces 860, while the base die 850 is coupled to another base die 865 by a bridge-based link 870. In another example, the base die 865 couples to a vertical chiplet 875 via a UCIe link in 3D, among other example architectures and package applications.

[0069] UCIe supports two broad usage models. The first usage model introduced above involves package-level integration to provide power-efficient, cost-effective performance. Thus, components attached at the board level, such as memory, accelerators, networking devices, modems, etc., can be integrated at the package level, with applicability from handhelds to high-end servers with dies from multiple sources connected through different package options even on the same package. The second use is to provide off-package connectivity using different types of media (e.g., optical, electrical cable, mmWave), using UCIe retimers to transport underlying protocols (e.g., PCIe, CXL) at the rack or even pod level, enabling resource pooling, resource sharing, and even message passing, using load-store semantics from the node level to beyond the rack / pod level to obtain more power-efficient, cost-effective performance at the edge and data center. FIG. 9 is a simplified block diagram illustrating an exemplary rack server system (e.g., UCIe can be utilized to facilitate off-package connectivity, including server-scale interconnects between devices). In such implementations, retimers may be provided to facilitate off-package connectivity (e.g., on top of rack servers (TORS) on-package). For example, among other example implementations, an off-package UCIe link may be utilized to couple a device (e.g., processor device 905) on a first chassis 910 to another device (e.g., processor 915) on a separate second chassis 920 (e.g., via switch device 925).

[0070] UCIe is a layered protocol shown in block diagram 1000 of FIG. 10. Physical layer 1005 is responsible for electrical signaling, clock, link training, and sideband signaling. Die-to-Die (D2D) adapter block 1010 provides link state management and parameter negotiation to the chiplets. It optionally ensures reliable delivery of data through its cyclic redundancy check (CRC) and link level retry mechanisms. When multiple protocols are supported, the D2D adapter defines an underlying arbitration framework for arbitration between data of multiple protocols on the link. In one exemplary implementation, UCIe may provide 256-byte flits (or flow control units) to define the underlying transfer mechanism when the adapter is responsible for reliable transfer. As described above, one or more other interconnect protocols may be natively mapped to the UCIe flit format. In one example, UCIe ensures seamless interoperability by mapping PCIe and CXL protocols and leveraging the existing ecosystem of these protocols. For example, SoC structures, link management, and security solutions already deployed with PCIe and CXL can be leveraged for UCIe. The usage models addressed are also comprehensive: data transfers using direct memory access, software detection, error handling, etc. are addressed with PCIe / CXL.io; memory use cases are handled through CXL.Mem; and cache requirements for applications such as accelerators are addressed with CXL.cache, among other example benefits and use cases. UCIe also defines a "streaming protocol" that can be used to map any other protocol to the flit format defined in UCIe.

[0071] The protocol layer 1015 may potentially follow any interconnect protocol, including protocols that are natively mapped to UCIe (e.g., CXL, PCIe, etc.), other well-defined or standardized interconnect protocols that may utilize UCIe's streaming or raw modes, or proprietary or vendor-defined protocols that may similarly utilize UCIe's streaming or raw modes. The protocol layer may implement the transaction layer of the corresponding protocol and implement flow control schemes defined in the protocol, among other protocol-specific features. The logic block (e.g., software, firmware, and / or hardware) that implements the protocol layer 1015 may interface with the UCIe D2D adapter 1010 using a flit-aware D2D interface (or FDI) 1020 and provide data to the D2D adapter according to the protocol for further processing. Data to be transmitted over the link using the PHY is received from the D2D adapter 1010 by the PHY over the FDI. The FDI interface may be defined in the UCIe protocol specification and may define a set of signals for communication and negotiation between the protocol layer block 1015 and the D2D adapter block 1010. For example, signals or messages may be defined in the FDI 1020 such that the D2D adapter 1010 (e.g., based on negotiation with a link partner device (e.g., a D2D adapter of the link partner device)) informs the protocol layer of the flit format to be utilized (e.g., a mapped flit format, a streaming mode flit format, a raw mode flit format, etc.), which the protocol layer 1015 may use to adapt data of the corresponding protocol to the identified flit format, among other examples.

[0072] Through the provision of streaming and raw modes, UCIe allows the option to transport data of any protocol over the UCIe link. Protocol data can be transported in flit format or as raw fabric wires if needed. Some protocols can be explicitly mapped, allowing on-board components such as accelerators, memory expanders, and I / O expanders to be moved seamlessly on-package. Flit modes include modes of operation where error detection and / or error correction bits / bytes (e.g., CRC) are inserted into the defined flit format by the D2D adapter. Flit modes include modes of operation where flits of another underlying protocol of data (e.g., PCIe, UPI, CXL, GenZ, AXI, etc.) are mapped onto the UCIe flit format, and streaming mode, where data of any other potential protocol is encoded or populated into the flit format (e.g., including CRC and any other specified fields defined in the UCIe flit). Raw mode, on the other hand, is a mode of operation in which all bytes of the UCIe flit are populated by the protocol layer (and the D2D adapter accepts the flit as is, without adding CRC, retries, or other values ​​to the flit, and forwards the data from the protocol layer to the physical layer without any modification). In some cases, CRC, retries, FEC, etc. in raw mode may be handled entirely by the protocol layer (rather than the UCIe D2D adapter), among other examples.

[0073] In some implementations, the D2D adapter 1010 may be provided as a separate block of circuitry (e.g., from those implementing the protocol layer block and / or the physical layer block 1005). In other implementations, one or both of the logic implementing the protocol layer or the physical layer may be co-located or integrated with the logic implementing the D2D adapter block 1010 (e.g., hardware circuitry), among other example implementations. The D2D adapter 1010 coordinates with the protocol layer 1015 (e.g., via the FDI 1020) and the physical layer 1005 (e.g., via the raw D2D interface (RDI) 1025) to ensure successful and reliable data transfer across the UCIe link. Each of these standardized interfaces (e.g., the FDI and RDI) may be standardized through a defined set of pins (for the physical interconnection of the respective protocol stack blocks), a defined set of signals, states, and state transitions to standardize the manner in which the respective protocol stack blocks interact and coordinate. Providing such standardized interfaces (e.g., FDI and RDI) for coupling blocks (e.g., IP blocks) implementing each layer of the protocol allows vendors and SoC builders to easily mix and match different layers from different IP providers with lower integration costs and faster time to market (e.g., allowing protocol layer blocks to work with D2D adapters and physical layer blocks from any different vendors that comply with the interface handshakes provided herein). Furthermore, considering that interoperability testing in post-silicon has greater overhead and costs associated with it, consistent understanding and development of bus functional models (BFMs) based on such interfaces may enable easier IP development for such UCIe protocol stack implementations, among other exemplary benefits.

[0074] The D2D adapter 1010 helps facilitate minimization of logic in the main data path and provides a highly latency optimized data path for the protocol flits. The D2D adapter may include logic to implement various functions that may be utilized by various protocols (of the protocol layer 1015), such as arbitration and multiplexer functions, error detection and / or error correction codes (e.g., cyclic redundancy check (CRC)), retry schemes, among other example features. As an example, a protocol layer implementing CXL may be coupled to the D2D adapter. When transporting the CXL protocol, the ARB / MUX functionality of the D2D adapter 1010 may be utilized to perform arbitration / multiplexing of multiple sub-protocols of CXL (e.g., CXL.mem, CXL.io, CXL.cache). The error detection, error correction, retry, and other functions provided by the D2D adapter 1010 may be utilized to support target bit error rate (BER) requirements (e.g., based on the speed implemented on the link). For example, if the raw BER is less than 1e-27, a CRC and retry scheme may be provided by the D2D adapter 1010. Additionally, the D2D adapter 1010 may be responsible for higher level link state machine and link bring-up protocols, coordination of protocol options for parameter exchange with a remote link partner, and, when supported, power management coordination with a remote link partner. For example, two link partner devices coupled using a UCIe link may utilize their respective D2D adapters to perform link training (e.g., by progressing through various link states defined in a link state machine) and determination of parameters to be used during operation of the link, among other examples.

[0075] A communication port of a computing device may be constructed from a protocol logic block, a D2D adapter block, and a physical layer block. The protocol logic block may interface with the D2D adapter block via an FDI interface, and the D2D adapter may interface with the physical layer block using an RDI interface. Each of the FDI and RDI interfaces may be implemented using physical pins, traces, conductors, or other communication paths along which the respective signals of the defined interfaces may be communicated. The protocol logic block (e.g., 1015), the D2D adapter block (e.g., 1010), and the physical layer block (e.g., 1015) implement the respective layers of the UCIe protocol, enabling those devices to establish a UCIe compliant link and communicate with another link partner computing device using the UCIe protocol.

[0076] The physical layer of the UCIe stack may be implemented through multiple physical layer modules to form the physical layer of the corresponding link. The modules form the main data path (e.g., on the pins of a physical bump or port) and are organized as a grouping of lanes. The modules form the indivisible granularity of the architectural design implementation of the UCIe analog front end (AFE). The number of lanes per module may be based on whether a standard or advanced package is utilized for the implementation. A single D2D adapter may be coupled to the physical layer implemented as one or more modules (for bandwidth scaling). The UCIe physical link (and each module that comprises the physical layer) may include a sideband connection (comprised of a first subset of lanes) and a main band connection (comprised of a second subset of the physical layer lanes). The UCIe module / physical layer main band connection provides the main data path in the UCIe. The main band connection may include a transmit clock, a data valid pin, and N data lanes per module. The sideband connection is used for parameter exchange, register access is used for debug / compliance, and coordination with the remote partner is used for link training and management. The sideband connection may include a forwarding clock pin and a data pin in each direction. In one example implementation, the forwarding clock may be fixed at a frequency of 800 MHz (e.g., at a frequency different from the main datapath (main band) rate) regardless of the main datapath rate. In some implementations, the sideband logic for the UCIe physical layer may be powered using auxiliary power, placing the sideband logic in an "always on" domain. Each module may include its own respective set of sideband pins / bumps / lanes.

[0077] 11 is a simplified block diagram 1100 illustrating an example physical layer block of a UCIe device. The physical layer may include a logical physical layer (logical PHY) 1105, or PHY logic, and multiple (M) sideband lanes 1110 and multiple (N) mainband lanes 1115. As described above, the mainband lanes may include one or more forward clock lanes 1120, multiple data lanes 1125, and valid lanes 1130. The logical PHY 1105 may perform link training, lane repair, lane inversion, scrambling / descrambling, sideband training, and forwarding, among other functions for the link. The physical layer may be coupled to a D2D adapter (e.g., implemented on a separate hardware or IP block) via an RDI interface 1025.

[0078] 12A-12C, simplified block diagrams 1200a-c are shown illustrating an example UCIe protocol stack, where different numbers of PHY modules are utilized to implement the respective physical layers of the UCIe protocol stack. In one example, PHY modules may be defined to provide 4 sideband lanes 1110 and 64 data lanes 1125, respectively. In FIG. 12A, an example implementation is shown in which a single module 1205 is used to connect to a D2D adapter 1010 that implements the physical layer of a port. FIG. 12B shows another example implementation in which two modules 1205a-b are utilized to implement the physical layer. Each module 1205a-b has its own logical PHY (e.g., 1105a,b), a set of sideband lanes (e.g., 1110a,b), and a set of mainband lanes (e.g., 1115a,b). The use of two modules 1205a-b effectively doubles the bandwidth of the link compared to an implementation using only a single module, such as in Figure 12A. The two modules 1205a-b connect to a single D2D adapter 1010. In yet another example, as shown in Figure 12C, a physical layer implementation is shown built using four modules 1205a-d to further scale the bandwidth of the link (e.g., by providing four x64 data lane groupings in each of the four modules to form a x320 data width), among other examples (which may include other numbers of defined PHY modules to implement the physical layer for the port).

[0079] According to the above, UCIe can support different data rates, widths, bump pitches, and channel reaches to ensure the widest possible interoperability. A sideband interface is defined to facilitate design and validation. Using PHY modules as the building block of the interconnect, it provides N single-ended, unidirectional, full-duplex data lanes (e.g., N=16 for standard package, N=64 for advanced package), one single-ended lane for enable, one lane for tracking, differential forwarding clock per direction, and four lanes (e.g., two per direction) for sideband signaling (e.g., single-ended, one 800MHz clock, and one data). Multiple clusters can be aggregated to provide higher performance per link.

[0080] In addition to the above features, advanced package options in UCIe may be provided to support still additional features. For example, the advanced package may support spare lanes to handle faulty lanes across pairs (e.g., extra lanes for clocks, enables, sidebands for repair purposes, etc.), where the standard package supports width reduction to handle faults. A die in a standard package design may be expected to interoperate with any other design on a standard package. Similarly, a die in an advanced package design may be expected to interoperate with any other die designed for advanced packages (e.g., even within a wide range of bump pitches from 25u to 55u). In general, the standard package (2D) is used for cost-effective performance. The advanced package may be used for power-efficient performance. Table 1 below summarizes exemplary key performance targets for an exemplary UCIe implementation. [Table 1] Table 1: Exemplary Key Performance Indicators (KPIs) Among other examples, assume that the die edge bandwidth density has a bump pitch of 45 μm (advanced package) and 110 μm (standard package), the energy efficiency includes all PHY related circuitry (e.g., transmitter, receiver, phase locked loop (PLL)), and the latency includes the D2D adapter and the physical layer latency on the transmitter and receiver.

[0081] As introduced above, the retimer can be used to extend UCIe connectivity beyond the package using off-package links (e.g., electrical or optical cabling communication media), for example, to connect packages at the rack / pod level. The UCIe retimer implements a UCIe interface to the die it connects to on its local package to ensure that flits are delivered between two dies on different packages according to the UCIe protocol. UCIe can also be used to transport any protocol, including load-store protocols, such as PCIe or CXL, over an alternative medium with longer reach, such as optical. For example, as one exemplary implementation, rack / pod level disaggregation can be implemented using the CXL 2.0 (or later) protocol. In this example, at the rack level, multiple compute nodes (e.g., in a virtual hierarchy) from different compute chassis connect to a CXL switch that connects to multiple CXL accelerators and / or type 3 memory devices that may be located in one or more separate drawers. In this example, each host may be implemented as a compute drawer that connects to the switch using optical connections running the CXL protocol through UCIe-based optical tiles. Among many other example implementations where UCIe links allow off-package connectivity, the switch may also have co-package optics, where the optical tiles connect to the main switch die using UCIe, while PCIe / CXL physical interconnects are used to connect to accelerator / memory devices.

[0082] 13, a simplified block diagram 1300 is shown illustrating an example system including two packages (e.g., 1310, 1315) connected by a UCIe link 1305. As introduced above, UCIe retimers can be used to enable different types of off-package interconnects 1305 to extend channel reach between two UCIe dies on different packages (e.g., 1310, 1315). A respective UCIe retimer (e.g., 1320, 1325) can be provided in each package to enable an off-package UCIe link (e.g., 1305). Additionally, the UCIe retimer 1320, 1325 of each package can be coupled to one or more of the dies (e.g., 1330, 1335) on that package by respective UCIe links (e.g., 1340, 1345). A die (e.g., 1330, 1335) may transfer data to a corresponding retimer (e.g., 1320, 1325) for communication with other dies on other packages through off-package links 1305, facilitated through the retimer. The off-package links may be significantly longer in physical length as compared to the on-package links (e.g., 1340) that couple the retimer (e.g., 1320) to the on-package die (e.g., 1330). An off-package retimer (e.g., 1325) to which a retimer (e.g., 1320) is connected using off-package links 1305 may be referred to as a "remote retimer partner."

[0083] The responsibilities of each UCIe retimer (e.g., 1320, 1325) in an exemplary system may include reliable transport of flits through the off-package interconnect 1305. In one example, a retimer (e.g., 1320, 1325) is configured to implement and support the use of forward error correction (FEC) and error detection (e.g., cyclic redundancy check (CRC)) natively defined by the underlying specification of the protocol (e.g., PCIe or CXL) it carries, so long as the external interconnect complies with the underlying error model (e.g., BER and error correlation) of the specification corresponding to the protocol it transports. In such an example, a UCIe link may be configured to utilize raw mode to tunnel the native bits of the protocol (e.g., PCIe or CXL flits) it transports. In this scenario, queue sizes (e.g., protocol layer buffers) are adjusted on the UCIe die to meet the underlying round-trip latency. In another example, the retimer can be configured to support and provide defined FEC, CRC, and retry functions to handle the BER of the off-package interconnect. In this case, the flits go over three independent links; each UCIe retimer performs an independent Ack / Nak for retries with the UCIe die in the package and a separate independent Ack / Nak for retries with the remote retimer partner. As another example, the retimer can provide its own FEC by replacing the native protocol's (e.g., PCIe or CXL) defined FEC with itself or by adding its FEC in addition to the native protocol's defined FEC, but utilizing the built-in CRC and replay mechanisms of the underlying protocol. In this example, queue sizes (e.g., protocol layer buffers, retry buffers, etc.) need to be adjusted on the UCIe die to meet the underlying round-trip latency.

[0084] The UCIe retimer may also be responsible for resolving link layer and protocol layer parameters with its remote retimer partner to ensure interoperability between UCIe dies end-to-end (E2E). For example, a retimer (e.g., 1320, 1325) is permitted to enforce the same link width, speed, protocol (including any associated protocol-specific parameters), and flit format for both packages (e.g., 1310, 1315). The specific mechanisms of resolution, including message forwarding for parameter exchange across the off-package interconnect, may be implementation-specific for the retimers, which must take into account their own capabilities along with the UCIe dies (e.g., 1330, 1335) capabilities on both packages (e.g., 1310, 1315) to ensure a consistent mode of operation. However, for robustness of the UCIe link (e.g., 1340, 1345), a "stall" response may be defined for the associated sideband message that may potentially be delayed to avoid unnecessary timeouts if the external interconnect 1305 needs more time for link-up or parameter resolution with the remote retimer partner. The retimer may respond (e.g., according to specification-defined rules) with a "stall" response to avoid such unnecessary timeouts while waiting or negotiating with its remote retimer partner. It is the retimer's responsibility to ensure that the UCIe link does not stall indefinitely. As another feature, the UCIe retimer (e.g., 1320, 1325) may be responsible for managing the link state defined for the D2D adapter (e.g., adapter link state machine (LSM)) or RDI state with the remote retimer partner, among other example considerations, to ensure correct E2E operation.

[0085] The retimer may additionally provide flow control and backpressure management logic to manage data transmitted from its on-package UCIe die (e.g., 1330, 1335). For example, data transmitted from the UCIe die to the UCIe retimer may be flow controlled using credits. The credits used to manage this link (e.g., 1340, 1345) may be in addition to any credit protocol defined for the underlying protocol of the data being transmitted (e.g., PRH, PRD credits in PCIe). UCIe die-to-die (D2D) credits may be defined for flow control between two UCIe retimers, and any data transmitted to the UCIe retimer is ultimately consumed by the remote UCIe die without any other dependency. The UCIe retimer (e.g., 1320, 1325) may implement receiver (Rx) buffers (e.g., 1350, 1355) for flits received from the UCIe die (e.g., 1330, 1335) in its package. The receiver buffer credits are advertised to the UCIe die during the initial parameter exchange for the D2D adapter, and the UCIe die may be required not to send any data to the UCIe retimer if it does not have credits to send. In one exemplary implementation, one credit corresponds to 256B of data (including any FEC, CRC, etc.). In one exemplary implementation, the credit return may be overloaded or encoded into the valid frame signal. Whenever the RDI state transitions from active, the credit counter in the UCIe die may be reassigned to the initial advertised value. The UCIe retimer may drain or dump data (as appropriate) in its receiver buffer before re-entering the active state. In some examples, data sent from the UCIe retimer to the UCIe die may not be flow controlled at the D2D adapter level. The UCIe retimer may instead have independent flow control with other UCIe retimers, if necessary, among other exemplary implementations.

[0086] The goal of UCIe is to leverage all of the previous software investments made in popular interconnect protocols, such as PCIe and CXL, while still defining the interface in an extensible manner for future innovative solutions. To this end, the software view of the protocol layers of UCIe can be configured to remain consistent with those associated with the underlying protocols supported by the protocol layer block of the port. For example, a host downstream UCIe port with a protocol layer capable of supporting the CXL protocol may appear to the software as a root port with CXL DVSEC functions and associated PCIe functions. Similarly, a UCIe host downstream port capable of supporting only the PCIe protocol may appear to the software as a root port with only associated PCIe functions. Similarly, all of the CXL / PCIe legacy advanced functions defined in the respective underlying protocol specifications, among other examples, may also apply to UCIe hosts and devices. In the case of streaming protocols, the view of the host or device side software may be implementation specific, since the underlying protocols used in streaming mode may themselves be implementation specific. In some implementations, the streaming solution may leverage software hooks already in place to support CXL and PCIe. In one example implementation, the upstream port connecting to the UCIe root port may be a PCI Express endpoint, a PCI Express switch, a CXL 2.0 or above compliant endpoint device, or a CXL switch. This allows the corresponding UCIe solution to be fully backward compatible with pre-UCIe software (e.g., supporting PCIe or CXL).

[0087] To enable UCIe-aware software, a formal functional split between different layers of the UCIe stack (e.g., between protocol layers, D2D adapter, and PHY) is defined, and the software has visibility into the functionality of each layer on a layer-by-layer basis. Furthermore, link bring-up rules and parameter negotiation are formalized, making use of the layer interfaces (e.g., RDI and FDI) and sideband message channels provided in the PHY module. The formalization of link bring-up rules and parameter negotiation uses sideband messages. Dedicating a sideband channel for all link management negotiation messages and transactions allows the main band to be optimized for latency. The sideband link provides a backchannel for link training and an interface for sideband access of the link partner's registers. It is also used for link management packets and parameter exchange with the remote link partner. The same protocol can be used for local die sideband access through FDI and RDI. The PHY block can be responsible for framing and transporting sideband packets over the UCIe link. Direct sideband access to the remote die may originate from the D2D adapter (e.g., 1010) or the PHY layer block (e.g., 1005). The D2D adapter forwards remote die sideband access to the PHY block via RDI for frames and transport. Such access may include register access request, completion, or other messages. Link training activity performed by the physical layer may be passed to the D2D adapter through physical layer registers, readable by the D2D adapter, that record the results of the link training. The D2D adapter may utilize this information after link training to determine which flit formats / capabilities to advertise. The layered negotiation flow separates link layer flit format optimization from protocol feature negotiation. This makes it easy to extend this flow to a wide variety of protocols, among other exemplary advantages.

[0088] Registers may be defined in the D2D adapter and PHY layers for use during bring-up and training of the UCIe link. Such registers may include function registers (e.g., read-only), control registers (e.g., read-write), and status registers. Such registers may also be utilized to allow software access and control of the UCIe port. In one exemplary implementation, separate registers (e.g., Dedicated Vendor Specific Extensions (DVSEC) registers) may be defined for the D2D adapter and port's PHY (software is permitted to use the presence of these separate registers to identify the device as UCIe). For example, a PCIe DVSEC structure may be utilized.

[0089] 14, a representation 1400 of a high level DVSEC structure is shown illustrating an example implementation of a basic capability register set 1400 used to operate a UCIe link. Software may use the presence of this DVSEC to distinguish between UCIe devices and other devices (e.g., standard PCIe or CXL devices, UCIe root ports, and standard PCIe or CXL root ports, etc.) In the example of FIG. 14, for example, capability registers 1405, control registers 1410, and status registers 1415 may be included within the set of registers.

[0090] The D2D adapter capability register may be utilized to reflect the capabilities of the D2D adapter and is used to advertise and negotiate protocols and parameters with a link partner (e.g., a D2D adapter of a link partner device). Information in the D2D capability register may include, among other examples, whether low mode is supported, whether optimized mode (using spare lanes) is supported, whether CXL is supported, whether PCIe is supported, whether retries are supported, whether the D2D adapter is configured to support multiple protocol layer blocks, whether the D2D adapter supports parity or other error detection / correction schemes, whether the D2D adapter supports streaming modes or protocols, etc. The PHY capability register may include other information such as the number of PHY modules included in the PHY, the byte mapping used, whether the link is terminated or not, and the I / O link speeds enabled by the PHY, among other example information.

[0091] In general, the capability registers may be implemented as pre-populated read-only registers to indicate the capabilities of the associated D2D adapter or PHY layer block. Software may read the capability registers and utilize this information to determine whether these capabilities should be applied (enabled) or disabled. The enablement or disablement of the corresponding capabilities may be set by software or hardware implementations, controlled in corresponding control registers. The D2D adapter and PHY may utilize the values ​​set in the control registers to determine how to proceed with negotiating the final characteristics and capabilities of the link (e.g., with the remote link partner and / or retimer). The status registers may be utilized to record and set to reflect the results of the negotiation with the remote link partner. In one exemplary implementation, among other exemplary implementations, the PCIe / CXL features implemented in the UCIe D2D adapter are not independently controlled, but leverage the control using the existing DVSEC registers defined in the CXL specification.

[0092] As described above, the protocol layer block, the D2D adapter, and / or the PHY block may utilize a local sideband channel (e.g., of the RDI or FDI interface) to access local registers. The protocol layer block has indirect access to the remote die registers using a sideband mailbox mechanism. In some implementations, the mailbox registers (e.g., 1420a-f) reside in the D2D adapter, and it is the responsibility of the D2D adapter to initiate a remote die register access request when it receives a corresponding access trigger for the mailbox register through the FDI. In case of multiple protocol stacks, the D2D adapter tracks which protocol stack sent the original request and routes the completion back over the FDI to the appropriate protocol stack. Because the protocol layers may only allow indirect access to the remote die registers and direct access to the local die registers, register access requests and completions may be limited to a certain number of simultaneous requests (e.g., 1) on the FDI interface. In some implementations, sideband requests that expect a response may be subject to a defined timeout (e.g., a timeout of 8 ms). In such an implementation, a "stall" encoding may be provided in the associated packet for the retimer to prevent a timeout if the retimer needs extra time to respond to a request. When stalling to prevent a timeout, it is the responsibility of the retimer to send a corresponding stall response once every 4 ms. The retimer may also ensure that it does not stall indefinitely, escalating the event to link down after reasonable attempts to complete a resolution that required the requester to stall. If the requester receives a response with a "stall" encoding, it resets the timeout counter.

[0093] In some cases, registers may be fragmented between different layers, e.g., certain bits of a given register are physically present in the protocol layer block, other bits are present in the D2D adapter, and other bits are present in the PHY block. The UCIe may do hierarchical decoding for these registers. In a fragmented register, if a bit is not physically present in a given layer, it may be implemented as read-only tied to 0. In such cases, reads return 0 for those bits from that layer, and writes have no effect on those bits. As an example, for reads, the protocol layer forwards these requests over the FDI to the D2D adapter, which ORs the data responded to by the D2D adapter with its local register before responding to the software. Among other examples, if any bit of that register is present in the PHY layer before responding to the protocol layer, the D2D adapter may do the same.

[0094] Tables 1-5 represent example implementations of registers that may be utilized in implementing a UCIe port. For example, Table 1 shows an example implementation of a UCIe link capabilities register that is utilized by software to discover basic characteristics of a UCIe link. [Table 2] Table 1: Example UCIe Link Capability Registers

[0095] Table 2 shows an example implementation of the UCIe Link Control Register in which the control bits for the link are set. [Table 3] Table 2: Example UCIe Link Control Registers

[0096] Table 3 shows an example implementation of a UCIe Link Status Register in which status bits for a link are set (eg, during and after negotiation with a link partner). [Table 4] Table 3: Example UCIe Link Status Register

[0097] Table 4 shows another example implementation of the UCIe link capability register to include an identification of support for various optimization modes. [Table 5] Table 4: Example UCIe Link Capability Registers

[0098] Table 5 shows another example implementation of a UCIe PHY Status Register that corresponds to the example UCIe Link Capability Register shown in Table 4. [Table 6] Table 6: Example UCIe Link PHY Status Registers

[0099] Table 7 shows an example of a PHY capability register. In some implementations, the PHY layer capability register may be global (in that a separate register is provided for the set of modules that make up the PHY, rather than per module). [Table 7] Table 7: Exemplary PHY Capability Registers

[0100] Table 8 shows examples of PHY control registers. In some implementations, the PHY layer control registers may be global for the PHY. [Table 8] Table 8: Example PHY Control Registers

[0101] Table 9 shows another example of a PHY status register. The status register can be accessed and read by the D2D adapter to understand the characteristics of the link after training of the link by the PHY block and adjust the capabilities it advertises to the link partner D2D adapter based on the results of these link trainings. In some implementations, the PHY layer status register can be global for the PHY. [Table 9] Table 9: Example PHY Status Registers

[0102] The registers utilized by the software for the D2D adapter and / or PHY layer block may include registers hosted in the local memory of the respective blocks, and the MMIO mapping register block defined in the first version of the specification. UCIe aware firmware and software in the system may use these registers to discover the presence and capabilities of the UCIe link. Table 10 summarizes how such discovery may occur in one example implementation. [Table 10] Table 10: Exemplary Software Access Schemes

[0103] An area of ​​memory (e.g., a CIRB area) may be defined at a static location on the device to store each register for the device. Configuration space registers may be accessed using configuration reads and configuration writes. Register blocks in the memory mapped area may be accessed using standard memory reads and writes. In some implementations, the retimer registers are not directly accessible to the host software, but are accessed through a window mechanism through the sideband interface. For debugging and runtime link health monitoring, the host software may also access UCIe related registers in any partner die using the sideband interface (e.g., using the same window mechanism), among other exemplary implementations.

[0104] Parameter exchange and negotiation in UCIe may modularize the functionality between the D2D adapter and the PHY block, as discussed below. For example, FIG. 15A shows a simplified block diagram illustrating an example parameter exchange and negotiation utilizing a UCIe D2D adapter block 1010 and a PHY block 1005 (including one or more PHY modules). A layered negotiation flow may be used that allows the PHY layer block 1005 and the D2D adapter circuit block 1010 to be developed by different IP vendors but still interoperate from a UCIe perspective. In one example implementation, the PHY layer 1005 performs link training for the link. Software may initiate link bring-up and link training (e.g., by writing values ​​to registers (e.g., link control registers) of the D2D adapter), which may then send a signal over RDI to the PHY layer to trigger the link training operation. The PHY layer 1005 may read 1510 registers (e.g., 1505) of the D2D adapter 1010 to determine the capabilities of the D2D adapter and determine the capabilities that the PHY block 1005 may advertise during link training 1515, such as the maximum speed to advertise. In one example, the PHY block may be connected to a D2D adapter that does not support retry flows. As a result, among other example elements and implementations, the PHY block may decide to only advertise speeds where the BER is low enough for the link to function reliably without retries. During link training, the PHY may train the link for each module by utilizing the module's respective sideband lanes to communicate with the remote link partner and negotiate parameters of the link. In one example implementation, the link training 1515 may include transmitter (TX) data to clock point test, TX data to clock eye width sweep, receiver (RX) data to clock point training, and RX data to clock eye width sweep.Parameters and messages to adjust the link training may be sent over the sideband lanes, and test patterns are sent on the data lanes during various link training steps.

[0105] 15A, once link training 1515 is complete, the D2D adapter 1010 reads 1525 registers (e.g., 1520) in the PHY layer block 1005 to identify the results of link training 1515 and adjusts any functionality affected by the results of link training to be consistent with the PHY layer 1005 before advertising capabilities to the link partner D2D adapter. For example, if spare lanes are determined to be unavailable during link training, the D2D adapter may not advertise support for optimized modes that may require the use of such spare lanes, among other examples.

[0106] Protocol and flit format negotiation 1530 and protocol specific parameters 1535 may be performed following link training, where the D2D adapter communicates with the link partner D2D adapter through a training sideband channel of the link. In UCIe, parameter exchange and protocol negotiation may be performed in a multi-phase manner as well as modularized between the D2D adapter and the PHY layer. This allows the architecture to separate link / flit optimization from protocol characteristics. In other contemporary protocols, such as CXL, link and flit format negotiation and optimization are mixed within one negotiation, which does not allow for easy scaling, among other exemplary issues. In UCIe, support for a protocol may be determined first, such as by identifying the presence or absence of a DVSEC associated with a particular underlying protocol, where the presence of such a register indicates support for that protocol. For example, in the case of CXL, the presence or absence of a Flexbus Port DVSEC determines whether CXL is a supported feature in the D2D adapter. Once the protocol and link / flit capabilities have been negotiated with the remote link partner, a second phase of negotiation is performed to determine, among other examples, the details of protocol support (e.g., in the case of CXL, this second phase determines whether CXL.io or CXL.cache or CXL.mem or a combination thereof is supported).

[0107] As one illustrative example, after training of the link by the PHY layer block, the link parameters configured during training may be recorded in a PHY status register (e.g., defined in the UCIe Physical Layer DVSEC function). The D2D adapter may read the PHY status register to identify, for example, whether the link can support one or more optimization modes supported by the D2D adapter. For example, the D2D adapter may read the PHY status register to determine whether spare lanes are available after the lane repair flow (e.g., by reading the corresponding register bit names "Three_Spare_Lanes_Ready" and "Two_Spare_Lanes_Ready") and determine whether the required spare lanes are available to enable the optimization mode to be advertised by the D2D adapter to the link partner D2D adapter. If sufficient spare lanes are not available, the D2D adapter does not advertise support for the corresponding optimization mode to the remote link partner. In one example, if the optimized mode flit format requires N spare lanes, the D2D adapter ensures at least N+1 spare lanes by reading the link training results before advertising support for the format. Any extra lanes (on top of the N lanes used for the optimized mode) are provisioned for field repair to increase the probability that the optimized mode will not fail over the lifecycle of the part. As another example, if link training determines that the BER threshold requires retry support for normal link operation, the PHY status register can be read (e.g., from a register bit "Adapter_Retry_Needed") to inform the D2D adapter. If this bit is set and the D2D adapter can support retry, the D2D adapter advertises this capability to the remote link partner during parameter exchange (e.g., using a sideband message over the training link).In one example implementation, if adapter retry is required, some optimization modes (eg, the 2-lane optimization mode), among other example implementations, may be disabled.

[0108] 15B, a diagram 1500b showing a general parameter exchange flow is shown. When a UCIe link (channel) is brought up and trained, the first D2D adapter 1010 communicates a message (e.g., {AdvCap.Adapter} sideband message) via its corresponding PHY 1005 over the link's sideband channel 1540 that is passed to the link partner's D2D adapter 1550 to be received at the receiver of the link partner PHY 1545, advertising the first endpoint's capabilities 1555 (based on both the D2D adapter's 1010 capabilities and the results of the link training). Similarly, the link partner D2D adapter 1550 advertises its capabilities 1560 via a return sideband message over the link's sideband channel 1540. In some implementations, the link partner D2D adapter 1550 adapts the capabilities it advertises based on the contents of the capabilities message 1555 it receives, omitting capabilities that are not advertised as those supported by the other D2D adapter 1010. As an example, if PCIe or CXL protocol support is advertised, the link partner D2D adapter 1550 (e.g., of an upstream port on a link partner device) waits for a first message 1555 from the other D2D adapter 1010 (e.g., of a downstream port of the device), reviews the capabilities advertised in the message 1555, and then sends its own sideband message 1560 of the advertised capabilities to advertise whether it also supports PCIe or CXL. When the D2D adapter 1010 receives the return capability advertisement message 1560, it determines which capabilities are jointly supported by both D2D adapters and responds with a finalized configuration sideband message 1565 (e.g., a {FinCap.Adapter} sideband message) to finalize the parameter exchange, allowing each D2D adapter to finalize its configuration and communicate this information to the protocol layer block to establish the flit format to be used on the link.

[0109] Referring to FIG. 15C, another diagram 1500c is shown summarizing the high level stages of an example link initialization of an inter-die link 1590 coupling each of the two link partner dies to their respective PHY blocks (e.g., 1005, 1545), D2D adapters (e.g., 1010, 1550), and protocol layer blocks (e.g., 1015, 1592). Four stages may be completed before protocol flit transfers can begin on the main band channel of the link. In this example, the first stage, stage 0, resets the die and its ports in preparation for link initialization. The die reset flows 1570, 1575 are die specific and occur independently for each die. FIG. 15C shows the flows 1570, 1575 at different sizes, indicating that different dies may take different amounts of time to complete their respective implementations of stage 0. Stage 1 (1580) involves sideband detection and training. Stage 2 (1585) involves main band training and repair. Stage 3 (1595) involves parameter exchange between D2D adapters (e.g., 1010, 1550) and may negotiate the protocols and flit formats to be used on the link.

[0110] Stage 3 (1595) may be considered as D2D initialization. When stages 1 and 2 are completed, the RDI state machine transitions to the active state. Upon successful completion of parameter exchange, the D2D adapter reflects the results to the protocol layer blocks connected over the FDI and transitions to executing the FDI bring-up flow. Once the FDI is in the active state, stage 3 of link initialization is completed and protocol flit transfers may begin. If there are two protocol stacks negotiated for operation over the same UCIe link, the FDI bring-up flow is executed independently for each protocol stack. In some implementations, the data width on the FDI is a function of the frequency of operation of the UCIe stacks and the total bandwidth transferred over the UCIe physical link (which depends on the number of lanes and the speed at which the lanes operate). The data width on the RDI is fixed to at least one byte per physical lane per module controlled by the D2D adapter. For example, the flit format in the RDI example configuration may be mapped to a 64 lane module on the UCIe PHY (e.g., in an advanced package configuration).

[0111] As introduced above, each D2D adapter (e.g., 1010, 1550) determines the results of the PHY layer training (e.g., 1580, 1585) to determine what (if any limitations) exist that prevent the D2D adapter from utilizing (and advertising) all of its capabilities, and the characteristics of the link that may require the D2D adapter to implement its various link reliability features (e.g., retries, CRC, FEC, etc.). For example, from the PHY training results, the D2D adapter may determine whether and / or when to retry or utilize other reliability features for a given link speed and configuration determined by the PHY during link training. If the D2D adapter can support retries, it advertises this capability to the remote link partner during parameter exchange. For a UCIe retimer, the D2D adapter may also determine the flow control mechanism to be used, such as the credits advertised for the retimer receiver buffer (e.g., where each credit corresponds to 256B of main band data storage).

[0112] Various capabilities are negotiated between link partners during parameter exchange. Capabilities (if enabled) are communicated to the remote link partner using sideband messages. For example, capabilities are "advertised" by the D2D adapter setting a bit value to 1b in a field corresponding to one of the potential capabilities defined in the Capability Advertisement (e.g., {AdvCap.Adapter}) sideband message. Examples of capabilities may include the following: [Table 11] Table 11: Example functions within a parameter exchange

[0113] Finalizing the configuration based on the parameter exchange may form the basis of an agreement by the D2D adapter regarding the flit format to be used on the main band of the link. The parameter exchange may be utilized to negotiate and determine the protocol to be used and the specific flit format to be used (e.g., where a protocol (e.g., CXL) defines multiple different flit formats under the protocol). Once the flit format and protocol parameters are negotiated and the configuration is finalized for the link 1590, the flit transfer 1594 may begin. For example, in one exemplary implementation, if CXL is advertised by both link partners, CXL is the last protocol to be negotiated. Thus, the CXL bit is set to 1b in the finalized configuration message, and the PCIe, streaming, and any other protocol bits are set to 0b in the finalized configuration message. In another implementation, support for a particular mode of the CXL protocol (e.g., 256B or 68B) may be explicitly advertised in the first round of parameter exchange messages. As an example, if "68B flit mode" is advertised by both link partners, it may be set to 1b in the Finalized Configuration message, while if "CXL 256B flit mode" is advertised by both link partners, it may be set to 1b in the Finalized Configuration message. When CXL (or CXL 256B or 68B) is negotiated, an additional handshake or sideband message exchange may be performed (e.g., using "Advertised Capabilities for CXL" and "Finalized Capabilities for CXL" messages) to determine which details of the multiple CXL flit modes are negotiated. This additional handshake may follow the same flow pattern as in the example of FIG. 15B.

[0114] Continuing with this example, if CXL is not advertised by any one of the link partners but PCIe is advertised by both, then it is the final protocol. In such a case, the "PCIe" bit is set to 1b and the other protocol bits (e.g., "CXL" and "Streaming") are set to 0b in the finalized configuration message. If CXL and PCIe are not negotiated but "Streaming" is advertised by both link partners, then it is the final protocol and "Streaming" is set to 1b and "CXL" and "PCIe" are set to 0b in the finalized configuration message. In one example, if no common protocol can be negotiated, the D2D adapter may transition the RDI to a link error state and take the necessary steps for error escalation.

[0115] As another example of finalizing parameters, if "Raw_Mode" is advertised by both link partners, then "Raw_Mode" is set to 1b in the Finalized Configuration message (e.g., {FinCap.Adapter}). If both link partners advertise "Retry", then adapter retry is enabled and "Retry" is set to 1b in the Finalized Configuration message. If both link partners advertise "Multi-Protocol Enable", then both stack 0 and stack 1 are enabled by the adapter and all three parameters ("Multi-Protocol Enable", "Stack 0 Enable" and "Stack 1 Enable") are set to 1b in the Finalized Configuration message. If "Multi-Protocol Enable" is not negotiated, then the least common multiple is used to determine if stack 0 or stack 1 is enabled and the corresponding bit is set to 1b in the Finalized Configuration message. If both stack enables are advertised, then stack 0 is selected as the operating mode and only stack 0 enable is set to 1b in the finalized configuration message. For latency optimized format, if CXL_LatOpt_Fmt5 or CXL_LatOpt_Fmt6 are advertised by both, then the corresponding bit is set to 1b in the finalized configuration message, etc.

[0116] If the PCIe or CXL protocol is not advertised and the streaming protocol is negotiated, it may not be known whether it is a downstream port or an upstream port, and each side may advertise its capabilities independently. In some implementations, additional vendor-defined sideband messages may be defined and exchanged to negotiate corresponding vendor-specific extensions used to configure specifics of the streaming protocol, etc. If no vendor-specific extensions are negotiated, the finalized configuration may not be implicitly determined to be in raw mode. In such examples, among other examples, a finalized configuration message may not be sent for the streaming protocol, and the D2D adapter a priori determines the vendor-specific requirements (e.g., in design integration or during stage 0 of link bring-up).

[0117] The particular flit format selected is based on the negotiated capabilities (e.g., after being reflected in the link status register following finalization of the configuration). By way of example, FIG. 16 is a decision tree diagram 1600 illustrating one exemplary algorithm used by link partner D2D adapters to interpret the finalized parameter configuration to jointly determine the flit format to be used corresponding to this configuration. For example, the decision tree may consider the finalized configuration, such as whether parity is supported (1605), whether raw mode is supported (1610), whether optimized mode is supported (1615, 1620, 1625), whether streaming is supported, and whether CXL256B, CXL68B, or PCIe flit modes are supported (1630, 1635, 1640, 1645), to ultimately determine which of a plurality (e.g., 10) possible flit formats (e.g., 1650a-j) will be used in the main band communication. The determined flit mode is communicated by the D2D adapter to the protocol layer block which uses it to generate data according to the agreed upon flit format.

[0118] Tables 12 and 13 show another exemplary scheme for determining an appropriate flit format to use based on the finalized configuration negotiated by the link partner's D2D adapter. For example, Table 12 shows a truth table for determining which protocol to utilize based on values ​​in the finalized configuration message. For example, a first finalized configuration message FinCap.Adapter may be utilized in the first handshake, and a second finalized configuration message FinCap.CXL is used when multiple CXL sub-protocols or flit formats are defined and CXL is advertised as supported by both link partners. Table 12 shows various combinations of values ​​(1b indicating enabled) in the finalized configuration message and the resulting protocol determined by the D2D adapter from the corresponding combination of values ​​in the finalized configuration message. [Table 12] Table 12: Exemplary truth table for determining protocol from finalized parameters

[0119] Table 13 is a truth table for determining the particular flit format to be used based on the combination of parameters in the finalized configuration (advertised in the finalized configuration messages (eg, FinCon.Adapter and FinCon.CXL)). [Table 13] Table 13: An exemplary truth table for determining the flit format from the finalized configuration.

[0120] In some implementations, such as those discussed above, among other example implementations, once the protocol and flit format are negotiated during initial link bring-up, they cannot be changed until the UCIe PHY transitions to a reset state.

[0121] 17A-17F illustrate example flit formats that may be selected for use by a link partner's D2D adapter based on a finalized configuration of parameters, such as those discussed above. For example, FIG. 17A illustrates an example format 1700a (e.g., format 1) for a raw mode. FIG. 17B illustrates an example flit format 1700b (e.g., format 2) for a CXL 2.0 68B flit mode. FIG. 17C illustrates an example flit format 1700c (e.g., format 3) for a standard 256B flit mode for PCIe. FIG. 17D illustrates an example flit format 1700d (e.g., format 4) for a standard 256B flit mode for CXL. FIG. 17E illustrates an example flit format 1700e (e.g., format 5) for a latency optimized mode for CXL. FIG. 17F shows an example flit format 1700f (eg, format 6) for another latency optimized mode for CXL, among other possible flit formats.

[0122] 18A-18D show example formats of a register access packet according to one example implementation. For example, FIG. 18A shows an example register access request packet format 1800a according to one example. FIG. 18B shows an example register access completion packet format 1800b. FIG. 18C shows an example format of a message without a data payload, such as a link management packet, a no operation (NOP) packet, or a vendor-defined message packet, among other examples. FIG. 18D shows an example message with a data payload packet format 1800d. Table 14 below summarizes some of the fields included in the example packets shown in FIGS. 18A-18D. [Table 14] Table 14: Field Descriptions for Register Access Request Table 15 shows an exemplary message encoding for inclusion in a register access request. [Table 15] Table 15: Exemplary Message Encodings

[0123] It should be noted that the above apparatuses, methods, and systems may be implemented in any of the electronic devices or systems described above. As a specific example, the following figures provide an exemplary system for utilizing the solutions described herein. As the following systems are described in more detail, a number of different interconnects are disclosed, described, and reviewed from the above description. Although some of the above examples are based on CXL or PCIe-based interconnects and devices, it should be understood that the solutions and features discussed above may be just as easily applied to other interconnects or protocols used to couple sockets, packages, boards, and the like within various computing platforms. As will be readily apparent, the above-described advances and UCIe-based protocols may be applied to any of the interconnects, fabrics, or architectures discussed herein, as well as to other similar interconnects, fabrics, or architectures not expressly specified or described herein.

[0124] Referring to Figure 19, an embodiment of a block diagram for a computing system including a multi-core processor is depicted. Processor 1900 includes any processor or processing device, such as a microprocessor, embedded processor, digital signal processor (DSP), network processor, handheld processor, application processor, co-processor, system on chip (SOC), or other device for executing code. In one embodiment, processor 1900 includes at least two cores, namely cores 1901 and 1902, which may include asymmetric cores or symmetric cores (in the illustrated embodiment). However, processor 1900 may include any number of processing elements, which may be symmetric or asymmetric.

[0125] In one embodiment, a processing element refers to hardware or logic that supports a software thread. Examples of hardware processing elements include thread units, thread slots, threads, process units, content, context units, logical processors, hardware threads, cores, and / or any other element capable of maintaining a processor state, such as an execution state or architectural state. In other words, in one embodiment, a processing element refers to any hardware capable of being associated with code, such as a software thread, an operating system, an application, or independently other code. A physical processor (or processor socket) typically refers to an integrated circuit that potentially contains any number of other processing elements, such as cores or hardware threads.

[0126] A core often refers to logic located on an integrated circuit capable of maintaining independent architectural states, where each independently maintained architectural state is associated with at least some dedicated execution resources. In contrast to a core, a hardware thread typically refers to any logic located on an integrated circuit capable of maintaining independent architectural states, where the independently maintained architectural states share access to the execution resources. As will be appreciated, the nomenclature boundaries between hardware threads and cores overlap when certain resources are shared and other resources are dedicated to architectural states. However, often cores and hardware threads are viewed by the operating system as individual logical processors, and the operating system can schedule operations on each logical processor separately.

[0127] As shown in FIG. 19, physical processor 1900 includes two cores, namely cores 1901 and 1902. Here, cores 1901 and 1902 are considered to be symmetric cores, i.e., cores with the same configuration, functional units, and / or logic. In other embodiments, core 1901 includes an out-of-order processor core and core 1902 includes an in-order processor core. However, cores 1901 and 1902 may be individually selected from any type of core, such as native cores, software-managed cores, cores adapted to execute a native instruction set architecture (ISA), cores adapted to execute a translated instruction set architecture (ISA), co-designed cores, or other known cores. In a heterogeneous core environment (i.e., asymmetric cores), some form of translation, such as binary translation, may be utilized to schedule or execute code on one or both cores. However, as further discussion, the functional units shown in core 1901 are described in more detail below. In the illustrated embodiment, the units in core 1902 operate similarly.

[0128] As shown, core 1901 includes two hardware threads 1901a and 1901b, which may also be referred to as hardware thread slots 1901a and 1901b. Thus, in one embodiment, in some cases, a software entity, such as an operating system, views processor 1900 as four separate processors, i.e., four logical processors or processing elements capable of simultaneously executing four software threads. As alluded to above, a first thread may be associated with architecture state register 1901a, a second thread may be associated with architecture state register 1901b, a third thread may be associated with architecture state register 1902a, and a fourth thread may be associated with architecture state register 1902b. Here, each of the architecture state registers (1901a, 1901b, 1902a, and 1902b) may also be referred to as a processing element, thread slot, or thread unit, as described above. As shown, architecture state registers 1901a are replicated in architecture state registers 1901b so that separate architecture state / context can be stored for logical processor 1901a and logical processor 1901b. Other smaller resources in core 1901, such as instruction pointers and renaming logic in allocator and renamer block 1930, can also be replicated for threads 1901a and 1901b. Some resources, such as the reorder buffer, ILTB 1920, load / store buffers, and queues in reorder / retirement unit 1935, can be shared through partitioning. In some cases, other resources, such as general-purpose internal registers, page table base registers, low-level data caches and data-TLB 1915, execution units 1940, and portions of out-of-order unit 1935, are shared entirely.

[0129] Processor 1900 typically includes multiple other resources, which may be fully shared, shared through partitioning, or dedicated by / to the processing elements. FIG. 19 illustrates merely an exemplary processor embodiment, along with exemplary logical units / resources of the processor. Note that the processor may include or omit any of these functional units, as well as any other known functional units, logic, or firmware not shown. As shown, core 1901 includes a simplified, typical out-of-order (OOO) processor core. However, different embodiments may utilize an in-order processor. The OOO core includes a branch target buffer 1920 for predicting branches to be executed / taken, and an instruction translation buffer (I-TLB) 1920 for storing address translation entries for instructions.

[0130] The core 1901 further includes a decode module 1925 coupled to the fetch unit 1920 for decoding the fetched elements. In one embodiment, the fetch logic includes separate sequencers associated with each of the thread slots 1901a, 1901b. Typically, the core 1901 is associated with a first ISA that defines / specifies instructions executable on the processor 1900. Often, machine code instructions that are part of the first ISA include portions of instructions (called opcodes) that reference / specify the instruction or operation to be performed. The decode logic 1925 includes circuitry that recognizes these instructions from their opcodes and passes the decoded instructions to the pipeline for processing as defined by the first ISA. For example, as discussed in more detail below, in one embodiment, the decoder 1925 includes logic designed or adapted to recognize specific instructions, such as transactional instructions. Recognition by the decoder 1925 results in the architecture or core 1901 taking specific predefined actions to perform the tasks associated with the appropriate instruction. It is important to note that any of the tasks, blocks, operations, and methods described herein may be executed in response to a single or multiple instructions, some of which may be new or old. Note that in one embodiment, multiple decoders 1926 recognize the same ISA (or a subset thereof). Alternatively, in a heterogeneous core environment, decoder 1926 recognizes a second ISA (either a subset of the first ISA or a separate ISA).

[0131] In one example, the allocator and renamer block 1930 includes an allocator for reserving resources such as a register file for storing instruction processing results. However, in some cases, threads 1901a and 1901b are capable of out-of-order execution and the allocator and renamer block 1930 also reserves other resources such as a reorder buffer for tracking instruction results. Unit 1930 may also include a register renamer for renaming program reference registers / instruction reference registers to other registers within processor 1900. Reorder / retirement unit 1935 includes components such as the reorder buffer, load buffer, etc., discussed above, to support out-of-order execution and subsequently store buffers to support in-order retirement of instructions executed out-of-order.

[0132] In one embodiment, the scheduler and execution units block 1940 includes a scheduler unit for scheduling instructions / operations on the execution units. For example, floating point instructions are scheduled on ports of execution units that have an available floating point execution unit. Register files associated with these execution units are also included for storing information instruction processing results. Exemplary execution units include floating point execution units, integer execution units, jump execution units, load execution units, store execution units, and other known execution units.

[0133] A low level data cache and data translation buffer (D-TLB) 1950 is coupled to the execution unit 1940. The data cache stores recently used / operated on elements, such as data operands potentially held in memory coherency states. The D-TLB is for storing recent virtual / linear to physical address translations. As an example, the processor may include a page table structure for dividing physical memory into virtual pages.

[0134] Here, the cores 1901 and 1902 share access to a higher level or further out cache, such as a second level cache associated with the on-chip interface 1910. Note that "higher level" or "further out" refers to cache levels going up from the execution units or further away from the execution units. In one embodiment, the higher level cache is the last cache in the memory hierarchy on the processor 1900, such as a last level data cache, i.e., a second or third level data cache. However, the higher level cache is not so limited, as it may be associated with or include an instruction cache. Rather, a trace cache, which is a type of instruction cache, may be coupled after the decoder 1925 to store traces that were decoded some time ago. Here, in some cases, instructions refer to macroinstructions (i.e., the overall instructions recognized by the decoder), which can be decoded into several microinstructions (micro-operations).

[0135] In the illustrated configuration, the processor 1900 also includes an on-chip interface module 1910. Historically, a memory controller, described in more detail below, has been included in a computing system external to the processor 1900. In this scenario, the on-chip interface 1910 is for communicating with devices external to the processor 1900, such as a system memory 1975, a chipset (which often includes a memory controller hub for connecting to the memory 1975 and an I / O controller hub for connecting to peripheral devices), a memory controller hub, a northbridge, or other integrated circuits. In this scenario, the bus 1905 may include any known interconnect, such as a multi-drop bus, a point-to-point interconnect, a serial interconnect, a parallel bus, a coherent (e.g., cache coherent) bus, a layered protocol architecture, a differential bus, and a GTL bus. In accordance with the principles discussed above, the bus 1905 and other links, interconnects, and fabrics that interconnect elements in the system of FIG. 19 may be implemented using UCIe. In some implementations, one or more of the elements shown in the system of Figure 19 may be provided and interconnected on the same package. In some cases, some of the interconnected elements may be in different packages and may be coupled by off-package links, among other examples.

[0136] Memory 1975 may be dedicated to processor 1900 or shared with other devices in the system. Common examples of types of memory 1975 include DRAM, SRAM, non-volatile memory (NV memory), and other known storage devices. It should be noted that devices 1980 may include a graphics accelerator, graphics processor, or graphics card coupled to a memory controller hub, data storage coupled to an I / O controller hub, a wireless transceiver, a flash device, an audio controller, a network controller, or other known devices.

[0137] However, nowadays, more logic and devices are integrated on a single die such as a SOC, so each of these devices can be integrated on the processor 1900. For example, in one embodiment, a memory controller hub is provided in the same package and / or die as the processor 1900. Here, a portion of the core 1910 (the on-core portion) includes one or more controllers for interfacing with other devices such as a memory 1975 or a graphics device 1980. Configurations including interconnects and controllers for interfacing with such devices are often referred to as on-core (or un-core configurations). As an example, the on-chip interface 1910 includes a ring-type interconnect for on-chip communication and a high-speed serial point-to-point link 1905 for off-chip communication. However, in a SOC environment, much more devices such as network interfaces, co-processors, memory 1975, graphics processor 1980, and any other known computing device / computer interface can be integrated on a single die or integrated circuit to provide a small form factor with high functionality and low power consumption.

[0138] In one embodiment, the processor 1900 may execute compiler, optimizer, and / or translator code 1977 to compile, translate, and / or optimize application code 1976 to support or interface with the apparatus and methods described herein. A compiler often includes a program or set of programs for translating source text / code into target text / code. Typically, compiling program / application code with a compiler is done in multiple phases and passes to convert high-level programming language code into low-level machine or assembly language code. However, single pass compilers may be used for simple compilation. A compiler may use any known compilation technique and perform any known compiler operations such as lexical analysis, preprocessing, parsing, semantic analysis, code generation, code transformation, and code optimization.

[0139] Larger compilers often include multiple phases, but in most cases these phases are included in two general phases: (1) the front-end, i.e., typically where syntactic processing, semantic processing, and some transformation / optimization may occur, and (2) the back-end, i.e., where parsing, transformation, optimization, and code generation generally occur. Some compilers refer to the middle, which describes a blurring of the description between the front-end and back-end of the compiler. As a result, references to insertion, association, generation, or other operations of the compiler may occur in any of the above phases or passes, as well as in any other known phases or passes of the compiler. As an illustrative example, a compiler may potentially insert operations, calls, functions, etc. during one or more phases of compilation (e.g., inserting a call / operation in the front-end phase of compilation and then translating the call / operation into lower-level code during the translation phase). Note that compiler code or dynamic optimization code may insert such operations / calls during dynamic compilation and optimize the code for execution during run-time. As a specific illustrative example, binary code (already compiled code) may be dynamically optimized during run-time, where program code may include dynamically optimized code, binary code, or a combination thereof.

[0140] Similar to a compiler, a translator, such as a binary translator, translates code statically or dynamically to optimize and / or translate the code. Thus, a reference to running code, application code, program code, or other software environment may refer to (1) running a compiler program, optimization code optimizer, or translator, either dynamically or statically, to compile program code, maintain software structure, perform other operations, optimize code, or translate code, (2) running main program code, such as optimized / compiled application code, that includes operations / calls, (3) running other program code, such as libraries, associated with the main program code to maintain software structure, perform other software-related operations, or optimize code, or (4) any combination of (1)-(3).

[0141] Referring now to FIG. 20, a block diagram of a second system 2000 according to an embodiment of the present disclosure is shown. As shown in FIG. 20, the microprocessor system 2000 is a point-to-point interconnect system and includes a first processor 2070 and a second processor 2080 coupled via a point-to-point interconnect 2050. According to the principles discussed above, the links, buses, interconnects, and fabrics that interconnect the elements in the system may be implemented using UCIe. In some implementations, one or more of the elements shown in the system of FIG. 20 may be provided and interconnected on the same package. In some cases, some of the interconnected elements may be in different packages and may be coupled by off-package links, among other examples. Each of the processors 2070 and 2080 may be some version of a processor. In one embodiment, 2052 and 2054 are part of a serial point-to-point coherent interconnect fabric, such as a high-performance architecture. As a result, the solutions described herein may be implemented in UPI or other architectures.

[0142] Although shown with only two processors 2070, 2080, it should be understood that the scope of the disclosure is not so limited. In other embodiments, there may be one or more additional processors within a given processor.

[0143] Processors 2070 and 2080 are shown to include integrated memory controller units 2072 and 2082, respectively. Processor 2070 also includes point-to-point (PP) interfaces 2076 and 2078 as part of its bus controller unit, and similarly, second processor 2080 includes PP interfaces 2086 and 2088. Processors 2070, 2080 may exchange information using PP interface circuits 2078, 2088 via point-to-point (PP) interface 2050. As shown in FIG. 20, IMCs 2072 and 2082 couple the processors to their respective memories, i.e., memory 2032 and memory 2034, which may be portions of main memory local to each processor.

[0144] The processors 2070, 2080 exchange information with the chipset 2090 via individual PP interfaces 2052, 2054 using point-to-point interface circuits 2076, 2094, 2086, 2098, respectively. The chipset 2090 also exchanges information with the high performance graphics circuit 2038 via interface circuit 2092 through the high performance graphics interconnect 2039.

[0145] A shared cache (not shown) may be included within either processor or external to both processors, but connected to the processors via the PP interconnect, thereby allowing local cache information of either processor or both processors to be stored in shared memory when the processors are placed into a low power mode.

[0146] Chipset 2090 may be coupled to a first bus 2016 via an interface 2096. In one embodiment, first bus 2016 may be a bus such as a Peripheral Component Interconnect (PCI) bus, or a PCI Express bus or another third generation I / O interconnect bus, although the scope of the disclosure is not so limited.

[0147] As shown in FIG. 20, various I / O devices 2014 are coupled to a first bus 2016 along with a bus bridge 2018, which couples the first bus 2016 to a second bus 2020. In one embodiment, the second bus 2020 includes a low pin count (LPC) bus. In one embodiment, various devices are coupled to the second bus 2020, including, for example, a keyboard and / or mouse 2022, a communication device 2027, and a storage unit 2028, such as a disk drive or other mass storage device, which often includes instructions / code and data 2030. Additionally, an audio I / O 2024 is shown coupled to the second bus 2020. It should be noted that other architectures are possible. The components included and the interconnect architectures vary. For example, instead of the point-to-point architecture of FIG. 20, a system may also implement a multi-drop bus or other such architectures.

[0148] While the solutions discussed herein have been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom, and it is intended that the appended claims cover all such variations and modifications that fall within the true spirit and scope of the present disclosure.

[0149] A design may go through various stages from creation to simulation to fabrication. Data representing the design may represent the design in multiple ways. First, the hardware may be represented using a hardware description language or other functional description language, as is useful in simulation. In addition, a circuit level model with logic and / or transistor gates may be generated at some stages of the design process. Furthermore, most designs reach a data level at some stage that represents the physical placement of various devices in the hardware model. If conventional semiconductor fabrication techniques are used, the data representing the hardware model may be data specifying the presence or absence of various features on different mask layers of the mask used to fabricate the integrated circuit. In any representation of the design, the data may be stored in the form of any machine-readable medium. Memory, or magnetic or optical storage such as a disk, may be machine-readable media that store such information transmitted via light or electric waves that are modulated or otherwise generated to transmit the information. When an electrical carrier wave indicating or carrying the code or design is transmitted, a new copy is made to the extent that copying, buffering or transmission of the electrical signal is performed. Thus, a communications provider or network provider may store, at least temporarily, an article such as information encoded in a carrier wave on a tangible, machine-readable medium, embodying techniques according to embodiments of the present disclosure.

[0150] A module as used herein refers to any combination of hardware, software, and / or firmware. As an example, a module includes hardware such as a microcontroller associated with a non-transitory medium that stores code adapted to be executed by the microcontroller. Thus, in one embodiment, reference to a module refers to hardware specifically configured to recognize and / or execute code held on the non-transitory medium. Furthermore, in another embodiment, the use of a module refers to a non-transitory medium that includes code that is specifically adapted to be executed by a microcontroller to perform a number of predefined operations. As may be expected, in yet another embodiment, the term module (in this example) may refer to a combination of a microcontroller and a non-transitory medium. Often, the boundaries of multiple modules that are shown as separate typically vary and potentially overlap. For example, a first module and a second module may share hardware, software, firmware, or a combination thereof, while potentially maintaining some independent hardware, software, or firmware. In one embodiment, the use of the word "logic" includes hardware such as transistors, registers, or other hardware such as programmable logic devices.

[0151] In one embodiment, the use of the phrase "configured to" refers to arranging, assembling, manufacturing, offering to sell, importing, and / or designing an apparatus, hardware, logic, or element to perform a specified or determined task. In this example, an inoperative apparatus or element thereof is still "configured" to perform said specified task if it is designed, coupled, and / or interconnected to perform said specified task. As a purely illustrative example, a logic gate may provide a 0 or a 1 during operation. However, a logic gate "configured" to provide an enable signal to a clock does not include any underlying logic gate that may provide a 1 or a 0. Instead, the logic gate is coupled in some manner such that a 1 or 0 output enables the clock during operation. Again, note that the use of the term "configured" does not require an operation, but instead focuses on the hidden state of the apparatus, hardware, and / or element. In a hidden state, the device, hardware and / or element is designed to perform a particular task when the device, hardware and / or element is operational.

[0152] Additionally, in one embodiment, the use of the phrases "to," "capable of / to," and / or "operable to" refers to any device, logic, hardware, and / or element that is designed to enable use of the device, logic, hardware, and / or element in a specified manner. It should also be noted above that in one embodiment, the use of the words "to," "capable of," or "operable" refers to the hidden state of the device, logic, hardware, and / or element, where the device, logic, hardware, and / or element is not operational but is designed to enable use of the device in a specified manner.

[0153] Values, as used herein, include any known representation of a number, state, logic state, or binary logic state. Often the use of logic levels, logic values, or logic values, also referred to as "ones" and "zeros," simply represent binary logic states. For example, a one refers to a high logic level and a zero refers to a low logic level. In one embodiment, a storage cell, such as a transistor cell or a flash cell, may be capable of holding a single logic value or multiple logic values. However, other representations of values ​​in computer systems are used. For example, the decimal number 10 may be represented as the binary value 1010, or in hexadecimal as the letter A. Thus, values ​​include any representation of information capable of being held in a computer system.

[0154] Further, a state may be represented by a value or a portion of a value. As an example, a first value, such as a logical one, may represent a default or initial state, while a second value, such as a logical zero, may represent a non-default state. Additionally, in one embodiment, the terms reset and set refer to default and updated values ​​or states, respectively. For example, a default value potentially includes a high logical value, i.e., reset, while an updated value potentially includes a low logical value, i.e., set. It should be noted that any combination of multiple values ​​may be utilized to represent any number of states.

[0155] The above-described methods, hardware, software, firmware, or code embodiments may be implemented via instructions or codes stored on a machine-accessible, machine-readable, computer-accessible, or computer-readable medium executable by a processing element. A non-transitory machine-accessible / readable medium includes any mechanism that provides (i.e., stores and / or transmits) information in a form readable by a machine, such as a computer or electronic system. For example, non-transitory machine-accessible media include random access memory (RAM), such as static RAM (SRAM) or dynamic RAM (DRAM), ROM, magnetic or optical storage media, flash memory devices, electronic storage devices, optical storage devices, acoustic storage devices, other types of storage devices for holding information received from a transitory (propagated) signal (e.g., carrier wave, infrared signal, digital signal), and the like, and these non-transitory machine-accessible media should be distinguished from non-transitory media that can receive information from these media.

[0156] The instructions used to program the logic to execute the exemplary embodiments herein may be stored in the memory of the system, such as DRAM, cache, flash memory, or other storage. Additionally, the instructions may be distributed over a network or using other computer-readable media. Thus, a machine-readable medium may include any mechanism for storing or transmitting information in a form readable by a machine (e.g., a computer), including but not limited to tangible machine-readable storage such as floppy disks, optical disks, compact disks, read-only memories (CD-ROMs), and magnetic optical disk read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memories, or those used in transmitting information over the Internet via electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.). Thus, a computer-readable medium includes any type of tangible machine-readable medium suitable for storing or transmitting electronic instructions or information in a form readable by a machine (e.g., a computer).

[0157] The following examples relate to embodiments according to the present disclosure.

[0158] Throughout this specification, reference to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the disclosure. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" in various places throughout this specification do not necessarily all refer to the same embodiment. Furthermore, the particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0159] In the foregoing specification, the detailed description has been given with reference to certain exemplary embodiments. It will be apparent, however, that various modifications or changes may be made thereto without departing from the broader spirit and scope of the invention as set forth in the appended claims. Accordingly, the specification and drawings should be regarded in an illustrative rather than a restrictive sense. Moreover, the above use of embodiment and other exemplary language does not necessarily refer to the same embodiment or the same example, but may refer to different individual embodiments and potentially the same embodiment.

Claims

1. At least one non-transitory machine-readable storage medium having instructions stored thereon, the instructions being executable by a machine to cause the machine to: determining, based on the presence of a particular set of registers on a die, that the die is compliant with a Universal Chiplet Interconnect Express (UCIe) based protocol, where the die is coupled to another link partner die by a die-to-die (D2D) link based on the UCIe based protocol; accessing a functional register in a particular set of registers; reading available capabilities from the capability register, where the available capabilities include capabilities of a D2D adapter and a physical layer (PHY) block present on the die; determining a set of features in the available features that are enabled for the D2D link; writing a value in a control register to enable the set of features; and writing a start link training value to the control register to start training of the D2D link; At least one non-transitory machine-readable storage medium that causes

2. The storage medium of claim 1 , wherein the available functions include a raw mode.

3. The storage medium of claim 1 , wherein the available functionality includes functionality for supporting multiple protocol layers.

4. 4. The storage medium of claim 1, wherein the D2D adapter is coupled to the PHY block on the die by a first interface and to a protocol layer block on the die by a second interface, the D2D adapter being located between the protocol layer block and the PHY block in a UCIe protocol stack.

5. The storage medium of claim 4 , wherein the PHY block performs training of the D2D link with the link partner die.

6. 6. The storage medium of claim 1, wherein the D2D adapter uses values ​​in the control register to negotiate with the link partner die a protocol and flit format to be used on the D2D link.

7. The storage medium of claim 6 , wherein the flit format is determined from one of a plurality of different flit formats defined in the UCIe-based protocol.

8. 7. The storage medium of claim 6, wherein the protocol is determined from one of a plurality of different protocols defined in the UCIe base protocol, and the flit format is determined based on the protocol.

9. 1. An apparatus comprising an inter-die adapter, The die-to-die adapter includes: A set of registers; determining, from the set of registers, a set of capabilities of the D2D adapter to advertise in negotiation with a link partner D2D adapter, where the D2D adapter is on a die and the link partner D2D adapter is located on a remote link partner die; sending a first capability advertisement message to the link partner D2D adapter to advertise the set of capabilities to the link partner D2D adapter; receiving a second capability advertisement message from the link partner D2D adapter, where the second capability advertisement message identifies a set of capabilities of the link partner D2D adapter; determining shared capabilities of the D2D adapter and the link partner D2D adapter; and determining a final configuration of a D2D link for coupling said die to said link partner die; Circuit for performing Includes, an apparatus.

10. 10. The apparatus of claim 9, wherein the link includes a main band channel and a separate sideband channel, the first capability advertisement message being transmitted on the sideband channel and the second capability advertisement message being received on the sideband channel.

11. The apparatus of claim 10 , wherein a format of data transmitted on the main band channel is based on the final configuration of the link.

12. The apparatus of claim 11 , wherein the D2D adapter selects one of a plurality of flit formats for the format based on the final configuration of the link.

13. 13. The apparatus of claim 10, wherein the circuitry is further configured to transmit a final configuration message over the sideband channel to the link partner D2D adapter to identify the shared capabilities to the link partner D2D adapter.

14. 14. The apparatus of claim 9, wherein the set of functions is based on a result of training of the link performed by a physical layer (PHY) block on the die, and the D2D adapter is located between the PHY block and a protocol layer block on the die to implement a protocol stack.

15. 15. An apparatus according to claim 9, wherein the set of capabilities is based on values ​​written by software to control registers in the set of registers.

16. The apparatus of claim 9 , wherein a retry function of the circuit is enabled based on the outcome of physical layer training of the D2D link.

17. First die; a second die coupled to the first die by a die-to-die (D2D) link, where the second die includes a port for connecting the second die to the D2D link, the port comprising: protocol layer logic for implementing a particular interconnect protocol; D2D adapter circuit; physical layer (PHY) circuit; and One or more registers associated with the D2D adapter circuit and the PHY circuit. Contains; and A software system for reading and writing values ​​to said one or more registers. Equipped with Here, the D2D adapter circuit is determining a subset of capabilities to be advertised to a D2D adapter on the first die based on values ​​in the one or more registers; and transmitting a capabilities message over a sideband channel of the D2D link to identify the subset of capabilities. system.

18. 20. The system of claim 17, wherein the first die includes a processor device and the second die includes one of another processor device, a hardware accelerator, or an input / output (I / O) device.

19. 19. The system of claim 17, wherein the first die and the second die are on the same package.

20. 20. The system of claim 17, wherein the first die includes a first retimer and the second die includes a second retimer, the first die and the second die are on separate packages, and the D2D link implements an off-package interconnect.

21. determining that the die complies with a Universal Chiplet Interconnect Express (UCIe) based protocol based on the presence of a particular set of registers on a die, where the die is coupled to another link partner die by a die-to-die (D2D) link based on the UCIe based protocol; accessing a functional register in said particular set of registers; reading available capabilities from the capability register, where the available capabilities include capabilities of a D2D adapter and a physical layer (PHY) block present on the die; determining a set of features in the available features that are enabled for the D2D link; writing a value in a control register to enable the set of features; and writing a start link training value to the control register to start training of the D2D link; A method for providing the above.

22. The method of claim 21 , wherein the available features include a low mode.

23. 23. The method of any one of claims 21 to 22, wherein the available capabilities include the capability to support multiple protocol layers.

24. 24. The method of claim 21, wherein the D2D adapter is coupled to the PHY block on the die by a first interface and to a protocol layer block on the die by a second interface, the D2D adapter being located between the protocol layer block and the PHY block in a UCIe protocol stack.

25. The method of claim 24, wherein the PHY block performs training of the D2D link with the link partner die.

26. 26. The method of claim 21, wherein the D2D adapter uses values ​​in the control register to negotiate with the link partner die a protocol and flit format to be used on the D2D link.

27. 27. The method of claim 26, wherein the flit format is determined from one of a plurality of different flit formats defined in the UCIe-based protocol.

28. 27. The method of claim 26, wherein the protocol is determined from one of a plurality of different protocols defined in the UCIe base protocol, and the flit format is determined based on the protocol.

29. A system comprising means for carrying out the method according to any one of claims 21 to 28.

Images